downloading web-page by TP

Get help with installation and running here.

Moderators: DataMystic Support, Moderators

nikolas1612
Posts: 22
Joined: Tue May 12, 2015 3:57 am

downloading web-page by TP

Postby nikolas1612 » Sun May 22, 2016 11:49 pm

is there any possibility to download by TP the source-code of the following web-page:

Code: Select all

https://sev.life/


Thanks

User avatar
DataMystic Support
Site Admin
Posts: 2138
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: downloading web-page by TP

Postby DataMystic Support » Mon May 23, 2016 4:06 pm

Yes, just put the url in the file list, and set the output filter to write to a new file.
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments

nikolas1612
Posts: 22
Joined: Tue May 12, 2015 3:57 am

Re: downloading web-page by TP

Postby nikolas1612 » Mon May 23, 2016 6:47 pm

Hmmm
Did you actually try it?
The result is always blank, while option "analyze file" produces "Internet error 0" message.

DFH
Posts: 636
Joined: Sun Dec 09, 2007 2:49 am
Location: UK

Re: downloading web-page by TP

Postby DFH » Mon May 23, 2016 10:30 pm

https://sev.life/
is not a file. It's a remote folder.

TP requires a list of files or a wildcard to be specified.

But can TP even input files over TSL or SSL ?

And how do we know that any files in that folder are free from malware?

David

nikolas1612
Posts: 22
Joined: Tue May 12, 2015 3:57 am

Re: downloading web-page by TP

Postby nikolas1612 » Mon May 23, 2016 11:55 pm

is not a file. It's a remote folder.

I know that, yet I was asking about downloading web-page (not a file). I cannot agree TP needs just files as I successfully downloaded web-pages with the links similar to the one you see above. The problem is just with some pages and this is one of them. I guess the matter is secure protocol used (https) but not for sure.

User avatar
DataMystic Support
Site Admin
Posts: 2138
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: downloading web-page by TP

Postby DataMystic Support » Tue May 24, 2016 11:03 am

Yes, TextPipe can download any url. It only checks for :// in the string, so https should still work.

No idea why https is not working.
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments

alnico
Posts: 74
Joined: Fri Oct 12, 2007 11:57 pm

Re: downloading web-page by TP

Postby alnico » Sat Aug 06, 2016 4:19 am

I also have issues with accessing sites using https

It seems like Textpipe is simply ignoring filenames beginning with https as there is no file input record created in the status tab when processing.
Furthermore, it skips these filenames in a split second (no time for even sending the request to the server).

Sometimes a https website can be accessed using http, if http resolves to https, but not always...

Here are a couple of restaurant sites for example:
testing in a browser...

--> This works:

http://seasalteatery.wordpress.com
resolves to
https://seasalteatery.wordpress.com
200 OK
Textpipe can access this website using filename: http://seasalteatery.wordpress.com


-->This does not work:

http://www.spoonandstable.com
resolves to
https://www.spoonandstable.com
301 moved permanently
Textpipe can not access this website using filename: http://www.spoonandstable.com
This is what is received:
<html>
<head><title>403 Forbidden</title></head>
<body bgcolor="white">
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>


Finally however, the problem really "seems" to be that Textpipes isn't even attempting to make a server request for https.

Brent
TP v8.9.9

User avatar
DataMystic Support
Site Admin
Posts: 2138
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: downloading web-page by TP

Postby DataMystic Support » Sun Aug 07, 2016 11:03 pm

I have confirmed the https redirection issue, and updated the code for Windows 10 where it now works perfectly.
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments

alnico
Posts: 74
Joined: Fri Oct 12, 2007 11:57 pm

Re: downloading web-page by TP

Postby alnico » Tue Aug 09, 2016 12:14 am

Thanks for fix Simon,

But this does not fix the bigger issue where https urls are skipped all together (unless this has been fixed since TP v8.9.9).
They simply are not even logged during processing the input file list. i.e. original request from nikolas1612 for https://sev.life/

User avatar
DataMystic Support
Site Admin
Posts: 2138
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: downloading web-page by TP

Postby DataMystic Support » Tue Aug 09, 2016 10:07 pm

Thanks - this was still an issue, and took a little bit to get it working.

Now it's all ready for TextPipe 10.0!
Regards,

Simon Carter, http://DataMystic.com/forums/index.php
http://PredictBGL.com - Insulin dose calculator for Type 1 diabetes
http://DownloadPipe.com - 250,000 free software downloads
http://DetachPipe.com - send huge email attachments


Return to “TextPipe Tips and Tricks, Questions and Support”

Who is online

Users browsing this forum: No registered users and 3 guests