Back

Scheduled web scraping in python

I'm trying to simple download a website at scheduled times everyday - similar to what Travis does here in C#

https://www.quantconnect.com/forum/discussion/1085/contribution-scheduled-data-scraping-from-any-webpagesource/p1


However, i'm prevented from doing this in the standard way via using urllib/urllib2. I'm a bit new to python.net and tried to implement something similar to what Travis did, but can't seem to get it to work. Any help is much appreciated. 

from System.Net import WebClient

site = WebClient().DownloadString("https://www.google.com/")
self.Log("what is this?" + site) # Shows no value

Also - I would like to make a feature request for the bs4 python module.

Thanks!
 

Update Backtest








We are working on a build-in method that will allow us to download data.
Hopefully it will be available next week.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed