Back

Is it possible to use HTTP to fetch JSON data on scheduler?

I've looked through the docs and i'm wondering if there is a way that I can setup a scheduler to make an HTTP request before the market opens so that I can have my external server process data and return it to the algorithm before i'm ready to trade. 

Currently I have to hard code it and then re-launch the algo every day, which works but is not ideal. I'm also new to C# so I wasn't sure if there was a standard way to do this outside of the LEAN framework.

// server data ["AAPL", "GOOG", "ETC"]


Schedule.On(DateRules.EveryDay(), TimeRules.AfterMarketOpen("SPY", -20), () =>
{
List<string> mySymbols = HTTP_GET_JSON("http://myserver/getdata");

foreach (var symbol in mySymbols) {
Log(symbol);
}
});

 

Update Backtest








Have you tried using the HttpWebRequest class?

 

http://stackoverflow.com/questions/8270464/best-way-to-call-a-json-webservice-from-a-net-console

 

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thanks for the help Ray! I just tried to follow the first Stack Overflow answer and i'm getting a "Sorry System.IO is not whitelisted" when building. 

From what I understand System.IO is needed to use the `GetResponseStream` method in the example.

Do you know if there's another way to use it with LEAN?

0

If the file isn't huge, you could read it directly into a string using WebClient.DownloadString:
 

var json = new WebClient().DownloadString(url);

To use WebClient class, you'll also need to reference the System.Net namespace:
 

using System.Net;

 

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


We can download a strings using System.Net.WebClient and convert the downloaded string into the List of string with Newtonsoft.Json:

using System.Net;
using Newtonsoft.Json;

// In Initialize
Schedule.On(
DateRules.EveryDay(),
TimeRules.AfterMarketOpen("SPY", -20), () =>
{
using (var client = new WebClient())
{
var json = client.DownloadString("http://myserver/getdata");

var mySymbols = JsonConvert.DeserializeObject<List<string>>(json);

foreach (var symbol in mySymbols) {
Debug(Time + ": " + symbol);
}
}
});

We can tested it with 

var json = "[\"AAPL\", \"GOOG\", \"ETC\"]";

 

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thanks Stefano and Alexandre, it's working great now!

Much appreciated!

0

Would love to know how to do this in Python! Any tips?

0

I tried this in Python, but get an `Sorry, 'open(' is not whitelisted within QuantConnect. Remove this code statement` error.

import urllib2
import json

def UpdateOnTimer(self):
# Test an HTTP request
response = urllib2.urlopen('https://api.github.com/events')
data = json.load(response)
print data
0

Hi, 

1. Do I use DownloadFile if I want to download a csv instead?
1a. If so, where exactly will the file be stored?
1b. How do I know that there won't be a file conflict?

2. How do I pass basic header authentication to the URL that I'm trying to request the csv from?

Thank you.

0

This is how I got it to work with some help from QC supprot:
 

response = self.Download('https://api.github.com/events')
data = json.loads(response)
self.Log(response, str(data))



 

0

Actually, I figured it out since I'm using C#. 

Using WebClient, one can pass in the credentials to the headers:

string credentials = Convert.ToBase64String(Encoding.ASCII.GetBytes("username:pwd"));
client.Headers[HttpRequestHeader.Authorization] = string.Format("Basic {0}", credentials);

I wanted to download a CSV, not a JSON, so if the API allows CSV downloads, something similar to the following can be used. For dropbox, it will allow a CSV content download if you add the querystring raw=1 (else, it will download the HTML)

string uri = "https://www.dropbox.com/s/*.csv?raw=1";
byte[] bytes = client.DownloadData(uri);
csvText = Encoding.ASCII.GetString(bytes);

I do get errors about string not being in the correct format, but it seems like it still saw the data fine so I haven't been able to debug where that incorrect format came from.

0

oh you are searching dropbox support.

why havent you take a look at the respository with some sample algos from the QC staff.

scroll down to the dropbox examples:

https://github.com/QuantConnect/Lean/tree/master/Algorithm.CSharp
0

Not exactly. As specified in my comment, I just wanted to download any csv from any API, especially with authentication. Therefore, I didn't search for dropbox in specific. I'll try to search for more specific examples from LEAN next time.

0

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed