Is it possible to use HTTP to fetch JSON data on scheduler? by Adam Brodzinski

Adam Brodzinski INVESTOR

| Reward Discussion

I've looked through the docs and i'm wondering if there is a way that I can setup a scheduler to make an HTTP request before the market opens so that I can have my external server process data and return it to the algorithm before i'm ready to trade.

Currently I have to hard code it and then re-launch the algo every day, which works but is not ideal. I'm also new to C# so I wasn't sure if there was a standard way to do this outside of the LEAN framework.

// server data  ["AAPL", "GOOG", "ETC"]





Schedule.On(DateRules.EveryDay(), TimeRules.AfterMarketOpen("SPY", -20), () =>

{

  List<string> mySymbols = HTTP_GET_JSON("http://myserver/getdata");



  foreach (var symbol in mySymbols) {

    Log(symbol);

  }

});

Update Backtest

Adam Brodzinski

Accepted Answer

Update Backtest

Notebook

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Ray Bohac

Have you tried using the HttpWebRequest class?

http://stackoverflow.com/questions/8270464/best-way-to-call-a-json-webservice-from-a-net-console

Adam Brodzinski

163 ,

Thanks for the help Ray! I just tried to follow the first Stack Overflow answer and i'm getting a "Sorry System.IO is not whitelisted" when building.

From what I understand System.IO is needed to use the `GetResponseStream` method in the example.

Do you know if there's another way to use it with LEAN?

Stefano Raggi

If the file isn't huge, you could read it directly into a string using WebClient.DownloadString:

var json = new WebClient().DownloadString(url);

To use WebClient class, you'll also need to reference the System.Net namespace:

using System.Net;

Alexandre Catarino

We can download a strings using System.Net.WebClient and convert the downloaded string into the List of string with Newtonsoft.Json:

using System.Net;

using Newtonsoft.Json;



// In Initialize

Schedule.On(

    DateRules.EveryDay(), 

    TimeRules.AfterMarketOpen("SPY", -20), () =>

{

    using (var client = new WebClient())

    {

        var json = client.DownloadString("http://myserver/getdata");

					

        var mySymbols = JsonConvert.DeserializeObject<List<string>>(json);



        foreach (var symbol in mySymbols) {

    	    Debug(Time + ": " + symbol);

        }

    }

});

We can tested it with

var json = "[\"AAPL\", \"GOOG\", \"ETC\"]";

Adam Brodzinski

163 ,

Thanks Stefano and Alexandre, it's working great now!

Much appreciated!

Drew Baker

219 ,

Would love to know how to do this in Python! Any tips?

Drew Baker

219 ,

I tried this in Python, but get an `Sorry, 'open(' is not whitelisted within QuantConnect. Remove this code statement` error.

import urllib2

import json        



    def UpdateOnTimer(self):

        # Test an HTTP request

        response = urllib2.urlopen('https://api.github.com/events')

        data = json.load(response)

        print data

Esther Ng

112 ,

Hi,

1. Do I use DownloadFile if I want to download a csv instead?
1a. If so, where exactly will the file be stored?
1b. How do I know that there won't be a file conflict?

2. How do I pass basic header authentication to the URL that I'm trying to request the csv from?

Thank you.

Drew Baker

219 ,

This is how I got it to work with some help from QC supprot:

        response = self.Download('https://api.github.com/events')

        data = json.loads(response)

        self.Log(response, str(data))

Esther Ng

112 ,

Actually, I figured it out since I'm using C#.

Using WebClient, one can pass in the credentials to the headers:

string credentials = Convert.ToBase64String(Encoding.ASCII.GetBytes("username:pwd"));

client.Headers[HttpRequestHeader.Authorization] = string.Format("Basic {0}", credentials);

I wanted to download a CSV, not a JSON, so if the API allows CSV downloads, something similar to the following can be used. For dropbox, it will allow a CSV content download if you add the querystring raw=1 (else, it will download the HTML)

string uri = "https://www.dropbox.com/s/*.csv?raw=1";

byte[] bytes = client.DownloadData(uri);

csvText = Encoding.ASCII.GetString(bytes);

I do get errors about string not being in the correct format, but it seems like it still saw the data fine so I haven't been able to debug where that incorrect format came from.

Michael Manus

10.2k ,

oh you are searching dropbox support.

why havent you take a look at the respository with some sample algos from the QC staff.

scroll down to the dropbox examples:

https://github.com/QuantConnect/Lean/tree/master/Algorithm.CSharp

Esther Ng

112 ,

Not exactly. As specified in my comment, I just wanted to download any csv from any API, especially with authentication. Therefore, I didn't search for dropbox in specific. I'll try to search for more specific examples from LEAN next time.

Adam Brodzinski INVESTOR

Update Backtest

Notebook

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!

Platform

QuantConnect Community

Alpha League Competition: $1,000 Weekly Prize Pool

Filter Discussion by Tags

Community Ambassadors

Earn free QuantConnect cloud credits by helping others, each week we airdrop credits to the most active community members.

232,145 Quants.

Online Now

Stay Connected