QUANTCONNECT COMMUNITY
I spent some time searching for this but I was unable to determine how to do this. I'm using Alpaca Markets as a brokerage, they use Polygon.io for their data feed. Now, previously I've been pulling n days worth of O,H,L,C,V historic data into memory via Alpaca's API and then perform backtesting against the data.
Is it possible to fetch bulk data from another provider and "set" it into and algo for backtesting?
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
QuantConnect™ 2024. All Rights Reserved
|
|
Share New Research 
Alexandre Catarino
Yes, you can fetch bulk data and use it as custom data in Lean/QuantConnect.
Please check out the docs, under Importing Custom Data section.
Dan beaulieu
I've read that document but it's not clear how its possible to take data that I'm holding in memory and apply it. From what I can see in the documents, I can reference a file (dropbox example) or I can set up my own custom data source, but from what i can tell the custom data source is responsible for pulling data itself. If I'm missing something in that document please point me to it, because I am not finding a way to simply apply data that I'm holding in memory as my BaseData.
Alexandre Catarino
How have you "been pulling n days worth of O,H,L,C,V historic data into memory via Alpaca's API" codewise?
Dan beaulieu
Hey Alexandre, I appriciate your response. So prior to looking into QuantConnect. Purely with Alpaca's .Net SDK. I would call their ListMinuteAggregatesAsync method and then aggrigate their data into Candles. I would collect as much data as I needed and then perform backtesting against it.
So, I'd like to take this payload of data and then use it with the QuantConnect library if possible.
Dan beaulieu
Hey Alexandre,
Here's a code example fwiw:
public async Task<List<Candle>> GetCandles(string symbol, DateTime? start = null, DateTime? end = null) { Initialize(); try { var minutes = (end != default(DateTime)) ? await _client.ListMinuteAggregatesAsync(symbol, start, end) : await _client.ListMinuteAggregatesAsync(symbol, DateTime.Now.Date); return minutes .Items.ToList() .OrderBy(x => x.Time) .Select(x => new Candle(x.Time, x.Open, x.High, x.Low, x.Close, x.Volume)).ToList(); } catch (Exception x) { System.Diagnostics.Debug.WriteLine($"{x}"); // TODO: log exception } return null; } public async Task<List<Candle>> GetBuffer(string symbol, int numberOfDays) { Initialize(); var candles = new List<Candle>(); // want to get all days up to previous day foreach (var index in Enumerable.Range(2, numberOfDays).Reverse()) { var daysPast = index * -1; var result = await GetCandles(symbol, DateTime.Now.AddDays(daysPast).Date, DateTime.Now.AddDays(daysPast + 1).Date); if (result?.Any() ?? false) { candles.AddRange(result); } } return candles; }Then I would just call something like this:
var buffer = await _alpaca.GetBuffer(CurrentAlgorithm.Symbol, numberOfDays: CurrentAlgorithm.BacktestLookbackDays);Dan beaulieu
So, after re-reading the documentation and browsing the QuantConnect code it appears that what I'm asking for is not possible. In order for me to use QuantConnect I will need to set up my own BaseData class using Alpacas web API rather than their .Net SDK.
Is this correct Alexandre? If so, I have some follow-up questions which are likely more suitable for a new thread as they are out of the scope of the original question.
Alexandre Catarino
It is only possible to use it if you are using Lean locally because you would need the SDK to be installed in QuantConnect Cloud.
Locally, you would need to implement a data queue handler and a history provider. I think it would be kind of "hacky" to use a historical request in a data queue, but that is what it looks you are aiming for.
Alternatively, you can go for the easiest solution: use your code to download the data, write it as csv files and use the import custom data feature.
In the CustomDataBitcoinAlgorithm, we use Bitstamp's web API and probably you can do the same with Alpaca's.
Dan beaulieu
Thanks for taking the time to explain this to me Alexandre.
Dan beaulieu
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!