Back

Is it possible to backtest against in-memory data that came from another API?

I spent some time searching for this but I was unable to determine how to do this. I'm using Alpaca Markets as a brokerage, they use Polygon.io for their data feed. Now, previously I've been pulling n days worth of O,H,L,C,V historic data into memory via Alpaca's API and then perform backtesting against the data. 

Is it possible to fetch bulk data from another provider and "set" it into and algo for backtesting?

Update Backtest







0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Yes, you can fetch bulk data and use it as custom data in Lean/QuantConnect.
Please check out the docs, under Importing Custom Data section.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


I've read that document but it's not clear how its possible to take data that I'm holding in memory and apply it.  From what I can see in the documents, I can reference a file (dropbox example) or I can set up my own custom data source, but from what i can tell the custom data source is responsible for pulling data itself. If I'm missing something in that document please point me to it, because I am not finding a way to simply apply data that I'm holding in memory as my BaseData.

0

How have you "been pulling n days worth of O,H,L,C,V historic data into memory via Alpaca's API" codewise?

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hey Alexandre, I appriciate your response. So prior to looking into QuantConnect. Purely with Alpaca's .Net SDK. I would call their ListMinuteAggregatesAsync method and then aggrigate their data into Candles. I would collect as much data as I needed and then perform backtesting against it.

So, I'd like to take this payload of data and then use it with the QuantConnect library if possible.

0

Hey Alexandre,

Here's a code example fwiw:

public async Task<List<Candle>> GetCandles(string symbol, DateTime? start = null, DateTime? end = null)
{
Initialize();
try
{
var minutes =
(end != default(DateTime))
? await _client.ListMinuteAggregatesAsync(symbol, start, end)
: await _client.ListMinuteAggregatesAsync(symbol, DateTime.Now.Date);

return
minutes
.Items.ToList()
.OrderBy(x => x.Time)
.Select(x => new Candle(x.Time, x.Open, x.High, x.Low, x.Close, x.Volume)).ToList();

}
catch (Exception x)
{
System.Diagnostics.Debug.WriteLine($"{x}");
// TODO: log exception
}

return null;
}

public async Task<List<Candle>> GetBuffer(string symbol, int numberOfDays)
{
Initialize();
var candles = new List<Candle>();

// want to get all days up to previous day
foreach (var index in Enumerable.Range(2, numberOfDays).Reverse())
{
var daysPast = index * -1;
var result = await GetCandles(symbol, DateTime.Now.AddDays(daysPast).Date, DateTime.Now.AddDays(daysPast + 1).Date);
if (result?.Any() ?? false)
{
candles.AddRange(result);
}
}

return candles;
}

Then I would just call something like this:

var buffer = await _alpaca.GetBuffer(CurrentAlgorithm.Symbol, numberOfDays: CurrentAlgorithm.BacktestLookbackDays);
0

So, after re-reading the documentation and browsing the QuantConnect code it appears that what I'm asking for is not possible. In order for me to use QuantConnect I will need to set up my own BaseData class using Alpacas web API rather than their .Net SDK.

Is this correct Alexandre? If so, I have some follow-up questions which are likely more suitable for a new thread as they are out of the scope of the original question.

0

  • There is two question here. First, let's talk about Alpaca .Net SDK:

It is only possible to use it if you are using Lean locally because you would need the SDK to be installed in QuantConnect Cloud. 

Locally, you would need to implement a data queue handler and a history provider. I think it would be kind of "hacky" to use a historical request in a data queue, but that is what it looks you are aiming for.

Alternatively, you can go for the easiest solution: use your code to download the data, write it as csv files and use the import custom data feature.

  • You have also mentioned Alpaca's web API. That is what I was assuming in your first question.

In the CustomDataBitcoinAlgorithm, we use Bitstamp's web API and probably you can do the same with Alpaca's.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thanks for taking the time to explain this to me Alexandre. 

0

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed