Share New Research 
Hi everyone! I am quite new to quantconnect and have only done some simple research with pandas dataframe before. I would appreciate if any of you can recommend a way to get future bid ask data for building regression model that generates trading signals.
I am trying get the historical bidclose, bidsize, askclose, asksize, close, and open with the self.history method. I am taking reference from this doc
https://www.quantconnect.com/docs/algorithm-reference/historical-dataHowever, when I run the code, it shows an error message. May I ask are bid ask data not availlable? How can I find out what kind of columns can i get with this self.history method?
I have looked at a lot of examples, but did not find an example that how to get a list of historical bid ask data of future. I would appreciate if any of you can give me some recommendations.
import clr
clr.AddReference("System")
clr.AddReference("QuantConnect.Algorithm")
clr.AddReference("QuantConnect.Common")
from System import *
from QuantConnect import *
from QuantConnect.Algorithm import *
import datetime
from datetime import timedelta
import numpy as np
from sklearn.linear_model import LinearRegression
import pandas as pd
import statsmodels.api as sm
class ScikitLearnLinearRegressionAlgorithm(QCAlgorithm):
def Initialize(self):
self.SetStartDate(2021, 2, 6) # Set Start Date # BTC Future Start Date 2009, 1, 1
self.SetEndDate(2021, 3, 6) # Set End Date
self.lookback = 30*24*60 # number of previous days for training # one month # regression
self.testing = 10 #testing period table
self.SetCash(10000) # Set Strategy Cash
#1. Request BTC futures and save the BTC security
self.BTC = self.AddFuture(Futures.Currencies.BTC, Resolution.Minute)
#2. Set our expiry filter to return all contracts expiring within 35 days
#self.BTC.SetFilter(0, 35)
lambda x: x.FrontMonth()
self.BTC.SetFilter(lambda x: x.FrontMonth())
self.Schedule.On(self.DateRules.WeekEnd(),self.TimeRules.At(23, 59) ,self.Regression)
#self.exchange = self.Securities[self.BTC.Symbol].Exchange
# if exchange.ExchangeOpen:
self.Schedule.On(self.DateRules.EveryDay(self.BTC.Symbol), self.TimeRules.Every(timedelta(minutes=1)), self.Trade)
self.run = 0
def Regression(self):
# Historical data is used to train the machine learning model
history = self.History([self.BTC.Symbol], self.lookback, Resolution.Minute)
bidclose = list(history['bidclose'])
bidsize = list(history['bidsize'])
askclose = list(history['askclose'])
asksize = list(history['asksize'])
openprice = list(history['open'])
close = list(history['close'])
volume = list(history['volume'])
df = pd.DataFrame({"bidclose":bidclose, "bidsize":bidsize, "askclose":askclose, "asksize":asksize, "open":openprice, "close":close,"volume":volume})
df['bidpricechange_lag']=df['bidclose']-df['bidclose'].shift(1)
df['askpricechange_lag']=df['askclose']-df['askclose'].shift(1)
df['bidsizediff_lag']=df['bidsize']-df['bidsize'].shift(1)
df['asksizediff_lag']=df['asksize']-df['asksize'].shift(1)
df=df.dropna(axis=0)
deltaVolumeBid=[]
for i in df.index:
if df.loc[i,'bidpricechange_lag'] > 0:
deltaVolumeBid.append(df.loc[i,'bidsize'])
elif df.loc[i,'bidpricechange_lag'] < 0:
deltaVolumeBid.append(0)
else:
deltaVolumeBid.append(df.loc[i,'bidsizediff_lag'])
df['deltaVolumeBid']=deltaVolumeBid
deltaVolumeAsk=[]
for j in df.index:
if df.loc[j,'askpricechange_lag'] > 0:
deltaVolumeAsk.append(0)
elif df.loc[j,'askpricechange_lag'] < 0:
deltaVolumeAsk.append(df.loc[j,'asksize'])
else:
deltaVolumeAsk.append(df.loc[j,'asksizediff_lag'])
df['deltaVolumeAsk']=deltaVolumeAsk
df['Return']=df['close'].shift(-1)/df['open'].shift(-1)-1 #open # default trading open?
df['VOI']=df['deltaVolumeBid']-df['deltaVolumeAsk']
df['OIR']=(df['bidsize']-df['asksize'])/(df['bidsize']+df['asksize'])
df['SP']=df['askclose']-df['bidclose']#0?
df['VOI_SP']=(df['deltaVolumeBid']-df['deltaVolumeAsk'])/df['SP']
df['OIR_SP']=((df['bidsize']-df['asksize'])/(df['bidsize']+df['asksize']))/df['SP']
df['VOI_SP_lag1']=df['VOI_SP'].shift(1)
df['VOI_SP_lag2']=df['VOI_SP'].shift(2)
df['VOI_SP_lag3']=df['VOI_SP'].shift(3)
df['VOI_SP_lag4']=df['VOI_SP'].shift(4)
df['VOI_SP_lag5']=df['VOI_SP'].shift(5)
df['OIR_SP_lag1']=df['OIR_SP'].shift(1)
df['OIR_SP_lag2']=df['OIR_SP'].shift(2)
df['OIR_SP_lag3']=df['OIR_SP'].shift(3)
df['OIR_SP_lag4']=df['OIR_SP'].shift(4)
df['OIR_SP_lag5']=df['OIR_SP'].shift(5)
df=df.dropna(axis=0)
Model=smf.ols(formula='Return~VOI_SP + VOI_SP + VOI_SP_lag1 + VOI_SP_lag2 + VOI_SP_lag3 + VOI_SP_lag4 + VOI_SP_lag5 + OIR_SP + OIR_SP_lag1 + OIR_SP_lag2 + OIR_SP_lag3 + OIR_SP_lag4 + OIR_SP_lag5', data=train).fit() #data = df
df['yhat']=model.predict(df)
self.long= df['yhat'].quantile(90)
self.closelong = df['yhat'].quantile(75)
self.long = df['yhat'].quantile(10)
self.closeshort = df['yhat'].quantile(25)
self.model=Model
self.run=1
def Trade(self):
if self.run == 0:
self.Regression
# Historical data is used to train the machine learning model
history = self.History([self.BTC.Symbol], self.lookback, Resolution.Minute)
bidclose = list(history['bidclose'])
bidsize = list(history['bidsize'])
askclose = list(history['askclose'])
asksize = list(history['asksize'])
openprice = list(history['open'])
close = list(history['close'])
volume = list(history['volume'])
df = pd.DataFrame({"bidclose":bidclose, "bidsize":bidsize, "askclose":askclose, "asksize":asksize, "open":openprice, "close":close,"volume":volume})
df['bidpricechange_lag']=df['bidclose']-df['bidclose'].shift(1)
df['askpricechange_lag']=df['askclose']-df['askclose'].shift(1)
df['bidsizediff_lag']=df['bidsize']-df['bidsize'].shift(1)
df['asksizediff_lag']=df['asksize']-df['asksize'].shift(1)
df=df.dropna(axis=0)
deltaVolumeBid=[]
for i in df.index:
if df.loc[i,'bidpricechange_lag'] > 0:
deltaVolumeBid.append(df.loc[i,'bidsize'])
elif df.loc[i,'bidpricechange_lag'] < 0:
deltaVolumeBid.append(0)
else:
deltaVolumeBid.append(df.loc[i,'bidsizediff_lag'])
df['deltaVolumeBid']=deltaVolumeBid
deltaVolumeAsk=[]
for j in df.index:
if df.loc[j,'askpricechange_lag'] > 0:
deltaVolumeAsk.append(0)
elif df.loc[j,'askpricechange_lag'] < 0:
deltaVolumeAsk.append(df.loc[j,'asksize'])
else:
deltaVolumeAsk.append(df.loc[j,'asksizediff_lag'])
df['deltaVolumeAsk']=deltaVolumeAsk
df['Return']=df['close'].shift(-1)/df['open'].shift(-1)-1 #open # default trading open?
df['VOI']=df['deltaVolumeBid']-df['deltaVolumeAsk']
df['OIR']=(df['bidsize']-df['asksize'])/(df['bidsize']+df['asksize'])
df['SP']=df['askclose']-df['bidclose']#0?
df['VOI_SP']=(df['deltaVolumeBid']-df['deltaVolumeAsk'])/df['SP']
df['OIR_SP']=((df['bidsize']-df['asksize'])/(df['bidsize']+df['asksize']))/df['SP']
df['VOI_SP_lag1']=df['VOI_SP'].shift(1)
df['VOI_SP_lag2']=df['VOI_SP'].shift(2)
df['VOI_SP_lag3']=df['VOI_SP'].shift(3)
df['VOI_SP_lag4']=df['VOI_SP'].shift(4)
df['VOI_SP_lag5']=df['VOI_SP'].shift(5)
df['OIR_SP_lag1']=df['OIR_SP'].shift(1)
df['OIR_SP_lag2']=df['OIR_SP'].shift(2)
df['OIR_SP_lag3']=df['OIR_SP'].shift(3)
df['OIR_SP_lag4']=df['OIR_SP'].shift(4)
df['OIR_SP_lag5']=df['OIR_SP'].shift(5)
df=df.dropna(axis=0)
df['yhat']=model.predict(df)
predictedReturn = df.iloc[-1]['yhat']
if predictedReturn > 0:
self.SetHoldings(self.BTC.Symbol, 1)
else:
self.SetHoldings(self.BTC.Symbol, -1)
Tin Yat Chau
import clr clr.AddReference("System") clr.AddReference("QuantConnect.Algorithm") clr.AddReference("QuantConnect.Common") from System import * from QuantConnect import * from QuantConnect.Algorithm import * import datetime from datetime import timedelta import numpy as np from sklearn.linear_model import LinearRegression import pandas as pd import statsmodels.api as sm class ScikitLearnLinearRegressionAlgorithm(QCAlgorithm): def Initialize(self): self.SetStartDate(2021, 2, 6) # Set Start Date # BTC Future Start Date 2009, 1, 1 self.SetEndDate(2021, 3, 6) # Set End Date self.lookback = 30*24*60 # number of previous days for training # one month # regression self.testing = 10 #testing period table self.SetCash(10000) # Set Strategy Cash #1. Request BTC futures and save the BTC security self.BTC = self.AddFuture(Futures.Currencies.BTC, Resolution.Minute) #2. Set our expiry filter to return all contracts expiring within 35 days #self.BTC.SetFilter(0, 35) lambda x: x.FrontMonth() self.BTC.SetFilter(lambda x: x.FrontMonth()) self.Schedule.On(self.DateRules.WeekEnd(),self.TimeRules.At(23, 59) ,self.Regression) #self.exchange = self.Securities[self.BTC.Symbol].Exchange # if exchange.ExchangeOpen: self.Schedule.On(self.DateRules.EveryDay(self.BTC.Symbol), self.TimeRules.Every(timedelta(minutes=1)), self.Trade) self.run = 0 def Regression(self): # Historical data is used to train the machine learning model history = self.History([self.BTC.Symbol], self.lookback, Resolution.Minute) bidclose = list(history['bidclose']) bidsize = list(history['bidsize']) askclose = list(history['askclose']) asksize = list(history['asksize']) openprice = list(history['open']) close = list(history['close']) volume = list(history['volume']) df = pd.DataFrame({"bidclose":bidclose, "bidsize":bidsize, "askclose":askclose, "asksize":asksize, "open":openprice, "close":close,"volume":volume}) df['bidpricechange_lag']=df['bidclose']-df['bidclose'].shift(1) df['askpricechange_lag']=df['askclose']-df['askclose'].shift(1) df['bidsizediff_lag']=df['bidsize']-df['bidsize'].shift(1) df['asksizediff_lag']=df['asksize']-df['asksize'].shift(1) df=df.dropna(axis=0) deltaVolumeBid=[] for i in df.index: if df.loc[i,'bidpricechange_lag'] > 0: deltaVolumeBid.append(df.loc[i,'bidsize']) elif df.loc[i,'bidpricechange_lag'] < 0: deltaVolumeBid.append(0) else: deltaVolumeBid.append(df.loc[i,'bidsizediff_lag']) df['deltaVolumeBid']=deltaVolumeBid deltaVolumeAsk=[] for j in df.index: if df.loc[j,'askpricechange_lag'] > 0: deltaVolumeAsk.append(0) elif df.loc[j,'askpricechange_lag'] < 0: deltaVolumeAsk.append(df.loc[j,'asksize']) else: deltaVolumeAsk.append(df.loc[j,'asksizediff_lag']) df['deltaVolumeAsk']=deltaVolumeAsk df['Return']=df['close'].shift(-1)/df['open'].shift(-1)-1 #open # default trading open? df['VOI']=df['deltaVolumeBid']-df['deltaVolumeAsk'] df['OIR']=(df['bidsize']-df['asksize'])/(df['bidsize']+df['asksize']) df['SP']=df['askclose']-df['bidclose']#0? df['VOI_SP']=(df['deltaVolumeBid']-df['deltaVolumeAsk'])/df['SP'] df['OIR_SP']=((df['bidsize']-df['asksize'])/(df['bidsize']+df['asksize']))/df['SP'] df['VOI_SP_lag1']=df['VOI_SP'].shift(1) df['VOI_SP_lag2']=df['VOI_SP'].shift(2) df['VOI_SP_lag3']=df['VOI_SP'].shift(3) df['VOI_SP_lag4']=df['VOI_SP'].shift(4) df['VOI_SP_lag5']=df['VOI_SP'].shift(5) df['OIR_SP_lag1']=df['OIR_SP'].shift(1) df['OIR_SP_lag2']=df['OIR_SP'].shift(2) df['OIR_SP_lag3']=df['OIR_SP'].shift(3) df['OIR_SP_lag4']=df['OIR_SP'].shift(4) df['OIR_SP_lag5']=df['OIR_SP'].shift(5) df=df.dropna(axis=0) Model=smf.ols(formula='Return~VOI_SP + VOI_SP + VOI_SP_lag1 + VOI_SP_lag2 + VOI_SP_lag3 + VOI_SP_lag4 + VOI_SP_lag5 + OIR_SP + OIR_SP_lag1 + OIR_SP_lag2 + OIR_SP_lag3 + OIR_SP_lag4 + OIR_SP_lag5', data=train).fit() #data = df df['yhat']=model.predict(df) self.long= df['yhat'].quantile(90) self.closelong = df['yhat'].quantile(75) self.long = df['yhat'].quantile(10) self.closeshort = df['yhat'].quantile(25) self.model=Model self.run=1 def Trade(self): if self.run == 0: self.Regression # Historical data is used to train the machine learning model history = self.History([self.BTC.Symbol], self.lookback, Resolution.Minute) bidclose = list(history['bidclose']) bidsize = list(history['bidsize']) askclose = list(history['askclose']) asksize = list(history['asksize']) openprice = list(history['open']) close = list(history['close']) volume = list(history['volume']) df = pd.DataFrame({"bidclose":bidclose, "bidsize":bidsize, "askclose":askclose, "asksize":asksize, "open":openprice, "close":close,"volume":volume}) df['bidpricechange_lag']=df['bidclose']-df['bidclose'].shift(1) df['askpricechange_lag']=df['askclose']-df['askclose'].shift(1) df['bidsizediff_lag']=df['bidsize']-df['bidsize'].shift(1) df['asksizediff_lag']=df['asksize']-df['asksize'].shift(1) df=df.dropna(axis=0) deltaVolumeBid=[] for i in df.index: if df.loc[i,'bidpricechange_lag'] > 0: deltaVolumeBid.append(df.loc[i,'bidsize']) elif df.loc[i,'bidpricechange_lag'] < 0: deltaVolumeBid.append(0) else: deltaVolumeBid.append(df.loc[i,'bidsizediff_lag']) df['deltaVolumeBid']=deltaVolumeBid deltaVolumeAsk=[] for j in df.index: if df.loc[j,'askpricechange_lag'] > 0: deltaVolumeAsk.append(0) elif df.loc[j,'askpricechange_lag'] < 0: deltaVolumeAsk.append(df.loc[j,'asksize']) else: deltaVolumeAsk.append(df.loc[j,'asksizediff_lag']) df['deltaVolumeAsk']=deltaVolumeAsk df['Return']=df['close'].shift(-1)/df['open'].shift(-1)-1 #open # default trading open? df['VOI']=df['deltaVolumeBid']-df['deltaVolumeAsk'] df['OIR']=(df['bidsize']-df['asksize'])/(df['bidsize']+df['asksize']) df['SP']=df['askclose']-df['bidclose']#0? df['VOI_SP']=(df['deltaVolumeBid']-df['deltaVolumeAsk'])/df['SP'] df['OIR_SP']=((df['bidsize']-df['asksize'])/(df['bidsize']+df['asksize']))/df['SP'] df['VOI_SP_lag1']=df['VOI_SP'].shift(1) df['VOI_SP_lag2']=df['VOI_SP'].shift(2) df['VOI_SP_lag3']=df['VOI_SP'].shift(3) df['VOI_SP_lag4']=df['VOI_SP'].shift(4) df['VOI_SP_lag5']=df['VOI_SP'].shift(5) df['OIR_SP_lag1']=df['OIR_SP'].shift(1) df['OIR_SP_lag2']=df['OIR_SP'].shift(2) df['OIR_SP_lag3']=df['OIR_SP'].shift(3) df['OIR_SP_lag4']=df['OIR_SP'].shift(4) df['OIR_SP_lag5']=df['OIR_SP'].shift(5) df=df.dropna(axis=0) df['yhat']=model.predict(df) predictedReturn = df.iloc[-1]['yhat'] if predictedReturn > 0: self.SetHoldings(self.BTC.Symbol, 1) else: self.SetHoldings(self.BTC.Symbol, -1)Derek Melchin
Hi Tin Yat Chau,
The algorithm above provides the canonical futures symbol to the History method. To have data returned, we must provide the symbol of a futures contract.
See the attached backtest below.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Tin Yat Chau
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!