Back

Runtime Error: KeyNotFoundException

Hi guys, 

I'm getting an error when backtesting that is unexpected because I have tried to make the algo resillient to this exact scenario.

in the initialization phase set the following:

self.tickers = ['ESS','STOR']

in the OnData phase of the algo I start with

if False in [data.ContainsKey(x) for x in self.tickers]: return

and i have also tried 

if not all([data.ContainsKey(x) for x in self.tickers]): return

in both instances I get an KeyError (full error message at the bottom on 26 september 2019 for ticker 'ESS' (I have had the same error for other tickers on other dates). This specific example is with minute resolution data.

I would expect that if the ticker data is not present the algo would just skip this data point and continue with the next datapoint. 

anyone running into something similar or am I missing an elephant in the room?

 

Best,

 

Dirk

 

Runtime Error: KeyNotFoundException : 'ESS' wasn't found in the TradeBars object, likely because there was no-data at this moment in time and it wasn't possible to fillforward historical data. Please check the data exists before accessing it with data.ContainsKey("ESS")
at QuantConnect.Data.Market.DataDictionary`1[T].get_Item (QuantConnect.Symbol symbol) [0x00024] in <8e85730333624c7d90e25a24998a3d52>:0
at QuantConnect.Data.Market.TradeBars.get_Item (QuantConnect.Symbol symbol) [0x00000] in <8e85730333624c7d90e25a24998a3d52>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in <b0e1ad7573a24fd5a9f2af9595e677e7>:0
at OnData in main.py:line 35
:: self.pair.updateMetrics(data)
at updateMetrics in main.py:line 96
KeyNotFoundException : 'ESS' wasn't found in the TradeBars object, likely because there was no-data at this moment in time and it wasn't possible to fillforward historical data. Please check the data exists before accessing it with data.ContainsKey("ESS")
at QuantConnect.Data.Market.DataDictionary`1[T].get_Item (QuantConnect.Symbol symbol) [0x00024] in <8e85730333624c7d90e25a24998a3d52>:0
at QuantConnect.Data.Market.TradeBars.get_Item (QuantConnect.Symbol symbol) [0x00000] in <8e85730333624c7d90e25a24998a3d52>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in <b0e1ad7573a24fd5a9f2af9595e677e7>:0 (Open Stacktrace)

 

Update Backtest







0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hi dirk bothof ,

Could you please share a backest that we can use to reproduce the issue?

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


I do not believe it is possible to attach a failed backtest, so I have attached the algorithm below, copy paste should work :) (yes, this will lose money pretty fast ;) 

import pandas as pd
import numpy as np

class Trader(QCAlgorithm):

def Initialize(self):
self.SetBrokerageModel(BrokerageName.InteractiveBrokersBrokerage)
self.SetStartDate(2019,8,6) # Set Start Date
#self.SetEndDate(2019,9,10)
self.SetCash(50000) # Set Strategy Cash
self.tickers = ['ESS','STOR']
self.enter_threshold = 2
self.exit_threshold = -self.enter_threshold
self.enter_size = 0.5

self.pair = Pair(self.AddSecurity(SecurityType.Equity, self.tickers[0], Resolution.Minute).Symbol,
self.AddSecurity(SecurityType.Equity, self.tickers[1], Resolution.Minute).Symbol
)

#setting custom slippage model
for ticker in self.tickers:
self.Securities[ticker].SetSlippageModel(CustomSlippageModel(self))

self.SetWarmUp(self.pair.lookback)

def OnData(self, data):
'''OnData event is the primary entry point for your algorithm. Each new data point will be pumped in here.
-------
data: Slice object keyed by symbol containing the stock data
'''
if False in [data.ContainsKey(x) for x in self.tickers]: return

#updating the pairs
self.pair.updateMetrics(data)

if self.IsWarmingUp: return #no trading during warm up period

#potentially buying stock if we don't hold any
if self.pair.position == 0:
if self.pair.zScore < -self.enter_threshold: #check if we need to go long or short on the spread
self.SetHoldings(self.pair.a, self.enter_size)
self.SetHoldings(self.pair.b, -self.enter_size)
self.pair.position += -1
self.Log(self.pair.distanceRatio)

elif self.pair.zScore > self.enter_threshold:
self.SetHoldings(self.pair.a, -self.enter_size)
self.SetHoldings(self.pair.b, self.enter_size)
self.pair.position += 1
self.Log(self.pair.distanceRatio)
else:
pass

#potentially selling stocks if we already have position
elif (self.pair.position == -1 and self.pair.zScore > self.exit_threshold) or (self.pair.position == 1 and self.pair.zScore < -self.exit_threshold):
self.Liquidate(self.pair.a)
self.Liquidate(self.pair.b)
self.pair.position = 0
self.Log(self.pair.distanceRatio)

#no trading oppertunities found pass
else:
pass

class Pair(object):

def __init__(self,ticker_a,ticker_b):
'''initializes the algoritme, assigns variables, and loads a bit of historical data
so the algo is warmed up
---
ticker_a: ticker of the first stock of the pair
ticker_b: ticler of the second stock of the pair
'''
self.a = ticker_a
self.b = ticker_b
self.position = 0
self.lookback = 8*330
#initiating trading signal
self.price_a = np.array(0)
self.price_b = np.array(0)
self.ratio = np.array(0)
self.sma = 0
self.meanDistance = 0
self.distanceRatio = 0
self.zScore = 0


def updateMetrics(self,data):
'''Updates the metrics of the inserted pair with the data of the Slice Object
Keeps memory use small, and uses lists for all opperations (df or series would not work since there is not vector caluclations)
data: data object in the quantconnect model
'''

#making/updating metrics
self.price_a = np.append(self.price_a, (data.Bars[self.a]).Close)
self.price_b = np.append(self.price_b, (data.Bars[self.b]).Close)
self.ratio = np.append(self.ratio, self.price_a[-1]/self.price_b[-1])

if self.price_a.shape[0] > self.lookback:
#if the data list is longer than lookback the list is trimmed to not impact 'rolling' stats
self.price_a = self.price_a[1:]
self.price_b = self.price_b[1:]
self.ratio = self.ratio[1:]


if self.ratio.shape[0] >= self.lookback:
self.std = np.std(self.ratio)
self.sma = np.mean(self.ratio)
self.zScore = (self.ratio[-1] - self.sma) / self.std

class CustomSlippageModel:
def __init__(self, algorithm):
self.algorithm = algorithm

def GetSlippageApproximation(self, asset, order):
# custom slippage math
slippage = asset.Price * 0.0001 * np.log10(2*float(order.AbsoluteQuantity))
return slippage

 

0

Hi dirk bothof ,

Please use

if not all([data.Bars.ContainsKey(symbol) for symbol in self.tickers]):
return

since the algorithm is trying to update the state of the Pair object using data.Bars:

self.price_a = np.append(self.price_a, (data.Bars[self.a]).Close)
self.price_b = np.append(self.price_b, (data.Bars[self.b]).Close)

A given Symbol can be in data.Keys but not in data.Bars.Keys if there is no TradeBar data.
In this case, there was only dividend data when it tried to access TradeBar data.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed