QuantConnect Community

Universe Warmup Feature/Workaround

Back

Hi,

I am using an indicator which depends on the universe selection. The indicator has a length of 15, thus needs 15x universe selections to be ready and warmed up.

For backtesting, thats not a big deal. Trading can be started after the indicator is warmed up. 

But for live trading the indacator takes 15x universe selections, in my case 15 months. 

Is there a smarter way to warmup the indicator, than manually pre-filling the historical data?

Maybe some implementation idea or workaround?

Thx.

Update Backtest



5 Comments
 
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Hi Eugene,

Possible you could use the Object Store to perform the warm-up in a backtest, save the result to the object store, and then in live trading pull down the value from the object store? This is a relatively new technology so please work with us to report issues you have with it. 

Best
Jared

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Hi Jared,

I ll try out and post my experience here. Thx for the hint! 

Cu

Eugene

0

Hello Jared,

the object store function works just fine. See my code below.

Is there a possibility to see all used keys and used storage space for these keys? I kind of simple explorer...

Thx.

Cu Eugene

 

loading of indicator values during initialization:

public override void Initialize()
{

....

if (LiveMode)
{
Debug("Trading Live!");

if (ObjectStore.ContainsKey(natrKey))
{
natrValues = ObjectStore.ReadJson<decimal[]>(natrKey);
Debug("NATR initialized");
}
else
{
throw new Exception("No NATR values available");
}

if (ObjectStore.ContainsKey(timeKey))
{
timeValues = ObjectStore.ReadJson<DateTime[]>(timeKey);
Debug("Time initialized");
}
else
{
throw new Exception("No Time values available");
}

Debug("NATR History Initialized");
for(int i=0; i <= natrValues.Length-1; i++)
{
Debug("Time: "+timeValues[i]+" NATR: "+natrValues[i].ToString("0.000"));
}

}
}

 

saving of indicator values after calculation, in my case on security change:

public override void OnSecuritiesChanged(SecurityChanges changes)
{
...

// calculation of natrValues

...
ObjectStore.SaveJson(natrKey, natrValues);
ObjectStore.SaveJson(timeKey, timeValues);

}
0

Hi Eugene,

By running the following snippet, we are able to construct a dictionary which holds the ObjectStore keys as its keys and the corresponding size of the saved object (as returned from the Read method) as its values.

objectStoreSizes = {}
for _, j in enumerate(self.ObjectStore.GetEnumerator()):
key = str(j).split(",")[0][1:]
size = sys.getsizeof(self.ObjectStore.Read(key))
objectStoreSizes[key] = size

Note that this technique requires that the keys don't include any commas in their names.

See the attached backtest for a full working example. It is simply an extension of the ObjectStoreExampleAlgorithm.

Best,
Derek Melchin

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Srry for bumping an old thread, but I'm attempting to do the same thing as Eugene and looking at the examples above. 

Still struggling to understand how to adapt this to a universe selection algorithm in python. Mainly with how to read and write the fundamental data in a sensible data structure that can be loaded into a rollingwindow between deployments. For example, say I were working on building the accruals anomaly example from the strategy library into an Alpha Model, at the OnSecuritiesChanged step (or Universe selection for a classic framework algo) below. 

I'm rebalancing/refershing the universe quarterly, where I'll take each active security and update its symbolData object (or create one if the security is new and it wasn't found). 

The symbolData object contains rolling windows for a number of fundamental values (balance sheet current assets, liabilities etc...). So in the ObjecStore object, each security would have to have its own time series of fundamental values, for multiple values. How should we go about storing, retreiving and feeding this data in?

(really could use that historical fundamentals method in backtesting environment right about now :))

class AccrualAnomaly:

def __init__(self, *args, **kwargs):
'''Initializes a new default instance of the HistoricalReturnsAlphaModel class.
Args:
lookback(int): Historical return lookback period
resolution: The resolution of historical data'''
self.lookback = kwargs['lookback'] if 'lookback' in kwargs else 1
self.resolution = kwargs['resolution'] if 'resolution' in kwargs else Resolution.Daily
self.fine_count = kwargs['fine_count'] if 'fine_count' in kwargs else 20
self.nextRebalance = datetime.min
#self.predictionInterval = Time.Multiply(Extensions.ToTimeSpan(self.resolution), self.lookback)
self.symbolData = {}

self.longs = []
self.shorts = []
self.insightDuration = timedelta(weeks=14)
self.universeRefreshed = False

def Update(self, algorithm, data):
'''Updates this alpha model with the latest data from the algorithm.
This is called each time the algorithm receives data for subscribed securities
Args:
algorithm: The algorithm instance
data: The new data available
Returns:
The new insights generated'''

if not self.universeRefreshed: return []
self.universeRefreshed = False

insights = []

## We will liquidate any securities we're still invested in that we don't want to hold a position
## for the next month
for kvp in algorithm.Portfolio:
holding = kvp.Value
symbol = holding.Symbol
if holding.Invested and symbol not in self.longs and symbol not in self.shorts:
insights.append(Insight(symbol, self.insightDuration, InsightType.Price, InsightDirection.Flat, None, None))

## Emit Up (buy) Insights for our desired long positions
for symbol in self.longs:
insights.append(Insight(symbol, self.insightDuration, InsightType.Price, InsightDirection.Up, 0.01, None))

#for symbol in self.shorts:factor
#insights.append(Insight(symbol, self.insightDuration, InsightType.Price, InsightDirection.Down, 0.01, None))

return insights

def OnSecuritiesChanged(self, algorithm, changes):
if algorithm.Time < self.nextRebalance:
self.universeRefreshed = False
return
self.universeRefreshed = True
self.nextRebalance = Expiry.EndOfQuarter(algorithm.Time) # Options: Expiry.EndOfDay, Expiry.EndOfWeek, Expiry.EndOfMonth, Expiry.EndOfQuarter, Expiry.EndOfYear

added = [ x for x in algorithm.ActiveSecurities if x.Value.Fundamentals is not None ]

for security in added:
symbol = security.Value.Symbol
if symbol in self.symbolData.keys():
self.symbolData[symbol].storeFactors(security.Value)
else:
self.symbolData[symbol] = symbolData(symbol,4)

# grab symbols that are ready
ready = [ i for i in added if self.symbolData[i.Value.Symbol].is_ready ]

sortedByAccruals= sorted(ready, key=lambda x: self.symbolData[x.Value.Symbol].totalAccruals, reverse=True)

# etc..................

# take longs and shorts
self.longs = [x[0].Value.Symbol for x in sorted_stock][:self.fine_count]
#self.shorts = [x[0].Value.Symbol for x in sorted_stock][-self.fine_count:]

algorithm.Log(f"GoodFactors LONGS: {[x.Value for x in self.longs]}")

# clean up data for removed securities
for removed in changes.RemovedSecurities:
if removed.Symbol in self.longs:
self.longs.remove(removed.Symbol)


class symbolData(object):
def __init__(self, symbol,period):
self.symbol = symbol
self.is_ready = False
length = period + 1
self.assetWin = RollingWindow[float](length)
self.cashWin = RollingWindow[float](length)
self.liabilityWin = RollingWindow[float](length)
self.debtWin = RollingWindow[float](length)
self.taxWin = RollingWindow[float](length)
self.DandAWin = RollingWindow[float](length)
self.avgSizeWin = RollingWindow[float](length)
self.totalAccruals = 0.0

def storeFactors(self,fine):
self.addFactorsToWindows(fine)

self.is_ready = self.windowsReady()

if self.is_ready:
self.calculateWindowValues()

def windowsReady(self):
return self.assetWin.IsReady and self.cashWin.IsReady and self.liabilityWin.IsReady and self.debtWin.IsReady and self.taxWin.IsReady \
and self.DandAWin.IsReady and self.avgSizeWin.IsReady

def addFactorsToWindows(self,fine):
# take symbol in fine universe obejct as input, store factors into rolling windows
if fine.Fundamentals.FinancialStatements.BalanceSheet.CurrentAssets.Value != 0 : self.assetWin.Add(fine.Fundamentals.FinancialStatements.BalanceSheet.CurrentAssets.Value)
if fine.Fundamentals.FinancialStatements.BalanceSheet.CashAndCashEquivalents.Value != 0 : self.cashWin.Add(fine.Fundamentals.FinancialStatements.BalanceSheet.CashAndCashEquivalents.Value)
if fine.Fundamentals.FinancialStatements.BalanceSheet.CurrentLiabilities.Value != 0 : self.liabilityWin.Add(fine.Fundamentals.FinancialStatements.BalanceSheet.CurrentLiabilities.Value)
if fine.Fundamentals.FinancialStatements.BalanceSheet.CurrentDebt.Value != 0 : self.debtWin.Add(fine.Fundamentals.FinancialStatements.BalanceSheet.CurrentDebt.Value)
if fine.Fundamentals.FinancialStatements.BalanceSheet.IncomeTaxPayable.Value != 0 : self.taxWin.Add(fine.Fundamentals.FinancialStatements.BalanceSheet.IncomeTaxPayable.Value)
if fine.Fundamentals.FinancialStatements.IncomeStatement.DepreciationAndAmortization.Value != 0 : self.DandAWin.Add(fine.Fundamentals.FinancialStatements.IncomeStatement.DepreciationAndAmortization.Value)
if fine.Fundamentals.FinancialStatements.BalanceSheet.TotalAssets.Value != 0 : self.avgSizeWin.Add(fine.Fundamentals.FinancialStatements.BalanceSheet.TotalAssets.Value)

def calculateWindowValues(self):
#calculates the balance sheet accruals
delta_assets = self.assetWin[0] / self.assetWin[ self.assetWin.Count -1 ]
delta_cash = self.cashWin[0] / self.cashWin[ self.cashWin.Count -1 ]
delta_liabilities = self.liabilityWin[0] / self.liabilityWin[ self.liabilityWin.Count -1 ]
delta_debt = self.debtWin[0] / self.debtWin[ self.debtWin.Count -1 ]
delta_tax = self.taxWin[0] / self.taxWin[ self.taxWin.Count -1 ]
dep = sum(list(self.DandAWin)[:3]) # annual depreciation from q'ly D&A IS numbers
avg_total = (self.avgSizeWin[0] + self.avgSizeWin[ self.avgSizeWin.Count -1 ]) / 2 #accounts for the size difference

self.totalAccruals = ((delta_assets-delta_cash)-(delta_liabilities-delta_debt-delta_tax)-dep)/avg_total


 

0
Update Backtest



0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Loading...

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!

Discussion Tags