QUANTCONNECT COMMUNITY
Hi,
I am using an indicator which depends on the universe selection. The indicator has a length of 15, thus needs 15x universe selections to be ready and warmed up.
For backtesting, thats not a big deal. Trading can be started after the indicator is warmed up.
But for live trading the indacator takes 15x universe selections, in my case 15 months.
Is there a smarter way to warmup the indicator, than manually pre-filling the historical data?
Maybe some implementation idea or workaround?
Thx.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
QuantConnect™ 2024. All Rights Reserved
|
|
Share New Research 
Jared Broad
Hi Eugene,
Possible you could use the Object Store to perform the warm-up in a backtest, save the result to the object store, and then in live trading pull down the value from the object store? This is a relatively new technology so please work with us to report issues you have with it.
Best
Jared
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Eugene
Hi Jared,
I ll try out and post my experience here. Thx for the hint!
Cu
Eugene
Eugene
Hello Jared,
the object store function works just fine. See my code below.
Is there a possibility to see all used keys and used storage space for these keys? I kind of simple explorer...
Thx.
Cu Eugene
loading of indicator values during initialization:
public override void Initialize() { .... if (LiveMode) { Debug("Trading Live!"); if (ObjectStore.ContainsKey(natrKey)) { natrValues = ObjectStore.ReadJson<decimal[]>(natrKey); Debug("NATR initialized"); } else { throw new Exception("No NATR values available"); } if (ObjectStore.ContainsKey(timeKey)) { timeValues = ObjectStore.ReadJson<DateTime[]>(timeKey); Debug("Time initialized"); } else { throw new Exception("No Time values available"); } Debug("NATR History Initialized"); for(int i=0; i <= natrValues.Length-1; i++) { Debug("Time: "+timeValues[i]+" NATR: "+natrValues[i].ToString("0.000")); } } }saving of indicator values after calculation, in my case on security change:
public override void OnSecuritiesChanged(SecurityChanges changes) { ... // calculation of natrValues ... ObjectStore.SaveJson(natrKey, natrValues); ObjectStore.SaveJson(timeKey, timeValues); }Derek Melchin
Hi Eugene,
By running the following snippet, we are able to construct a dictionary which holds the ObjectStore keys as its keys and the corresponding size of the saved object (as returned from the Read method) as its values.
objectStoreSizes = {} for _, j in enumerate(self.ObjectStore.GetEnumerator()): key = str(j).split(",")[0][1:] size = sys.getsizeof(self.ObjectStore.Read(key)) objectStoreSizes[key] = sizeNote that this technique requires that the keys don't include any commas in their names.
See the attached backtest for a full working example. It is simply an extension of the ObjectStoreExampleAlgorithm.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
P Chen
Srry for bumping an old thread, but I'm attempting to do the same thing as Eugene and looking at the examples above.
Still struggling to understand how to adapt this to a universe selection algorithm in python. Mainly with how to read and write the fundamental data in a sensible data structure that can be loaded into a rollingwindow between deployments. For example, say I were working on building the accruals anomaly example from the strategy library into an Alpha Model, at the OnSecuritiesChanged step (or Universe selection for a classic framework algo) below.
I'm rebalancing/refershing the universe quarterly, where I'll take each active security and update its symbolData object (or create one if the security is new and it wasn't found).
The symbolData object contains rolling windows for a number of fundamental values (balance sheet current assets, liabilities etc...). So in the ObjecStore object, each security would have to have its own time series of fundamental values, for multiple values. How should we go about storing, retreiving and feeding this data in?
(really could use that historical fundamentals method in backtesting environment right about now :))
class AccrualAnomaly: def __init__(self, *args, **kwargs): '''Initializes a new default instance of the HistoricalReturnsAlphaModel class. Args: lookback(int): Historical return lookback period resolution: The resolution of historical data''' self.lookback = kwargs['lookback'] if 'lookback' in kwargs else 1 self.resolution = kwargs['resolution'] if 'resolution' in kwargs else Resolution.Daily self.fine_count = kwargs['fine_count'] if 'fine_count' in kwargs else 20 self.nextRebalance = datetime.min #self.predictionInterval = Time.Multiply(Extensions.ToTimeSpan(self.resolution), self.lookback) self.symbolData = {} self.longs = [] self.shorts = [] self.insightDuration = timedelta(weeks=14) self.universeRefreshed = False def Update(self, algorithm, data): '''Updates this alpha model with the latest data from the algorithm. This is called each time the algorithm receives data for subscribed securities Args: algorithm: The algorithm instance data: The new data available Returns: The new insights generated''' if not self.universeRefreshed: return [] self.universeRefreshed = False insights = [] ## We will liquidate any securities we're still invested in that we don't want to hold a position ## for the next month for kvp in algorithm.Portfolio: holding = kvp.Value symbol = holding.Symbol if holding.Invested and symbol not in self.longs and symbol not in self.shorts: insights.append(Insight(symbol, self.insightDuration, InsightType.Price, InsightDirection.Flat, None, None)) ## Emit Up (buy) Insights for our desired long positions for symbol in self.longs: insights.append(Insight(symbol, self.insightDuration, InsightType.Price, InsightDirection.Up, 0.01, None)) #for symbol in self.shorts:factor #insights.append(Insight(symbol, self.insightDuration, InsightType.Price, InsightDirection.Down, 0.01, None)) return insights def OnSecuritiesChanged(self, algorithm, changes): if algorithm.Time < self.nextRebalance: self.universeRefreshed = False return self.universeRefreshed = True self.nextRebalance = Expiry.EndOfQuarter(algorithm.Time) # Options: Expiry.EndOfDay, Expiry.EndOfWeek, Expiry.EndOfMonth, Expiry.EndOfQuarter, Expiry.EndOfYear added = [ x for x in algorithm.ActiveSecurities if x.Value.Fundamentals is not None ] for security in added: symbol = security.Value.Symbol if symbol in self.symbolData.keys(): self.symbolData[symbol].storeFactors(security.Value) else: self.symbolData[symbol] = symbolData(symbol,4) # grab symbols that are ready ready = [ i for i in added if self.symbolData[i.Value.Symbol].is_ready ] sortedByAccruals= sorted(ready, key=lambda x: self.symbolData[x.Value.Symbol].totalAccruals, reverse=True) # etc.................. # take longs and shorts self.longs = [x[0].Value.Symbol for x in sorted_stock][:self.fine_count] #self.shorts = [x[0].Value.Symbol for x in sorted_stock][-self.fine_count:] algorithm.Log(f"GoodFactors LONGS: {[x.Value for x in self.longs]}") # clean up data for removed securities for removed in changes.RemovedSecurities: if removed.Symbol in self.longs: self.longs.remove(removed.Symbol) class symbolData(object): def __init__(self, symbol,period): self.symbol = symbol self.is_ready = False length = period + 1 self.assetWin = RollingWindow[float](length) self.cashWin = RollingWindow[float](length) self.liabilityWin = RollingWindow[float](length) self.debtWin = RollingWindow[float](length) self.taxWin = RollingWindow[float](length) self.DandAWin = RollingWindow[float](length) self.avgSizeWin = RollingWindow[float](length) self.totalAccruals = 0.0 def storeFactors(self,fine): self.addFactorsToWindows(fine) self.is_ready = self.windowsReady() if self.is_ready: self.calculateWindowValues() def windowsReady(self): return self.assetWin.IsReady and self.cashWin.IsReady and self.liabilityWin.IsReady and self.debtWin.IsReady and self.taxWin.IsReady \ and self.DandAWin.IsReady and self.avgSizeWin.IsReady def addFactorsToWindows(self,fine): # take symbol in fine universe obejct as input, store factors into rolling windows if fine.Fundamentals.FinancialStatements.BalanceSheet.CurrentAssets.Value != 0 : self.assetWin.Add(fine.Fundamentals.FinancialStatements.BalanceSheet.CurrentAssets.Value) if fine.Fundamentals.FinancialStatements.BalanceSheet.CashAndCashEquivalents.Value != 0 : self.cashWin.Add(fine.Fundamentals.FinancialStatements.BalanceSheet.CashAndCashEquivalents.Value) if fine.Fundamentals.FinancialStatements.BalanceSheet.CurrentLiabilities.Value != 0 : self.liabilityWin.Add(fine.Fundamentals.FinancialStatements.BalanceSheet.CurrentLiabilities.Value) if fine.Fundamentals.FinancialStatements.BalanceSheet.CurrentDebt.Value != 0 : self.debtWin.Add(fine.Fundamentals.FinancialStatements.BalanceSheet.CurrentDebt.Value) if fine.Fundamentals.FinancialStatements.BalanceSheet.IncomeTaxPayable.Value != 0 : self.taxWin.Add(fine.Fundamentals.FinancialStatements.BalanceSheet.IncomeTaxPayable.Value) if fine.Fundamentals.FinancialStatements.IncomeStatement.DepreciationAndAmortization.Value != 0 : self.DandAWin.Add(fine.Fundamentals.FinancialStatements.IncomeStatement.DepreciationAndAmortization.Value) if fine.Fundamentals.FinancialStatements.BalanceSheet.TotalAssets.Value != 0 : self.avgSizeWin.Add(fine.Fundamentals.FinancialStatements.BalanceSheet.TotalAssets.Value) def calculateWindowValues(self): #calculates the balance sheet accruals delta_assets = self.assetWin[0] / self.assetWin[ self.assetWin.Count -1 ] delta_cash = self.cashWin[0] / self.cashWin[ self.cashWin.Count -1 ] delta_liabilities = self.liabilityWin[0] / self.liabilityWin[ self.liabilityWin.Count -1 ] delta_debt = self.debtWin[0] / self.debtWin[ self.debtWin.Count -1 ] delta_tax = self.taxWin[0] / self.taxWin[ self.taxWin.Count -1 ] dep = sum(list(self.DandAWin)[:3]) # annual depreciation from q'ly D&A IS numbers avg_total = (self.avgSizeWin[0] + self.avgSizeWin[ self.avgSizeWin.Count -1 ]) / 2 #accounts for the size difference self.totalAccruals = ((delta_assets-delta_cash)-(delta_liabilities-delta_debt-delta_tax)-dep)/avg_totalDerek Melchin
Hi P Chen,
Symbol objects and RollingWindows (and QC/C# specific objects in general) don't work well with ObjectStore and they aren't pickleable. To address the Rolling Windows, you can cast them to a list and store them in a dict:
dict = { 'assetWin': list(self.assetWin) 'cashWin': list(self.cashWin) }before pickling. Alternatively, you can use Python deques instead. Please see the G-Score Investing Strategy Library on how to store, retrieve, and feed historical fundamental data.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Eugene
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!