Correct Indicator Use in Framework and Alpha Model

Take a look at the examples here and here, with add some code for warm up with a history function (with a property that flags ready in your indicators if it is done warming up) for new securities, and cleaning consolidators for removed securities.

Have you looked into the Research environment? I've been brute force debugging with backtests as well, but in hindsight going through each call line-by-line in a Jupyter notebook is probably more efficient. Community posts (including ones from Jared/QC Staff) and digging through the LEAN source code also tended to be more helpful than the documentation for me.

Also the .Reset() method resets indicators to their initial state. Presumably, it is creating a new instance of the object with the parameters defined in def__init__(self).

Thanks for those examples - I will go through them.

I don't think one is supposed to use the history function with this kind of frequency - I think you are supposed to use rolling window since it is more efficient. Yes, I have looked at the research environment but you can't call course/fine from it and you can not loop through the data - or at least so I understand.  Agreed with posts/source code. I am probably going to have to download Lean to de-bug through the entire stack - the problem is it does not ship with any sample Fundamental data.

The other thing is that the alpha models I quote above do not use either history or the rolling window so I assume they are somehow "warming up" the indicators in another way. I'm not sure how at present, I'll have to go through them with a fine tooth comb.  But again you can't use "Update" or "OnSecuritiesChanged" in Research?

As to consolidators, I am only using one resolution. The RSI Alpha Model also uses only one resolution - Daily. So I am not sure what it is consolidating.  Perhaps there is some necessity to consolidate the different time periods used in the RSI? I don't know.

All this becomes evident with time. Much time! Too much time! But it is rather like trying to paint a large canvas while looking through a pinhole.

None of those Indicator Alpha models I quote above WARM UP indicators. 

If you are using the Course Fine Universe with indicators it seems you need to demand History each time a stock comes into play?

What is the best way to warm up indicators on an on-going basis where you are using the Course fine Universe?

Indicators

See code provided by above link. Seems we have to use History after all?  Very, very odd that those Alpha Models I quote above don't use history. If you have a big universe and long term indicators you are going to run into trouble with those algos. They will simply miss trades (Idicator not ready0 which otherwise could have been captured.

See code provided kindly by Alexandre Catarino:

from Selection.FundamentalUniverseSelectionModel import FundamentalUniverseSelectionModel
#https://www.quantconnect.com/forum/discussion/7132/how-to-access-history/p1

class VentralModulatedThrustAssembly(QCAlgorithm):

def Initialize(self):
self.SetStartDate(2019, 7, 9) # Set Start Date
self.SetCash(100000) # Set Strategy Cash

self.SetUniverseSelection(SelectionModel())
self.SetAlpha(ConstantAlphaModel(InsightType.Price, InsightDirection.Up, timedelta(1), 0.025, None))
self.SetPortfolioConstruction(EqualWeightingPortfolioConstructionModel())


class SelectionModel(FundamentalUniverseSelectionModel):

def __init__(self,
filterFineData = True,
universeSettings = None,
securityInitializer = None):

super().__init__(filterFineData, universeSettings, securityInitializer)
self.periodCheck = -1
self.coarse = dict()

def SelectCoarse(self, algorithm, coarse):
if algorithm.Time.year == self.periodCheck:
return Universe.Unchanged

self.coarse = {x.Symbol: x
for x in coarse if x.HasFundamentalData and x.Volume > 0 and x.Price > 0}

return [symbol for symbol,_ in self.coarse.items()]

def SelectFine(self, algorithm, fine):

DERatio = lambda x: x.OperationRatios.LongTermDebtEquityRatio.ThreeMonths

fine = [x for x in fine if DERatio(x)
and x.CompanyReference.CountryId == "USA"
and (x.CompanyReference.PrimaryExchangeID in ["NYS", "NAS"])
and (algorithm.Time - x.SecurityReference.IPODate).days > 180
and x.EarningReports.BasicAverageShares.ThreeMonths *
x.EarningReports.BasicEPS.TwelveMonths *
x.ValuationRatios.PERatio > 1.52e8]
if len(fine) == 0:
return Universe.Unchanged

self.periodCheck = algorithm.Time.year

sortedByDEratio = sorted(fine, key=DERatio, reverse=False)[:50]
symbols = [x.Symbol for x in sortedByDEratio]

averages = dict()
history = algorithm.History(symbols, 200, Resolution.Daily).close.unstack(0)

for symbol in symbols:
# Remove NaN: symbol does not have 200 daily data points
df = history[symbol].dropna()
if df.empty:
continue

mom = Momentum(126)
for time, close in df.iteritems():
mom.Update(time, close)

# Adds Momentum to dict only if it is ready
if mom.IsReady:
averages[symbol] = mom

# Update with current data
for symbol, mom in averages.items():
c = self.coarse.pop(symbol, None)
mom.Update(c.EndTime, c.AdjustedPrice)

sortedbyMomentum = sorted(averages.items(), key=lambda x: x[1], reverse=True)

return [x[0] for x in sortedbyMomentum[:5]]

    As to consolidators, I am only using one resolution. The RSI Alpha Model also uses only one      resolution - Daily. So I am not sure what it is consolidating.

You should use Minute data and then consolidate them into Daily bars.

The Algorithm Framework is a helpful starting point in cleanly separating modules, but I've found that working with a more classical approach gives me more flexibility and control over what's going on without having to spend an hour digging through LEAN source code. I bypass the whole AlphaModel construction, and instead my Alpha model is defined in a function which is called in OnData() when certain criteria match (i.e. a flag for model is ready, time rules, entry/exit, etc), and any heavy computations are done on a scheduling function before markets open. This is more similar to the architecture of Quantopian, maybe give it a try. 

Adam W, I wholly agree with your point of view.  The classical model is along the lines I have always favoured in my own approach - a direct and simple route to what it is you want to achieve. At present however I am concentrating on the algorithm framework for a number of reasons.

Not least of which is that that is the direction Lean providers seem to want to go. In which case that is where their future development efforts will presumably be directed.

But the big question here is WARMING up indicators for the course/fine universe And this has been answered. You either use code similar to the above provided by Alexandre or you wait until the new warm up code quoted here by Jared is extended to all indicators and thoroughly tested:

self.EnableAutomaticIndicatorWarmUp = True

Not implemented for all indicators yet see Github.

We will always keep support for classic style algorithms as they are more suited for individual accounts. The framework is the architecture for a full firm in a single algo. I hope as the framework matures it's benefits will become 10x the classic style even for single accoounts

Absolutely, the benefits of the framework speak for themselves. I implement features of the framework architecture whenever possible, but for any persistent errors/bugs I simply work around it manually rather than spend hours trying to debug it with my limited C# knowledge. When the framework matures more and when one gets more accustomed to the API, I'd imagine it'll make the whole process significantly smoother for everyone.

Great thread. I second the concerns about lack of WarmUp within the docs for AlgoFramework. Even in the Bootcamp tutorials they failed to warmup the MOM indicator? And so, I assumed when using Indicators in AlphaModels there was no need to WarmUp. I have since learned the hard way, that is not the case!

Hey Mark,

Thank you for the feedback on the documentation! The 200-50 EMA Boot Camp describes how to warm up indicators with historical data using the framework.

Best
Rahul

How would this example work for the Bolliger bands  and MACD ? any examples somewhere 

Would be something like 

Hi Aalap,

I am unsure what your question is. The 200-50 EMA BootCamp mentioned by Rahul should be sufficient, except only the indicators are changed.

Best,
Shile Wen

Thanks I think I figured it out. I wanted to warmup the LowerBand and UpperBand values as well for BB indicator