Adding Momentum Filter before Filtering by Fine Data by Vncne

Vncne INVESTOR

| Reward Discussion

Hi! I have a simple liquid universe algo that filters fine data, but now I'm trying to add a momentum filter(EMAs) before having the algo filter by fine data for the liquid universe. I tried to incorporate the code from this bootcamp into my existing algo, but I'm encountering errors. I'm trying to find what I'm missing, but I can't figure it out.

I know the code is wrong because I removed the Universe Selection in Initialize; I only did that so a backtest could be produced so I could attach the algorithm to this post. If I left in the Universe Selection in Initialize it gives me the following error:

Runtime Error: AttributeError : 'TechnologyUniverseModule' object has no attribute 'History'

at <lambda> in FundamentalUniverseSelectionModel.py:59
at FilteredSelectCoarse in FundamentalUniverseSelectionModel.py:72
AttributeError : 'TechnologyUniverseModule' object has no attribute 'History' (Open Stacktrace)

The problem seems to be when pulling the history for the MA indicators for the securities? Is it because I created a class for the Universe Selection, while the bootcamp didn't? How then can I incorporate it in my case?

Thank you!

Update Backtest

Vncne

Accepted Answer

Update Backtest

Notebook

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Vncne

14.4k Pro ,

The coarse selection part was done like this:

def __init__(self, filterFineData = True, universeSettings = None, securityInitializer = None):

        #Initializes a new default instance of the TechnologyUniverseModule

        super().__init__(filterFineData, universeSettings, securityInitializer)

        self.numberOfSymbolsCoarse = 1000

        self.numberOfSymbolsFine = 100

        self.dollarVolumeBySymbol = {}

        self.lastMonth = -1

        self.averages = {}



def SelectCoarse(self, algorithm, coarse):

        

        if algorithm.Time.month == self.lastMonth: 

            

            return Universe.Unchanged

            

    

        coarse = sorted([x for x in coarse if x.HasFundamentalData and x.Price > 10],

                key = lambda x: x.DollarVolume, reverse=True)[:self.numberOfSymbolsCoarse] 



        

        for security in coarse:  

            

            symbol = security.Symbol

            

            if symbol not in self.averages:



                history = self.History(symbol, 200, Resolution.Daily)



                self.averages[symbol] = SelectionData(history) 





            self.averages[symbol].update(self.Time, security.AdjustedPrice)

            

            if  self.averages[symbol].is_ready() and self.averages[symbol].fast > self.averages[symbol].slow:

                

                self.dollarVolumeBySymbol.append(symbol)





        if len(self.dollarVolumeBySymbol) == 0:

            

            return Universe.Unchanged

            



        return list(self.dollarVolumeBySymbol.keys())

and for constructing the indicators:

class SelectionData():

    

    def __init__(self, history):

        self.slow = ExponentialMovingAverage(200)

        self.fast = ExponentialMovingAverage(50)

        

        

        for bar in history.itertuples():

            self.fast.Update(bar.Index[1], bar.close)

            self.slow.Update(bar.Index[1], bar.close)

    

    def is_ready(self):

        return self.slow.IsReady and self.fast.IsReady

    

    def update(self, time, price):

        self.fast.Update(time, price)

        self.slow.Update(time, price)

Shile Wen

63.5k Pro ,

Hi Vncne,

To use History in a Framework component, we need to call algorithm.History instead of self.History. In addition, I also fixed one instance where self.Time was used instead of algorithm.Time. Furthermore, the algorithm uses append on a dict, so I’ve also fixed this as well. Please see the attached backtest for reference.

Best,
Shile Wen

Vncne

14.4k Pro ,

You're the best Shile Wen , thanks! Your code fixed it, but I wanted to move the indicator filter to be done after the fine filter instead of right after the coarse filter and before the fine filter originally so the algorithm has less data it needs to filter through.

But I now get this error:

Runtime Error: AttributeError : 'FineFundamental' object has no attribute 'AdjustedPrice'
at <lambda> in FundamentalUniverseSelectionModel.py:46
AttributeError : 'FineFundamental' object has no attribute 'AdjustedPrice' (Open Stacktrace)

By the way, in the algorithm I attached to this reply, I removed the self.AddUniverseSelection() in Initialize in order for the algorithm to produce a backtest that I would be able to attach to this post;

#self.AddUniverseSelection(TechnologyUniverseModule())

self.AddRiskManagement(NullRiskManagementModel())

self.SetPortfolioConstruction(NullPortfolioConstructionModel())

self.SetExecution(ImmediateExecutionModel())

Thanks again, you're a huge help!

Vncne

14.4k Pro ,

Sorry, just another question; I also noticed that these produce different results:

coarse = sorted([x for x in coarse if x.HasFundamentalData and x.Price > 10],

         key = lambda x: x.DollarVolume, reverse=True)[:self.numberOfSymbolsCoarse]

        



for security in coarse:

            

   symbol = security.Symbol

            

   self.dollarVolumeBySymbol[symbol] = security.DollarVolume





return list(self.dollarVolumeBySymbol.keys())

and:

coarse = sorted([x for x in coarse if x.HasFundamentalData and x.Price > 10],

                key = lambda x: x.DollarVolume, reverse=True)[:self.numberOfSymbolsCoarse] 



self.dollarVolumeBySymbol = {x.Symbol:x.DollarVolume for x in coarse}



return list(self.dollarVolumeBySymbol.keys())

Why do the outputs of both of these differ?

Arthur Asenheimer

15.9k Pro ,

Hi,

in your first version, you're adding items to your dollarVolumeBySymbol dictionary without any refresh/cleanup. So the dictionary will keep all the keys from your past Coarse Selection calls.

In your second version, you're creating a new dictionary every time Coarse Selection is invoked.

To the question of your previous post (AttributeError: FineFundamental object has no attribute AdjustedPrice):

As the error message suggests, AdjustedPrice is not available in FineFundamental. You can see here that AdjustedPrice is not among the properties of the FineFundamental class.

This was something I also needed some time ago. I then decided to move some parts of my code to the CoarseSelection method but that wasn't a very elegant solution, to be honest.

Maybe Shile can offer a solution or workaround here.

Vncne

14.4k Pro ,

Hi Arthur Asenheimer , thanks for the response! Yeah I figured as such with the first version(in regards to not creating a new dict every time) So I guess this implies that there are times where symbols that should be removed from the universe are not liquidated? Maybe I'm misunderstanding again.

How would you suggest I incorporate this:

self.dollarVolumeBySymbol = {x.Symbol:x.DollarVolume for x in self.averages}

into this code, in order to make sure that the universe is always refreshed and up to date?

for security in sortedByDollarVolume:  

            

   symbol = security.Symbol

            

   if symbol not in self.averages:



      history = algorithm.History(symbol, 200, Resolution.Daily)



      self.averages[symbol] = SelectionData(history) 





   self.averages[symbol].update(algorithm.Time, security.AdjustedPrice)

            

   if self.averages[symbol].is_ready():

                

      self.dollarVolumeBySymbol[symbol] = security.DollarVolume

Vncne

14.4k Pro ,

What I tried:

def SelectCoarse(self, algorithm, coarse):

        

   selected = []

        

   if algorithm.Time.month == self.lastMonth:      

      return Universe.Unchanged

            

    

   sortedByDollarVolume = sorted([x for x in coarse if x.HasFundamentalData and x.Price > 10],

      key = lambda x: x.DollarVolume, reverse=True)[:self.numberOfSymbolsCoarse] 



        

   for security in sortedByDollarVolume:  

            

      symbol = security.Symbol       

      if symbol not in self.averages:



         history = algorithm.History(symbol, 200, Resolution.Daily)

         self.averages[symbol] = SelectionData(history) 





      self.averages[symbol].update(algorithm.Time, security.AdjustedPrice)

        

      if self.averages[symbol].is_ready():

                

         selected[symbol] = security.DollarVolume





   self.dollarVolumeBySymbol = {x.Symbol:x.DollarVolume for x in selected}

        

        

# If no security has met the QC500 criteria, the universe is unchanged.

   if len(self.dollarVolumeBySymbol) == 0:        

      return Universe.Unchanged

            



   return list(self.dollarVolumeBySymbol.keys())

Error:

Runtime Error: TypeError : list indices must be integers or slices, not Symbol
at <lambda> in FundamentalUniverseSelectionModel.py:59
at FilteredSelectCoarse in FundamentalUniverseSelectionModel.py:72
TypeError : list indices must be integers or slices, not Symbol (Open Stacktrace)

Vncne

14.4k Pro ,

Oh yeah, just to clarify regarding what you said about the first version not being refreshed/cleaned up; is this still true even if I added this function before the coarse function?

class TechnologyUniverseModule(FundamentalUniverseSelectionModel):



   def __init__(self, filterFineData = True, universeSettings = None, securityInitializer = None):

      self.numberOfSymbolsCoarse = 1000

      self.numberOfSymbolsFine = 100

      self.dollarVolumeBySymbol = {}

      self.lastMonth = -1

      self.averages = {}

      super().__init__(filterFineData, universeSettings, securityInitializer)



   def SelectCoarse(self, algorithm, coarse):

Would self.dollarVolumeBySymbol = {} technically refresh the dict/list every time before proceeding to the coarse and fine filters?

Arthur Asenheimer

15.9k Pro ,

The __init__ method will be called only once at the beginning when creating an instance of the class TechnologyUniverseModule. So this will not cause a refresh every time before proceeding the coarse/fine selection.

Re your question to pick vers 1 or vers 2, there is no right answer. But in most cases it's more accurate to refresh your list/dict. So vers 2 is more appropriate here.

In your CoarseSelection method you can do:

self.dollarVolumeBySymbol = { .... some dict comprehension logic ... }

self.dollarVolumeBySymbol = {} for symbol in mylist: self.dollarVolumeBySymbol[symbol] = value

I recommend you to learn the OOP principles and python basics. There is plenty of good free ressources online.

Arthur Asenheimer

15.9k Pro ,

And one more thing: the debugger can be very helpful in such situations and let you find out a lot stuff yourself.

Vncne

14.4k Pro ,

I'm actually working on a Udemy course now! But I'm also trying to learn QC api mostly through trial and error, so I apologize for all the basic questions;

in CoarseSelection method I tried:

for security in sortedByDollarVolume:  

            

   symbol = security.Symbol

            

   if symbol not in self.averages:



      history = algorithm.History(symbol, 200, Resolution.Daily)



      self.averages[symbol] = SelectionData(history) 





   self.averages[symbol].update(algorithm.Time, security.AdjustedPrice)

            

   if self.averages[symbol].is_ready():

                

      self.dollarVolumeBySymbol[symbol] = security.DollarVolume





self.dollarVolumeBySymbol = {x.Symbol:x.DollarVolume for x in self.dollarVolumeBySymbol \

if self.averages[x.Symbol].is_ready()}

But it returns an error:

Runtime Error: AttributeError : 'Symbol' object has no attribute 'Symbol'
at <lambda> in FundamentalUniverseSelectionModel.py:59
at FilteredSelectCoarse in FundamentalUniverseSelectionModel.py:72
AttributeError : 'Symbol' object has no attribute 'Symbol' (Open Stacktrace)

I apologize again and I appreciate the patience of everyone on QC for the beginners, and thanks again!

Arthur Asenheimer

15.9k Pro ,

The error message desribes it quite well: "Symbol" object has no attribute "Symbol".

You're trying to do x.Symbol where x is already the Symbol object. Just remove the ".Symbol" part in the corresponding line and it should work then. I guess the error is thrown in the line

self.dollarVolumeBySymbol = {x.Symbol : x.DollarVolume for x in self.dollarVolumeBySymbol if self.averages[x.Symbol].is_ready()}

Replace this line with:

self.dollarVolumeBySymbol = { symbol : dollarVolume for symbol, dollarVolume in self.dollarVolumeBySymbol.items() if self.averages[symbol].is_ready() }

Vncne

14.4k Pro ,

Ah okay, I assumed that the symbol = security.Symbol was confined within the for loop that I put it in. I tried following your suggestion and implemented it like this:

def SelectCoarse(self, algorithm, coarse):

        

   selected = {}

        

   if algorithm.Time.month == self.lastMonth: 

      return Universe.Unchanged

            

    

   sortedByDollarVolume = sorted([x for x in coarse if x.HasFundamentalData and x.Price > 10], key = lambda x: x.DollarVolume, reverse=True)[:self.numberOfSymbolsCoarse] 

        

   for security in sortedByDollarVolume:  

            

      symbol = security.Symbol

            

      if symbol not in self.averages:



         history = algorithm.History(symbol, 200, Resolution.Daily)



         self.averages[symbol] = SelectionData(history) 





      self.averages[symbol].update(algorithm.Time, security.AdjustedPrice)    

              

      selected[symbol] = security.DollarVolume





   self.dollarVolumeBySymbol = {symbol:dollarVolume for symbol, dollarVolume in selected.items() if self.averages[symbol].is_ready()}



   if len(self.dollarVolumeBySymbol) == 0:

      return Universe.Unchanged



   return list(self.dollarVolumeBySymbol.keys())

It seems to be working, but does that look right to you?

Vncne

14.4k Pro ,

Hi Shile Wen, after following your and Arthur's suggestions, I was more or less able to run backtests on the algorithm with relatively no more issues. However, when deploying the algorithm live, I've been encountering a different issue. Everyday at around 11:30, the live deployment has been producing the error below most likely due to an overload of history requests:

Error:

Runtime Error: Algorithm took longer than 10 minutes on a single time loop. CurrentTimeStepElapsed: 10.0 minutes

Most likely from this code:

for security in sortedByDollarVolume:  

            

   symbol = security.Symbol

            

   if symbol not in self.averages:



      history = algorithm.History(symbol, 200, Resolution.Daily)



      self.averages[symbol] = SelectionData(history)





   self.averages[symbol].update(algorithm.Time, security.AdjustedPrice)



   selected[symbol] = security.DollarVolume

Too many history requests. This is within the CoarseFilter function. An obvious solution would be to move the history request to the FineFilter function so that there is a significantly lower number of securities to request history for, but like Arthur said above, AdjustedPrice is not available in FineFundamental. So, moving the history request to FineFilter produces another different error:

A suggestion I received from someone else was to save AdjustedPrice from Coarse in a class variable dictionary so that it may also be used in FineFilter. I'm not exactly sure how to do that, but I tried it like this; I saved it in Coarse like:

self.AdjustedPrice = {x.Symbol:x.AdjustedPrice for x in sortedByDollarVolume}

Then attempted to use it in Fine like:

for security in sortedByDollarVolume:

            

   symbol = security.Symbol

            

   if symbol not in self.averages:



      history = algorithm.History(symbol, 200, Resolution.Daily)



      self.averages[symbol] = SelectionData(history)

                



   self.averages[symbol].update(algorithm.Time, self.AdjustedPrice)



   selected[symbol] = security.DollarVolume

But now end up with this error:

Runtime Error: Trying to dynamically access a method that does not exist throws a TypeError exception. To prevent the exception, ensure each parameter type matches those required by the Update method. Please checkout the API documentation.
at <lambda> in FundamentalUniverseSelectionModel.py:46
TypeError : No method matches given arguments for Update (Open Stacktrace)

Derek Melchin

STAFF Pro ,

Hi Vncne,

The error in the code above surfaces because we are passing the `AdjustedPrice` dictionary to the update method when it's expecting a float. To resolve this, we can define SelectCoarse to include

self.last_coarse = {cf.Symbol: cf for cf in sorted_by_dollar_volume}

Then in SelectFine, we can access the AdjustedPrice with

self.last_coarse[symbol].AdjustedPrice

We can then reduce the number of History requests in SelectFine to just one with

new_symbols = []

for security in sorted_by_dollar_volume:

    symbol = security.Symbol

    if symbol not in self.averages:

        new_symbols.append(symbol)

history = algorithm.History(new_symbols, 200, Resolution.Daily)



selected = []

for security in sorted_by_dollar_volume:

    symbol = security.Symbol

    if symbol in new_symbols:

        self.averages[symbol] = SelectionData(history.loc[symbol])

    self.averages[symbol].update(algorithm.Time, self.last_coarse[symbol].AdjustedPrice)

    if self.averages[symbol].is_ready() and self.averages[symbol].fast > self.averages[symbol].slow:

        selected.append(symbol)

This adjustment just requires us to change the SelectionData constructor to read

for time, row in history.iterrows():

    self.fast.Update(time, row.close)

    self.slow.Update(time, row.close)

See the attached backtest for reference.

Best,
Derek Melchin

Vncne

14.4k Pro ,

Thank you Derek!! With a little modification, I was able to get it working the way I wanted using your suggestion!

Vncne INVESTOR

Update Backtest

Notebook

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!

Platform

QuantConnect Community

Alpha League Competition: $1,000 Weekly Prize Pool

Filter Discussion by Tags

Community Ambassadors

Earn free QuantConnect cloud credits by helping others, each week we airdrop credits to the most active community members.

231,853 Quants.

Online Now

Stay Connected