Scheduled Universe Selection with Coarse and Fine Filters?

Hi Mostafa,

It's not trivial, indeed.

Did you read https://www.quantconnect.com/docs/algorithm-reference/universes?

I also recommend looking at this example: https://github.com/QuantConnect/Lean/blob/master/Algorithm.Python/EmaCrossUniverseSelectionAlgorithm.py

It shows the basics, like using a “SymbolData” class to contain your indicators for each symbol added in the universe.

Hope it helps,

Fred

Thanks Fred for your reply. Yes, I read almost all what's written about the universe and universe selection in the Algorithm Reference as well as some examples on the github.com, I understand the ideas of Coarse and Fine Universe selection, and the scheduled Universe selection, but I haven't seen both points together in one example. As when we have the universe selection as scheduled, we can't put AddUniverse with Coarse and Fine filters under the initialize method at the same time, that didn't work with me. It's all about how to do both together? i.e. (scheduled selection + coarse and fine filters).

Sorry Mostafa, I'm not sure I'm tracking your “scheduled universe selection” vs “coarse and fine filters”. Universe selection is done once a day, during the night. It calls coarse then fine then OnSecuritiesChanged. You can specify coarse and fine filters through methods or you can encapsulate them in a class (and there are pre-designed classes for universe selection in the framework).

I don't think you can perform (easily) universe selection more often than once a day. If you want to do it less frequently, you can program some logic using timestamps, for instance, so some days are skipped in the coarse and fine filters.

Cheers!

Fred

Hi Mostafa, Fred,

Sadly there currently is no easy way of doing universe selection for coarse and fine universe multiple times per day or at a different time as evident by this post;

This posts also links you to the github issue if you want to check for progress on it.

https://www.quantconnect.com/forum/discussion/7699/universe-selection-scanner-that-can-occur-multiple-times-a-day/p1

Here might be a potential hack for it but probably not very feasible;

https://www.quantconnect.com/forum/discussion/8726/volume-of-the-day-for-universe-of-stocks/p1

And yes as Fred already said correctly, you can make the universe selection less frequent relatively easily but not more frequent.

But also I am not entirely sure if they even refresh the Universe data more frequently than daily, I think currently it is only being refreshed daily (the coarse and fine data), so there probably wouldn't be much to gain by running it more frequently than the data updates.

Just a side note for others reading this post;

 You could probably perform your fundamental (coarse and fine fundamentals etc.) daily and then have your own custom universe selection run on the universe returned by the coarse and fine if you want to check for intraday technicals or similar.

Thanks!

Thanks Mak and Fred.

Actually, I don't need to run universe selection more frequent. Just once a day at a different time (i.e. 7.20am instead of standard 12am).

I think this part Mak advised could be a potential solution:

You could probably perform your fundamental (coarse and fine fundamentals etc.) daily and then have your own custom universe selection run on the universe returned by the coarse and fine if you want to check for intraday technicals or similar.

That raises these new questions:

1. Is it possible to apply filters only within (coarse and fine filters) without doing any sorting first, and return all securities matching the conditions without specifying a certain number to return?
2. Any coding examples of implementing this above approach (coarse+fine+custom) filters?

Thanks

One more question:

3. Is it possible to put conditions on EMA/SMA filters inside fine filter instead of coarse filter?

Hey Mostafa, 

Not sure right now but maybe this post can be helpful to you;

https://www.quantconnect.com/forum/discussion/8045/universe-coarse-fine-selection-with-past-indicators-values-indexed-in-rollingwindow/p1

Thanks!

Hello Mostafa,

Based on this:" Actually, I don't need to run universe selection more frequent. Just once a day at a different time (i.e. 7.20am instead of standard 12am). ", just perform coarse, return all coarse symbols (don't filter or just perform basic filtering - like not returning 8000+ assets), and schedule a method to run each trading day at 7:20AM to perform the filtering you want on active securities based on the criteria you want.

It'll work but it'll take lots of RAM as you'll be tracking lots of data. Depending on the efficiency/complexity of your filter, it might also take lots of CPU. So, you might need to sub to a beefier backtesting/live trading node.

Fred

Mostafa Omran, regarding this: “Is it possible to put conditions on EMA/SMA filters inside fine filter instead of coarse filter?”

Yes, you can do this by calling self.History() for each symbol, and invoking the EMA/SMA indicators for each. 

I shared a post with a relevant example here.

Also see this relevant post, that discusses pre-market screening.