Back

[Feature] Coarse Universe Selection

Hey hey hey!

Late last night we launched our newest feature: Universe selection. This allows the algorithm to define the universe of symbols it would like to receive data for by specifying a function that returns the symbols! QuantConnect currently provides what we call 'coarse' universe selection data. This is an aggregate of daily that includes the daily close, daily volume, and the dollar volume (daily close * EMA30(volume)).

You can add coarse universe selection criteria by using the AddUniverse method. Here's an example of adding a coarse universe selection that takes all symbols with a daily dollar volume of over $1 billion:// in Initialize()
const decimal OneBillion = 1000m*1000m*1000m;
AddUniverse(coarse =>
{
return from c in coarse
where c.DollarVolume >= OneBillion
select c.Symbol;
});

What this code means is that we want to filter the complete unverse of US equity symbols by selecting all symbols with a DollarVolume >= OneBillion! We'll automatically handle pumping the right data for your selected symbols in your OnData method. Also, we won't remove securities that have open orders or positions.

Have a peek at the attached demo coarse universe algorithm example. We welcome feedback and questions!

Update Backtest






The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.



So, let me make sure I understand this:

(1) The universe is a set of symbols, defined by a user-supplied filter function.
(2) Every (UniverseSettings.Resolution) time span, the universe will be updated by calling the filter function.
(3) If and only if the set of symbols changes, OnSecuritiesChanged is called and given a SecuritiesChanges object.

Is that an accurate summation? Any potential gotchas or limitations we should be aware of?
0

1. Absolutely spot on! A little more specific though, in general, a universe takes in some data (in this case CoarseFundamental) and returns a filtered set of symbols. In another forum post I demonstrated how you can use your own custom data sources for universe selection data! There also exist overloads that only accept a DateTime argument and return a set of symbols, useful for hitting an external API that doesn't lend itself to our custom data system.

EDIT: Here's a link to the other forum post where I shared a custom data universe example.

2. Correct, more specifically, it's every resolution step within the market hours of the universe's market. For instance, in the example project attached it is a US-Equity market universe, so it will fire a midnight the day after trading, so after Monday's trading, Tuesday at midnight all the way until after Friday's trading which is Saturday at midnight. Likewise, for minute data (no extended hours) it will start firing at 9:31AM and continue until 4:00PM NewYork time.

3. Correct, OnSecuritiesChanged will only fire if the securities have actually changed. An interesting point is that you can define multiple universes. If this is done, it will contain all the changes (set union) between the two universes. Another important note pertaining to multiple universes is that one universe can't remove another universe's securities.

One thing that will soon be implemented is the ability to define the minimum time a stock will exist in the universe. This will be useful if you require indicators or state that require multiple days to become usable. I expect to have this ready before the end of next week.

We've held off on pushing this feature until it was pretty solid, so if you find anything you wouldn't expect please let us know. You may enjoy taking a peek at the various overload here that offer varying degrees of flexibility. For ultimate flexibility you could derive from the Universe type directly, but that's pretty advanced :)
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


so we can simulate the s&p500 by setting top 500 by market cap?
0

The coarse data set does not currently include a figure for market cap, but you can simulate the sp500 by taking the top 500 stocks by daily dollar volume.
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Great work Michael. Will the list of universe selection data points be expanded at some point? Trying to back test the Dogs of the Dow strategy and require yield.

Thanks!
0

@Nat -- Yes! We started with this simple data-set but will expand it with other fundamental values in the next few months.
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


When I add the top method to Initialize I receive a message that OneBillion doesn't exist in the current context. I've tried changing it to an integer and looking for members that in DollarVolume property but couldn't find anything. Any thoughts?

0

You can add const decimal OneBillion = 1000m*1000m*1000m;. It's just a constant to find stocks with volumes over a billion, Likewise, you could change the volume filter to anything you like!

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Perfect! Thank you for updating the code. I modified what you provided to filter the securities based on price. Now all I need to do is implement a gap up and gap percentage method then I can move on to other parts of my algo. Here's what I did.

2


So this might be a dumb question, but can you apply indicators to the coarse universe selection function? Wondering if I could say, for example, "grab all stocks whose 1-year momentum percent is greater than 10%". :-)

Excellent work guys!

0

Ah found it in an older thread, sorry about that! For those who come here wondering the same thing, here is how you conduct technical analysis in the Coarse Universe filter:

https://github.com/QuantConnect/Lean/blob/master/Algorithm.CSharp/EmaCrossUniverseSelectionAlgorithm.cs

1

Stephen - You can use indicators on the universe selection, that's actually what I'm working on right now. I've been using the link you provided in another post.
https://github.com/QuantConnect/Lean/blob/master/Algorithm.CSharp/EmaCrossUniverseSelectionAlgorithm.

Community - I've been looking at the LogReturns indicator and found a way to compare the difference in the current price with yesterday's close. Translating those findings into the coarse universe selection is a bit more tricky. Similar to what Stephen was saying, I want to scan the entire universe for stocks that have gapped up. Then scan those to find ones that have the highest gain.
This is one of the biggest parts of my strategy and I've been working on it for a while. So if you could point me in the right direction or give me an example that would be fantastic.


Best,
-Travis

2


Hey Travis, you don't need to copy all of your indicators into the project. They should be available without adding the file.

Sadly the gap open condition requires open and close prices, but the coarse universe data set only has the daily closing price, so I'm not sure you can detect gap up/down on open. If you're just looking for stocks that had the largest one day gains, take a look at this example algorithm. I used the WSJ top gainers page to find stocks that gained the most in one day... and then shorted them!

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hi Michael,

Quick question for ya: If you have indicators that are being used within the Coarse Universe Linq statement, are your indicators updated daily or are they updated at the pace of the Universe Resolution?

Example of the indicators being declared and updated within the Linq statement:

https://github.com/QuantConnect/Lean/blob/master/Algorithm.CSharp/EmaCrossUniverseSelectionAlgorithm.cs

I'm getting too-good-to-be-true results in my algorithm. It uses a certain indicator that requires 20 days of daily data. My Universe resolution is set to minute though, and I'm wondering if I'm getting a self-fulfilling prophecy problem here (maybe benefiting from fill-forwarding?).

0

Micheal - Can you backtest live data or is it exclusive to live mode? I've tried to run this algorithm live and made sure the links to WSJ were functional though no trades are being made. I was also inspecting the HTML on the site to find more data to parse. It doesn't seem to be pulling in any data even after I left it running overnight.

Trying to run a price filter where d.Price

0

Hi Travis,

I thought I saw someone use custom data with their backtest. Is that the same as "live data" in this case?

https://www.quantconnect.com/forum/discussion/418/bubble-algorithm-using-cape-ratio-macd-and-rsi#Item_12

Hopefully I'm not misunderstanding you!

0

When using indicators in any universe selection function, they will be updated at the same frequency as the universe gets updated. In the case of coarse universes, they will update on a daily time frame. Another thing to note is that we do not apply fill forward to universe selection. If you need more help please post an algorithm demonstrating your issue/concern.

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Travis - I'll run the algorithm over this evening with some debugging to see if things are working as expected. Did you make any changes to the example algorithm? I'm not sure what you mean by 'backtest live data.' The NyseTopGainers custom data type is written to support both live and backtesting cases. It supports the backtest case by hitting dropbox (I made a small script to scrape the pages and built a csv from it)

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Great, thanks for the validation, Michael!

0

Clarification on my above statement: We don't apply fill forward to the universe selection data itself, but subscriptions added via universe selection can be set to have fill forward data using the UniverseSettings.FillForward property. This is set to true by default.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Micheal - I made a few changes to the example to parse price and use WSJ's composite market data. I tried running my own version and the one that you made but neither work. That's pretty cool how you generated that spreadsheet. I was wondering where you got it from.

0

Update!
I redact my previous question about backtesting live data since it doesn't make sense. Lol

What I meant was that I wanted to pull in data directly from a site at runtime without having to make the algorithm go live.

I found a page scraper extension on the chrome web store to generate csv files.

Here's a guess at how to add a price filter
if(line.StartsWith(@"3.69"))
{
//return price
}

Still working on that part

I got the algorithm to pull in data from today and what you see below is actually the algorithm buying UNT on January 6th 2015 which is today's current highest gainer.

0


Hi, guys. One question. How can I update witth custom data, for example Open:

where _vars.UpdateSkew(cf.EndTime, cf.Price)

0

Figured cusytom updates out. Anotehr one. What is the best way to close symbols not based on SecurityChanges, but defining specific ("where"?) rules?

0

Thanks for sharing
0

Hi,

I want to simulate and use the same data as in Quantopian and given by the author or articles/ research papers. For this reason a SP500 selection is often used. The definition is:

  1. Market capitalization must be greater than or equal to $6.1 billion USD
  2. Annual dollar value traded to float-adjusted market capitalization is greater than 1.0
  3. Minimum monthly trading volume of 250,000 shares in each of the six months leading up to the evaluation date

When is this to be added to the selection? Selecting on volume and dollarvolume is not the same as above.

J.

0

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed