Share New Research 
Hi team,
Have we made changes to our fundamental data recently. The backtests of one of my algo that relies on fundamental data now has drastically different result from the run before 11th Oct. I also need to make some updates to avoid errors like:
- change self.Time - x.SecurityReference.IPODate to self.Time.replace(tzinfo=timezone('US/Eastern')) - x.SecurityReference.IPODate. Look like IPODate now become offset-aware ?
- PreferredSharesNumber and TreasurySharesNumber seems to be nan for a lot of symbols which were ok before.
Your help is very much appreciated.
Thanks,
Jared Broad
Hi Nguyen Huu Quan
We recently (yesterday) made a substantial change documented here that makes fundamental data 10x faster. All our tests signaled the data returned the same results, but we'll look into those properties above immediately and get back to you today with the results.
Can you please share an example symbol(s) returning NaN that were not before?
Best
Jared
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Jared Broad
Just a note the reason for the NaNs could be that before the default for "no value" was 0 which is quite misleading in cases like “PreferredSharesNumber" as 0 is a valid result. We changed those properties to NaN since technically they did not have data (i.e. unknown). Now if it indicates 0 it genuinely means there are no preferred shares.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Nguyen Huu Quan
Thanks Jared. I also logged the data ticket here
Nguyen Huu Quan
Some of the data are listed below. Unfortunately I don't log the fundamental data in the previous runs so cannot really tell what has changed, only finding that the algo now is giving very different result from before. Not sure if there were more changes than just these fields.
sym:RH VB9XTDFRGZTX,cap:3905595285, ord:18928309.0,pre:nan, tre:2170196.0
sym:AHA R735QTJ8XC9X,cap:16747894349, ord:170100000.0,pre:nan, tre:60100000.0
sym:LRCX R735QTJ8XC9X,cap:38711399685, ord:144871000.0,pre:nan, tre:140970000.0
sym:NVDA RHM8UTD8DT2D,cap:132644880000, ord:2448000000.0,pre:nan, tre:1304000000.0
sym:MU R735QTJ8XC9X,cap:52641080000, ord:1108000000.0,pre:nan, tre:77000000.0
sym:AHA R735QTJ8XC9X,cap:16747894349, ord:170100000.0,pre:nan, tre:60100000.0
sym:AMD R735QTJ8XC9X,cap:45690770063, ord:1114000000.0,pre:nan, tre:5000000.0
sym:DH R735QTJ8XC9X,cap:63347207496, ord:506677740.0,pre:nan, tre:nan
sym:AAPL R735QTJ8XC9X,cap:1187462571250, ord:17772944000.0,pre:nan, tre:nan
sym:NVDA RHM8UTD8DT2D,cap:132644880000, ord:2448000000.0,pre:nan, tre:1304000000.0
sym:AHA R735QTJ8XC9X,cap:16747894349, ord:170100000.0,pre:nan, tre:60100000.0
sym:LRCX R735QTJ8XC9X,cap:38711399685, ord:144871000.0,pre:nan, tre:140970000.0
sym:AMD R735QTJ8XC9X,cap:45690770063, ord:1114000000.0,pre:nan, tre:5000000.0
sym:AAPL R735QTJ8XC9X,cap:1187462571250, ord:17772944000.0,pre:nan, tre:nan
sym:NVDA RHM8UTD8DT2D,cap:132644880000, ord:2448000000.0,pre:nan, tre:1304000000.0
Nguyen Huu Quan
The above is datapoint at 2019-12-31 00:00:00
Martin Molinero
Hello Nguyen Huu Quan !
Thank you for the reports. We shipped a fix removing the time zone awareness of the datetime objects, this is already in production starting from version v15951.
Regarding the `PreferredSharesNumber` and `TreasurySharesNumber`, we've reviewed the previous behavior and confirmed these were not available either and were silently returning 0 as Jared has pointed out. We suggest checking the boolean `HasValue` property, for example `PreferredSharesNumber.HasValue`
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Nguyen Huu Quan
Thanks Martin Molinero , I updated the algo to return 0 where fundamental data is NaN. The backtest got better but still quite different/worse than what I had before, which makes me feel something is still different.
Nguyen Huu Quan
Hi Martin Molinero , I dug deeper into the backtest and found some changes in the behaviour. The algo has this logic inside coarse selection:
sortedByDollarVolume = sorted([x for x in coarse if x.HasFundamentalData and x.DollarVolume > 3000000 and x.Price >0],
key = lambda x: x.DollarVolume, reverse=True)[:200]
In the old backtests, on the date 2020-04-29, symbol NEM passed this selection and was selected. However in the latest backtests, HasFundamentalData for NEM seems to return false and NEM was excluded.
Could you help take a look if this change is expected ? I think this may happen for some other symbols as well which caused the diff in the backtests.
Thanks
Levi Freedman
My algorithm was broken today also, but now seems to be working. But in digging through the github notes on this topic I found something rather alarming - does the Morningstar data not include delisted stocks?
Jared Broad
Hey Nguyen Huu Quan,
Thank you for the helpful comment; we dug into it and identified an issue. The fix is being built and deployed now and will return to production in 2-3 hours.
Best
Jared
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Jared Broad
Just a note: if you're doing a pure coarse filter, you shouldn't include HasFundamentalData; as not all coarse will include fundamental data.
sorted([x for x in coarse if x.DollarVolume > 3000000 and x.Price >0], key = lambda x: x.DollarVolume, reverse=True)[:200]
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Nguyen Huu Quan
Many thanks Jared Broad for the fast turnaround. Please kindly let me know once the fix is in PROD so I can verify again.
Also thanks for the suggestion. The algo filters further using other fundamental factors in the fine selection so I added HasFundamentalData to ensure the presence of data.
Jared Broad
Yes we now include delighted stocks. Full announcement next week
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Nguyen Huu Quan
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!