Back

OnSecuritiesChanged not producing equal additions and removals with fixed size Universe Selection

Hi everyone,

I've been struggling for a couple of days now trying to employ a simple value strategy in the algorithm framework.

I've attached an extremely short backtest below just to highlight my problem.

My universe selection just involves a couple of filters and I eventually produce a universe consisting of 40 securities:

FilterFactor returns: 

filter_factor[:20] + filter_factor[-20:]

SelectCoarse and SelectFine:

def SelectCoarse(self, algorithm, coarse):
algorithm.Log("Generating universe...")
universe = self.FilterDollarVolume(coarse)
return [c.Symbol for c in universe]

def SelectFine(self, algorithm, fine):
universe = self.FilterFactor(self.FilterFinancials(fine))
algorithm.Log(f"Universe consists of {len(universe)} securities")
return [f.Symbol for f in universe]

As you can see I log the length of universe and it is consistently 40 securities as expected.

In my alpha generation I also logged the amount of ActiveSecurities (as I now understand that this includes holded securities, not just the selected universe), securities in my list, insights emitted, and in the OnSecuritiesChanged() function, the number of changes.AddedSecurities and changes.RemovedSecurities.

Now as the universe emitted is always a fixed size (40) I was expecting that the number of securities added must be the same as the number of securities removed. However, as you can see the number of insights emitted decreases as the removed securities outweighs the added securities.

Am I missing something with regards to OnSecuritiesChanged? Is it the changes in ActiveSecurities? If so how can I get just the universe I selected in my UniverseSelectionModel? Is this a problem with timings and data resolution?

Furthermore, when I run this backtest for a longer period, even 3 months, it crashes with a NullReferenceException below (unfortunately, as it errored, I cannot attach it as a backtest):

Runtime Error: NullReferenceException : Object reference not set to an instance of an object
at QuantConnect.Algorithm.QCAlgorithm.ProcessInsights (QuantConnect.Algorithm.Framework.Alphas.Insight[] insights) [0x0003d] in <e78654be6f7946aebb4dc6b6fa374eb7>:0
at QuantConnect.Algorithm.QCAlgorithm.OnFrameworkData (QuantConnect.Data.Slice slice) [0x001a4] in <e78654be6f7946aebb4dc6b6fa374eb7>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in <b0e1ad7573a24fd5a9f2af9595e677e7>:0 NullReferenceException : Object reference not set to an instance of an object
at QuantConnect.Algorithm.QCAlgorithm.ProcessInsights (QuantConnect.Algorithm.Framework.Alphas.Insight[] insights) [0x0003d] in <e78654be6f7946aebb4dc6b6fa374eb7>:0
at QuantConnect.Algorithm.QCAlgorithm.OnFrameworkData (QuantConnect.Data.Slice slice) [0x001a4] in <e78654be6f7946aebb4dc6b6fa374eb7>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in <b0e1ad7573a24fd5a9f2af9595e677e7>:0

Stacktrace:

System.Exception: NullReferenceException : Object reference not set to an instance of an object
at QuantConnect.Algorithm.QCAlgorithm.ProcessInsights (QuantConnect.Algorithm.Framework.Alphas.Insight[] insights) [0x0003d] in <e78654be6f7946aebb4dc6b6fa374eb7>:0
at QuantConnect.Algorithm.QCAlgorithm.OnFrameworkData (QuantConnect.Data.Slice slice) [0x001a4] in <e78654be6f7946aebb4dc6b6fa374eb7>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in <b0e1ad7573a24fd5a9f2af9595e677e7>:0 ---> Python.Runtime.PythonException: NullReferenceException : Object reference not set to an instance of an object
at QuantConnect.Algorithm.QCAlgorithm.ProcessInsights (QuantConnect.Algorithm.Framework.Alphas.Insight[] insights) [0x0003d] in <e78654be6f7946aebb4dc6b6fa374eb7>:0
at QuantConnect.Algorithm.QCAlgorithm.OnFrameworkData (QuantConnect.Data.Slice slice) [0x001a4] in <e78654be6f7946aebb4dc6b6fa374eb7>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in <b0e1ad7573a24fd5a9f2af9595e677e7>:0
at Python.Runtime.PyObject.Invoke (Python.Runtime.PyTuple args, Python.Runtime.PyDict kw) [0x00033] in <c56ab175820d412caf052e079c2ab9ef>:0
at Python.Runtime.PyObject.InvokeMethod (System.String name, Python.Runtime.PyTuple args, Python.Runtime.PyDict kw) [0x00007] in <c56ab175820d412caf052e079c2ab9ef>:0
at Python.Runtime.PyObject.TryInvokeMember (System.Dynamic.InvokeMemberBinder binder, System.Object[] args, System.Object& result) [0x0003e] in <c56ab175820d412caf052e079c2ab9ef>:0
at (wrapper dynamic-method) System.Object.CallSite.Target(System.Runtime.CompilerServices.Closure,System.Runtime.CompilerServices.CallSite,object,QuantConnect.Data.Slice)
at QuantConnect.AlgorithmFactory.Python.Wrappers.AlgorithmPythonWrapper.OnFrameworkData (QuantConnect.Data.Slice slice) [0x0007a] in <f1078e33caab485c92dfcdad501667ef>:0
at QuantConnect.Lean.Engine.AlgorithmManager.Run (QuantConnect.Packets.AlgorithmNodePacket job, QuantConnect.Interfaces.IAlgorithm algorithm, QuantConnect.Lean.Engine.DataFeeds.ISynchronizer synchronizer, QuantConnect.Lean.Engine.TransactionHandlers.ITransactionHandler transactions, QuantConnect.Lean.Engine.Results.IResultHandler results, QuantConnect.Lean.Engine.RealTime.IRealTimeHandler realtime, QuantConnect.Lean.Engine.Server.ILeanManager leanManager, QuantConnect.Lean.Engine.Alpha.IAlphaHandler alphas, System.Threading.CancellationToken token) [0x0114d] in Lean.Engine.Alpha.IAlphaHandler alphas, System.Threading.CancellationToken token) [0x0114d] in <f7db25daa76942e79f8886b2a88d5c7b>:0
--- End of inner exception stack trace ---

Is this perhaps due to a stock that's not meant to be in the universe not having data?

I appreciate this is a very long post but I would be very grateful for any help with this as I've been bashing my head against a wall for quite a few days trying to get this to work!

Thanks,

Ollie

Update Backtest








0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


FYI, here are a sample of the logs:

Generating universe...
2003-01-03 00:00:00 : Universe consists of 40 securities
2003-01-03 00:00:00 : 6 securities to add
2003-01-03 00:00:00 : 6 securities to remove
2003-01-03 00:00:00 : Generating alphas...
2003-01-03 00:00:00 : 46 securities in ActiveSecurities
2003-01-03 00:00:00 : 40 securities in self.securities
2003-01-03 00:00:00 : Generated 40 insights for 40 securities
2003-01-03 00:00:00 : Generating targets...
2003-01-03 00:00:00 : Created a portfolio of 40 targets
2003-01-04 00:00:00 : Generating universe...
2003-01-04 00:00:00 : Universe consists of 40 securities
2003-01-04 00:00:00 : 2 securities to add
2003-01-04 00:00:00 : 8 securities to remove
2003-01-04 00:00:00 : Generating alphas...
2003-01-04 00:00:00 : 48 securities in ActiveSecurities
2003-01-04 00:00:00 : 36 securities in self.securities
2003-01-04 00:00:00 : Generated 36 insights for 36 securities
2003-01-04 00:00:00 : Generating targets...
2003-01-04 00:00:00 : Created a portfolio of 36 targets
2003-01-07 00:00:00 : Generating universe...
2003-01-07 00:00:00 : Universe consists of 40 securities
2003-01-07 00:00:00 : 3 securities to add
2003-01-07 00:00:00 : 11 securities to remove
2003-01-07 00:00:00 : Generating alphas...
2003-01-07 00:00:00 : 51 securities in ActiveSecurities
2003-01-07 00:00:00 : 34 securities in self.securities
2003-01-07 00:00:00 : Generated 34 insights for 34 securities
2003-01-07 00:00:00 : Generating targets...
2003-01-07 00:00:00 : Alpha insight weights don't add up to 1: 0.9999999999999999
2003-01-07 00:00:00 : Created a portfolio of 34 targets

 

1

Hi Ollie, 

Firstly, beautifully written code. It's great to see the framework used so well. 

Second: There might be something weird going on with using the daily data and using the expiry function end of the day. If using daily data and an intraday expiry it won't expire until the next day (as the data sets the "time" in backtesting). I changed it to minute resolution and the security additions look as expected.

Lastly, the exception was in the ProcessInsights method. I didn't know where that was so I searched Github, and found the method. This clued me into that this method is a general method that pipes the insights from Alpha -> Portfolio -> Risk -> Execution.

Once there I realized it might not be an issue in the Alpha model - so subbed out the portfolio and execution for generic ones. This worked so I put your portfolio construction back.

The only way to get an object is null error in Process Insights was if one of the targets was returning null. So I put some debugging on that. I reduced the universe to 4 assets to speed up my looping and testing.

677_1586441546.jpg

I saw one of the targets was returning as None and added a null check:

677_1586441620.jpg

Happy Coding! 

 

3

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed