QuantConnect Community

There is a bug in the source strategy: the benchmark SPY should not be invested. Here is the modifed version of original strategy as comparison. They should produce the same result.

Hi Huang Zhe,

The "bugs" and differences in performance are due to the poor construction of the original algorithm. We have created an updated version of this algorithm, which can be seen in the attached backtest.

Best,
Shile Wen

Hi Shile Wen,

Thank you very much for your excellent codes compared to which my original algorithm was indeed poorly constructed. However I did follow your documentation to intentionally construct it that way. The intentional reason is that I want to benefit from the Algorithm Framework to build seperate modules to improve risk management of the original algorithm. For this purpose, I need to first replicate exactly the same performance of the original algorithm with access to each model in the Algorithm Framework and then add and test different modules.

1. It is to my surprise that your updated algorithm has no execution model nor any exectution orders such as MarketOrder(), SetHoldings(), Liquidate() etc and it trades. This is different from what is suggested by the documentation, which is: to execute orders, Market/Stop/Limit orders or SetHoldings(), Liquidate() are mandatory; and to use Algorithm Framework, the Execution Model is mandatory. I didn't find any clue in the doc that the Execution Model in Algorithm Framework can be optional which is the case in your updated algorithm. I also checked the ConstantAlphaModel() (

) and the EqualWeightingPortfolioConstructionModel() (

) used in your updated algo but neither of them contains any execution codes. Therefore I can only think that there is some defult execution modules behind the scene to which I don't have access. 

2. I also find that your updated algorithm doesn't replicate the same performance of the orignal algorithm for the same period as I published in my first comment. The orginal net profit 332.724% and Sharpe ratio 0.78 has now become 164.850% and 0.594. Some major differences are also visible on the charts. Given that the original algorithm is quite simple and straight forward and I don't find any logical difference in your updated algorithm, it seems still not easy to replicate it using the Algotihm Framework. Would you please provide any reason for this?

3. Your updated algorithm mostly solved my problem but it also bypassed it as there will be no longer control of the execution process. My poorly constructed algorithm intentionnally and manually used all models in Algorithm Framework except the Risk Managment Model which is optional accroding to the doc. As I understand from the doc, ActiveSecurities contain the securites currently has holding or selected by UniverseSelectionModel. When the FineUniverse returns, QCAlgorithm compares its return to current ActiveSecurites to decide AddedSecurites and RemovedSecurites. Therefore if a holding security is not selected by the UniverseSelectionModel and gets liquidated, logically it should be removed from the ActiveSecurities. My problem is that it is not removed, and therefore when the UniverseSelectionModel selects it again, it does not appear in the AddedSecurites and therefore cannot be seen by the AlphaModel. I even tried to manually remove it after it was not selected and liquidated but it always stay in the ActiveSecurites which grows longer and longer. This is the main difference I found from the original algorithm whose ActiveSecurites keep in line with holding and UniverseSelection. Given that your updated alogrithm omitted the execution model and solved this problem, my hypothesis is that the inherited execution model contains some unaccessible mechanism which prevents the removal and that's why I regarded it as a "bug" which may be not. Do I understand anything wrong above or would you have any advice, please?

Your updated algorithm gives me much hints that cannot be read from the doc and I will also look for reasons why it still doesn't replicate the same performance. Meanwhile, I would appreciate if you could answer my 3 questions above, please. Thank you in advance.

Best

Zhe

Hi Zhe,

We think that it would be simpler to provide a better version that members can improve. For instance, it's not recommended to return an empty list of Symbol objects since it removes all the securities from the universe. Moreover, securities that are removed should be liquidated as soon as possible and it's not clear that logic with three booleans would do it well. Does self._changes.RemovedSecurities contains the right securities? Finally, only set holdings for added securities seems wrong: we should rebalance all holdings and not only use funds from removed securities. All those issues may be the source of the "bug" with ActiveSecurities.

1. By default, the ImmediateExecutionModel is set, consequently the combination of an alpha model and a portfolio selection model (PCM) is enough to place an order. The ImmediateExecutionModel calls MarketOrder for the targets created by the PCM and place them in an optimal way.

2. The new version has different results because set holdings were called for added securities only which is not right.

3. I think some securities are not removed from ActiveSecurities because the logic is not liquidating them when they should. You could move this logic:

to the top of OnData or to OnSecuritiesChanged:

 Please note that if the Universe Selection returns an empty list, all securities will be liquidated.

Hi Alexandre,

Thank you for your detailed explanation which makes me better understand LEAN.

0.1. The empty return of universe is used in the original algorithm of your tutorial strategy (attached in my first comment) which does not use Algorithm Framework. I thought AF uses the same logic which seems not to be the case now. Thank you for this important advice. Maybe QuantConnect can indicate in the doc several points to pay attention to when converting algorithm to AF?

0.2. Yes, self._changes.RemovedSecurities contains the right securities when a security is removed for the first time. But then since the same security cannot appear in AddedSecurities again, it won't show in RemovedSecurites, either.

0.3. "only set holdings for added securities seems wrong"
-- This is the logic of the original algorithm which produces better performance and my purpose here is to use AF to replicate it. So this logic is also be replicated using AF. Actually the original algorithm does not rebalance the untouched securities. It liquidates removed ones and uses available funds to set holdings for added ones. If the percentage in SetHoldings refers to that of available fund instead of portfolio value, then this logic should be correct. If not, then there should be insufficient fund error but the original algorithm works fine.

0.4. "we should rebalance all holdings and not only use funds from removed securities"
-- Yes but that will be a new algorithm, here I simply want to replicate the original algorithm using AF. I suppose that any algorithm can be developped using AF. If this is not true, then AF will limit production possibilities.

1. Thank you for this important information.

2. Yes, this is one reason. After reviewing the order history of the new version backtest, I found another reason that the PCM keeps rebalancing portfolio despite that Shile already set the rebalance to None. For instance, the new version liquidates holding securites around 17 or 18 of each month which should happens only in the beginning of each month. I cannot find more documentation about AF PCM. Only by reading the source codes of EqualWeightingPortfolioConstructionModel and using dir() in debug mode to guess and test, I still cannot find out why. Maybe QC can add more doc for methods in AF classes. For example, the doc only introduces CreateTargets() for PCM, but the EqualWeightingPortfolioConstructionModel contains a DetermineTargetPercent() method which is not documented. For the ExecutionModel, doc only introduces the Execute() method, but I see that in ImmediateExecutionModel there is an internal member self.targetsCollection and an external one OrderSizing which are not mentioned in doc at all. This may reduce productivity when using AF if user want to develep their own AF models because user won't use those hidden functions correctly by only guessing and testing without doc.

I copied source codes of EqualWeightingPortfolioConstructionModel and ImmediateExecutionModel into the algorithm and set the ExecutionModel to only deal with securites in the Added- or RemovedSecurites. This sovles the 17/18 liquidating problem. But instead of completely liquidating removed securites in the begining of month, it only partially liquidates. So if the same security is to be bought in the next month in the original algorithm, it won't appear in AddedSecurities of the modified new version. Therefore I still cannot achieve the same performance.

3. Previously I used flags to control liquidating because universe could return [] as that in the original algorithm. After taking your advice, I set universe to return Universe.Unchanged and the flags are removed. I applied your logic to liquidate removed securites on top of OnData(), OnSecuritiesChanged() of QCAlogorithm and ExecutionModel, respectively. But none of the 3 attempts succeeded, the old problem remains that once a security is universe unselected and liquidated, it won't be removed from ActiveSecurites and therefore won't appear in AddedSecurites and be rebought again. Please see the attached backtest. I think my poorly constructed algorithm mostly replicates the logic of the original algorithm only with this problem to be solved.

Will you please provide more advice?

Best,

Zhe

Hi Zhe, 

regarding 

>> 0.2. Yes, self._changes.RemovedSecurities contains the right securities when a security is removed for the first time. But then since the same security cannot appear in AddedSecurities again, it won't show in RemovedSecurites, either. <<

Why do you think so? I've added these code lines to test it:

And the Debug statements show that there are readded symbols (added, then removed and then added again to the universe) and "reremoved" symbols as well. (see attached backtest)

You might be right on all other points but then the cause will be somewhere else. 

However, this is an interesting thread and I hope Alexandre Catarino can help you. Basically, algorithms implemented with the Algorithm Framework style are much harder to debug as we are frequently switching between different modules/classes and the engine (LEAN) is written in C#. I guess we are not able to jump into the engine layer when debugging algorithms written in Python. 

Hi Arthur,

I tested your code and yes I do see removed securities being added again.

Previously I didn't test all removed securities and came to this conclusion by comparing the return of FineSelection, self.ActiveSecurities and self.AddedSecurites in debug mode. With your new finding, I would say that this problem does not apply to all symbols but it does happen to some symbols including those which made me have this conclusion.

I put this code in the end of FineSelection before the return:

and got outputs like:

2009-05-02 | active but uninvested symbols to add: ['BBD', 'MAR', 'CNX']

Take 'BBD' as example. It was bought on 2009-2-3 then sold on 2009-4-2.

On 2009-05-02, it is fine selected but also appears in self.ActiveSecurities with the attribute Invested==False. This suggests that when it was sold on 2009-4-2, it was not removed from ActiveSecurities. Then if I step out of FineSelection and step into OnSecuritiesChanged(), 'BBD' does not appear in changes.AddedSecurities and therefore cannot be passed onto AlphaModel when it should be rebought. Therefore I still think this is the reason that causes the inconsistency.

Then I looked at debug log using your code:

2009-10-02 | readded symbols: ['TJX', 'PCU', 'AEO', 'GFI']

This list constains symbols (e.g. 'TJX') that generate this error:

"The security does not have an accurate price as it has not yet received a bar of data...".

They couldn't be and were not traded and were successfully removed from ActiveSecurities when they were not FineSelected in the next month. Therefore they could be readded when they get fine selected again.

So I think the updated problem is: if a symbol has been traded, it won't be removed from ActiveSecurities which gets longer and longer all the time and prevents its listed symbols from being added and traded again. 

As it is impossilbe to debug between modules as you said, I can't see what happens when a symbol is liquidated and why it does not get removed from ActiveSecurities. My guess is that there is something to do with self.targetsCollection.ClearFulfilled(algorithm) in ImmediateExecutionModel because that seems to be a key difference from my ExecutionModel which does not need to calculate OrderSizing.GetUnorderedQuantity(algorithm, target) and therefore doesn't use self.targetsCollection.ClearFulfilled(algorithm). It might be a reigister to which LEAN checks before removing security from ActiveSecurities in Algorithm Framework. But this method is not documented so I can only guess.

Besides, I find that many methods or members in QCAlgorithm are not ducumented like these above, making it quite time consuming because I need to guess and test their usage.