QuantConnect Community

Goldie Yalamanchi,

The main difference in the luc prieur code is that he applies the momentum filter first and the fundamental filter second.
To solve the leverage problem you should calculate stocks and bond weights once per bar in the same place.
But let's talk about that in the "Amazing returns = superior stock selection strategy + superior in & out strategy".

Very much enjoying the discussion here; we are now officially at 200+ posts (plus the 100+ from Quantopian). A big thank you again to QuantConnect for hosting this discussion and allowing us to backtest strategies/ideas, plus the helpful comments and suggestions from the team (Jared Broad, Derek Melchin, Shile Wen)! To the thread's discussion participants: amazing work, keep it up! Very nice work on multiple in & out algos and looking beyond (stock selection, alternative assets selection, out-of-sample testing - possibly all warranting individual threads at some stage). Very interesting ideas, all!

Hello,

Would anyone be able to in some detail tell me what these 5 snippets are trying to accomplish?

Any help would be great!

1.     def rebalance_when_out_of_the_market(self):
        # Returns sample to detect extreme observations
        hist = self.History(
            self.SIGNALS + [self.MRKT] + self.FORPAIRS, 252, Resolution.Daily)['close'].unstack(level=0).dropna()
        hist_shift = hist.apply(lambda x: (x.shift(65) + x.shift(64) + x.shift(63) + x.shift(62) + x.shift(
            61) + x.shift(60) + x.shift(59) + x.shift(58) + x.shift(57) + x.shift(56) + x.shift(55)) / 11)

        returns_sample = (hist / hist_shift - 1)

2.        # Reverse code USDX: sort largest changes to bottom
        returns_sample[self.USDX] = returns_sample[self.USDX] * (-1)
3.       # For pairs, take returns differential, reverse coded
        returns_sample['G_S'] = -(returns_sample[self.GOLD] - returns_sample[self.SLVA])
        returns_sample['U_I'] = -(returns_sample[self.UTIL] - returns_sample[self.INDU])
        returns_sample['C_A'] = -(returns_sample[self.SHCU] - returns_sample[self.RICU])    
        self.pairlist = ['G_S', 'U_I', 'C_A']

4.        # Extreme observations; statist. significance = 1%
        pctl_b = np.nanpercentile(returns_sample, 1, axis=0)
        extreme_b = returns_sample.iloc[-1] < pctl_b

5.        # Determine waitdays empirically via safe haven excess returns, 50% decay
        self.WDadjvar = int(
            max(0.50 * self.WDadjvar,
                self.INI_WAIT_DAYS * max(1,
                                         np.where((returns_sample[self.GOLD].iloc[-1]>0) & (returns_sample[self.SLVA].iloc[-1]<0) & (returns_sample[self.SLVA].iloc[-2]>0), self.INI_WAIT_DAYS, 1),
                                         np.where((returns_sample[self.UTIL].iloc[-1]>0) & (returns_sample[self.INDU].iloc[-1]<0) & (returns_sample[self.INDU].iloc[-2]>0), self.INI_WAIT_DAYS, 1),
                                         np.where((returns_sample[self.SHCU].iloc[-1]>0) & (returns_sample[self.RICU].iloc[-1]<0) & (returns_sample[self.RICU].iloc[-2]>0), self.INI_WAIT_DAYS, 1)
                                         ))
        )
        adjwaitdays = min(60, self.WDadjvar)

Peter Guenther,

I performed out sample testing of 4 "In-Out" algorithms.
All of them using 12 input sources, complex methods to calculate momentum and outdays.
Exit signal generated as union of 8 individual signals.

In_out Tentor Testivis October 2020,

In_out_flex_v4 2020-11

In & Out ‒ parsimonious trades edition 2020-11

In & Out ‒ parsimonious trades edition Update (2nd) 2020-11

Was everything as intended in November?
What do you think?

@Vladimir Very interesting, but I should note, that to be able to draw statistically significant conclusions, you need many years of out-of-sample data. This is why you should preferably optimize the strategy on about 67% of the available data, and then validate it on the remaining 33%, which in this case would imply the  period 2016-2020. Even in that case the data available and the market conditions therein pretty limited. In any case the current algorithms have been optimized up till this year. This and the above reasoning is why a cross-validation method is the only reasonable method to perform out-of-sample testing in this case. The downside of course is, that you need some objective parameter optimization algorithm to prevent bias in the selection of the parameters. 

Vladimir: Thanks for running these tests and sharing the results!
"Was everything as intended?" -- It may sound funny but I would almost say "partly yes". I think the algo does capture "jitter" in the market, see some of the recent business headlines (surging COVID cases, new lockdown measures in multiple countries, concerning US employment numbers, questions re whether the S&P500 surged too high too quickly, questions re tech overpricing etc.). However, did the algo outperform the market? There the answer is clearly "no"; SPY and QQQ made about 10% in Nov. The algo was completely in bonds for most of the month, with an initial uptick at the beginning but, overall, a zero return (see Nathan Swenson's recent insights in Amazing returns = superior stock selection strategy + superior in & out strategy).

Goldie Yalamanchi was crunching the signals (see recent post in Amazing returns = superior stock selection strategy + superior in & out strategy) and found that the SHY signal does not add much in terms of total returns over the backtesting period from 1/1/2008. Yet, the SHY signal was the main reason for that the algo was out in Nov. Based on Goldie's finding, attached is the algo without the SHY signal and with a daily (instead of weekly) 'in' reschuffling (some people may be running the algo with this setting, i.e. removing the 'lazy equity trader approach). I still have to test this setting over the full backtesting period. For Nov, the attached backtest suggests that this setting does a better job (i.e. setting = remove SHY signal in lines 41 and 51, schedule 'in' reschuffling to daily instead of weekly in lines 71-72).

Peter Guenther,

Here is my opinion sent to you 6 weeks ago.

And now you're using 8 factors.
Think about it.

Peter Guenther,
Every strategy goes in and out, but this strategy goes in and out according to the waiting day rule.
Asking "Was everything as intended?" I mean, the strategy must wait at least 15 days after the exit signal before entering. 

    if self.dcount >= self.outday + adjwaitdays:
            self.be_in = True

But instead it made 10 trades in November.

Peter Guenther,

Here are some "Rules of the algo" from your post on Quantopian forum

Are they still active in current version of the algorithm?

Vladimir,

I will wait for Peter Guenther to answer. But from my debugging the "adjwaitdays" isn't always 15 days. I have seen it be 20 days or 24 days, etc.

If we wanted to always be 15 days... then this below calculation isn't needed...

 Instead I guess we would just hard code self.WDadjvar = 15, no? The above self.WDadjvar is based on how long the decay 50% value of the indicators is.

 
 

Just had the algo fail with an error, probably trying to enter a position:

Runtime Error: LiveTradingRealTimeHandler.Run(): There was an error in a scheduled event EveryDay: QQQ: 120 min after MarketOpen. The error was KeyError : 'close' Stack Trace: System.Exception: LiveTradingRealTimeHandler.Run(): There was an error in a scheduled event EveryDay: QQQ: 120 min after MarketOpen. The error was KeyError : 'close'

Hi folks... moved over from quantopian.

Here is my input to this algo; 

- IN-OUT signal unchanged

- IN and OUT assets determined by momentum (IN: QQQ/IWF, OUT: TLT/IEF)

Wow T.Smith!  Those are some awesome results and you did that with non-leveraged instruments.  Very Nice!

T.Smith, if you check the logs, there are some invalid orders due to lack of margin.  I see the order quantity is just slightly too high and the order gets rejected. 

Yes I noticed the same thing... I think QQQ needs to be sold first before IWF (Russell) is bought.

2020-09-11 11:31:00    TLT    Sell Market Fill: $164.259732735 USD -1048 Filled 
2020-09-11 11:31:00    QQQ    Buy Market Fill: $273.40 USD    632    Filled 
2020-09-25 11:31:00    IWF    Buy Market 810    Invalid 
2020-10-02 11:31:00    IWF    Buy Market 809    Invalid 
2020-11-06 11:30:00    QQQ    Sell Market Fill: $291.99 USD    -632    Filled

That small detail aside, it seems like an interesting iteration that still continues to use the SHY signal but has properly entered the market again in late half 2020 without issues. Something the last iteration of the algo has suffered from.



 

I may stand corrected, after applying the fix to sell QQQ before IWF buy... the algo proceeds along. But doesn't make a trade after 10/6 again. Just stuck in bonds (IEF).  Take a look below, maybe I made a mistake... but here is my backtest....

Goldie Yalamanchi are you saying there is an issue with the algo? I don't understand, or you want it to be more active in trades? I would prefer it to be in bonds, if you've noticed QQQ hasn't done much and looks like it's getting ready to roll over.

Yes, I agree.  Prefer bonds right now, although entry was too early with 10/6.  The maximum out time is 60 days if I'm reading code correctly, which would be 3 months.

So you can review the discussion around SHY. But it fires the signal over and over again due to SHY moving %1 or so. Bond yields are near 0%. TLT and IEF do well when the fed rate is low.   But they can't go lower than 0% can they?  Hence I believe the SHY signal to be a FED created signal. I guess they can go to negative % which has happened in some countries.

Here is a log of the signals.... if any are true it exits the IN state and switches to bonds. In 2020 this happens....

2020-10-06 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False
2020-10-07 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False
2020-10-08 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-10-09 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-10-12 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-10-20 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-10-21 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-10-22 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-10-23 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-10-26 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-11-03 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-11-04 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-11-06 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-11-10 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-11-11 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-11-12 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-11-13 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False
2020-11-19 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-12-02 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False

We have in other variations of this algo, removed SHY as a signal.  Again it is debateable because since Nov QQQ has had a nice move but the algo sat on the sidelines. I have removed SHY from the algo and it performs very nicely still.  Here is your algo minus the SHY signal....

Goldie Yalamanchi the removal of SHY seems to have icreased DD a lot, think the original was around 16-18%, now you're closer to 23%

This is an original version the one i've been running live, but for some reason today it failed with the above error message i posted before. 

No doubt, the SHY signal WAS very valuable in the past from 2008 or even 1999+ to prevent drawdowns. The only question is will it continue to be. Again defer to the conversation on the FED and rates, etc. Your opinion may vary. Obviously you may wish to keep the signal. Yes I think the above code before we added that fix... was just trying to buy IWM before it sold QQQ, so that is resolved now right? This below code I put in one of the fixes to address that.

These changes below they were to ensure that sells happen before buys. And in the case of where QQQ was switched to IWF... the code in the trade() handles to liquidate first.

Goldie Yalamanchi cheers for the trade logic!...

Regarding the want to drop SHY as an indicator. I like the fundamental idea behind having short term bonds as an indicator and believe wishing to remove it would be a poor decision. Doesn't it defeats the point of having a deterministic strategy if we edit it based upon present unclosed trades?

You maybe right T Smith but we have thought that rates are too low and SHY is triggering over and over again. Also going forward if rates are like 0% how can TLT really make a move up? Bond funds borrow from treasury at 10 or 20 year yields and lend out at higher % to make money right? So if they are at near zero which is a move that happened from 1990 all the way to 2020 to get to 0%, how can the bond yield be used as an indicator.

Very welcome to keep SHY as an indicator. I was just discussing the merits of removing it. It will be interesting to see how the market moves forward. Either the rates go the less than 0% which has happened in Germany and pretty much sovereign funds of all developed countries.  Maybe finally they will go back up and there will be rates to cut? There simply isn't enough tax revenue to do that at the moment. Idk, next market crash maybe bonds and stocks will both crash.

Goldie Yalamanchi not sure I posted the algo above that is the code I have supposedly if i remember correctly fixed any order logic, but you can confirm.

I totally agree with you there will come a time when both bonds / stocks will crash, it's just we don't know when or where. That is why I brought up the idea of implementing a tail risk strategy like Taleb. 

Quickly sharing some sensitivity results: Bootstrapping
For the past couple of weeks, I was playing around with the idea of using bootstrapping to test the In & Out algo’s sensitivity to possible futures that do not play out exactly (sometimes not even remotely) like the past, a concern that I reckon is present for any trading strategy that we create. Thus, the test may have some applicability beyond our In & Out discussion here. Specifically, bootstrapping can indicate how much an algo’s total return depends on the chronology and idiosyncratic events of the past, i.e. the realization that we use in our backtests. Happy to discuss the test and how it may be expanded; any questions/comments fire away, as usual.

Bootstrapping results for the In & Out
The following chart shows the distribution of the In & Out algo’s total excess returns for 5,000 fundamentally different stock price data scenarios that each span about 13 years (i.e., equivalent length as the time period that we use in our backtests from 1 Jan 2008 to today). The algo holdings are: In = SPY and Out = TLT/IEF. The displayed total excess return (x-axis) is the algo’s total return minus the SPY total returns. An excess return of zero (i.e. the zero vertical line in the chart below) means that the algo performs at par with the market (SPY).
On average, the algo yields a positive excess return (~450%) across the 5,000 different data scenarios. This looks promising since the data scenarios can be extremely different from the past (see the technical discussion below). In data scenarios that are unfavorable for the algo, the algo underperforms the market, which happens in about 10% of the data scenarios. This finding may help to measure risk regarding using the algo for investing. At the bottom 10% percentile, the underperformance is about 30% below the market over 13 years or about 2% below the market per annum.

Technical explanations and discussion
The algo is the In & Out percentile version including the signals PRDC, METL, NRES, DEBT, USDX, the pair comparisons GOLD vs SLVA, UTIL vs INDU, SHCU vs RICU, and the flexible waitdays approach. We bootstrap via drawing random monthly segments, with replacement, from the original timeseries of returns (1 Jan 2008 – 30 Nov 2020; all relevant assets i.e. SPY, signals, pair elements). Using monthly (e.g., vs daily) segments may preserve some of that evolving character of stock prices which are not completely random from one day to the next. Drawing one bootstrap sample gives us one randomly collated new returns series spanning about 13 years (i.e., a possible future). The returns series is then converted to a price series so that the algo can make its decisions as usual. Theoretically, this series could consist of about 155 repetitions of Jan 2008 or any other month; very unlikely, but not impossible. Yet, almost always it is a random mix of months such as Nov 2012 being followed by Mar 2008, being followed by Jun 2020, and so on. When an algo’s performance strongly depends on the past chronology of events, its returns will break down as we break the chronology via random sampling. When an algo’s performance strongly depends on a few favorable (for the algo) idiosyncratic events and patterns in the timeseries, random sampling will tend to make the algo’s performance break down in many of the bootstrap samples, showing up in the distribution chart via many below zero results. For each bootstrap sample, the algo’s decisions are simulated exactly how we would perform them if the bootstrap sample was the actual (a possible future?) price series.

Despite the fact that I don't completely agree with the latest development of "IN OUT"
it was nice to play with "Dual Momentum IN OUT".

Thanks to T Smith.

Enjoy.

T Smith: Echoing Vladimir's post, very well done and thanks for sharing this Momentum/In & Out strategy! This is interesting insights for all, particularly I think Joseph Kravets was musing about a rotation strategy between multiple ETFs. Great stuff!

Thanks Peter! I am starting to think that this could be overfitted but it seems effective. 

To throw a wrench into the conversation, you'll notice that in 4/2013, 7/2020, and a few dates in 2015/2016 (I forget which), you'll find that the algorithm switches out of in_market stocks to out_market stocks. My perspective might be a little contraversial, but I want to argue that the reason why SPY continued going up during these times despites multiple signals firing to exit the market is that the government and/or financial institutions were pumping the market - in 4/2013, it's the start of the economic recovery from the housing bubble, and the federal government allowed/incentivized banks to sell foreclosed houses for well below market, and in 8/2020 it was announced that 3 financial institutions were intentially pumping up the market (Nasdaq, specifically) to continue to go up.

It's just a thought and a different perspective to contribute to the group.

Here is my DUAL MOMENTUM-IN OUT v2

  Chak: Absolutely. I am wondering whether there might be an additional (hidden) indicator here that could be useful to overrule the algo's 'out' indication in certain situations. Eg similar to the notion of "don't fight the Fed", for which an indicator might be the size of the Fed's balance sheet or, ideally, the announcement of a planned expansion of the Fed's balance sheet (= pumping money into the market). Maybe such data could be matched in via a custom factor/uploaded data or via news scraping.

@Peter Guenther

Interesting results on the bootstrapping tests. It should be noted however, that bootstrapping in-sample results doesn't provide information about out-of-sample behaviour, unless the distributions underlying the in-sample results are representative out-of-sample. Bootstrapping is a resampling technique, where the observed samples are assumed to come from an independent and identically distributed population. The statistical properties of this population are estimated from the observed samples. However, in the event the original model is overfitted, the statistical proprties estimated from the model results are not representative for the out-of-sample population. So, while the bootstrapping will provide information about the model's sensitivity to the order of events, if the model is good, it will not provide such information for future events, if the model is bad.

Peter Guenther yes that must be figured out. All the variations of the algo are great, but those that include SHY as a signal have effectively stopped re-entering the market remaining in `OUT` state since 10/6/2020 and the SP500 has made a 9% gain since then, while the algo has sat out for almost 70 days. Granted there was a lot of chop, but I wonder if the algo will even trigger to back 'IN' state this year or for months into next year. There is another FOMC meeting this week, generally they will say.... we will keep rates low and SHY will keep the algo OUT (my prediction of course).

Goldie Yalamanchi: Yep, the SHY is a tricky one. Technically, it is not the Fed's low/zero interest policy/cuts that is making the SHY drift down (the algo is looking for that downdrift that falls into the bottom 1 percentile over a three month window). When the SPY drifts down it means that the market expects an increasing interest rate (= that the Fed raises the rate). A falling bonds price means a larger interest yield. The algo interprets this as a bad thing = increase in the costs of capital. And usually this would be a bad thing for equities. However, the current root cause behind the increasing interest rate expectations is actually something that is good for equities: the market expects the economy to rebound due to the latest Covid vaccine developments. So, we have two forces at play here, a positive (economic rebound) and a negative (economic rebound causing the Fed to increase rates at some stage increasing the cost of capital). Currently, the positive outweighs the negative. Yet this can flip when the market feels that the rebound (i.e. the positive side) is sufficiently priced in.

I reckon if we could disentangle investors' improving economic outlook, we might have a counter-signal that could be used to mute the SHY signal in certain situations in which it should actually be interpreted as something that is the symptom of a positive development.

Peter Guenther,

The latest IN-OUT strategy has 12 information sources.
The logic of decision-making is
if any one of your 12 top advisors says "out" but 11 others say "in"
the strategy should go "out".
Why you choose that rule?
Why one, not seven what is more logical?
Where in human activity you have seen such decision-making rule?
Think about it instead of looking what is wrong with "SHY".
Technically, there is nothing wrong with "SHY" chart.
The problem is in decision-making rule.

Hey Peter Guenther

Tried running this live but it doesn't enter trades, commented out warmup period, set self_bein to 1

Elsid Aliaj: One possible explanation: when the in_out indicator (self.be_in) is 1 (= says "in"), the algo waits until Friday to actually go in ('lazy trader approach'; I only reshuffle/enter equities Fridays). This happens via the second scheduling function (line 74 where it says "self.DateRules.WeekEnd()"). Another possibility is that the in_out indicator says "out" but we are past the scheduled time for the out reshuffling, which in your code happens every day at 10:35 (9:30 + 65 minutes; lines 66-70). Also in this case the algo would do nothing today and only trade tomorrow. Not sure whether this helps to identify the issue (?).

I'll hint on several things that can be done. Peter introduced the concept to quantconnect, I personally would like him to submit it as an alpha. Here are a few things I've tried:

1) If you look at the IN-OUT Plot function, it is possible to disentangle each signal's individual contribution to the algorithm's decision to exit the market. I see at least 3 ways each signal can contribute:

#1 in the extreme_b filter function that sets a hard boundary

#2 the individual contribution of each ETF to their signal

#3 in the covariance matrix to assess whether the recent signal falls with the 1%.

If you create a second plot similar to IN-OUT but have each ETF represent a line, you can see these weights.

2) if you were to omit other signals besides SHY and at each level in the decision-making process mentioned in #1, you'll find find something interesting.

3) as everyone have mentioned, yes, the model is overfitted. What can we do to resolve model overfit issues besides simply removing variables based on PSR, Sharpe Ratios, and return %?

If we can solve the below two issues, we'd be a step closer to a better model:

#1 Are there methods to set the weights of each ETFs' impact on both the 'IN' and 'OUT' strategy, while addressing the model's flexibility to adjust to future events (i.e., out of sample data)?

#2 Can we further use these weights to create a more refined strategy to include different IN/OUT Market scenarios? For example, are there instances there it's better to use TQQQ instead of QQQ when going back into the market, or to use GLD/UGL and TLT/UGL when going out of the market or you're already out of the market?

Peter Guenther Ok i'll just use the prior version i was using, but this line threw an error last time, not sure if anyone knows what the issue is, specifically close.

self.SIGNALS + [self.MRKT] + self.FORPAIRS, 252, Resolution.Daily)['close'].unstack(level=0).dropna()

Maybe it didn't have the closing data of QQQ that day? But it threw the error during 11:30 when it was calculating whether to stay in or out.

Elsid, I was running live and rather than throwing an error, it simply disabled.  When I logged in I simply had nothing running live.  I'll have to check if there's a way to check for errors.

Elsid Aliaj and Nathan Swenson, thanks for raising the issue! From Elsid's error message it seems that the history call did not result in the usual table with a 'close' column (containing the closing prices). Yet, not entirely sure what happened. Derek Melchin probably would know directly. He has re-written the code using a consolidator logic and this might be a way to avoid the error (?).

Not sure whether this can solve the issue but the consolidator version is in the thread Amazing returns = superior stock selection strategy + superior in & out strategy and search for Derek's message "To assist with development of this algorithm, I've added a manual warm up and consolidator for the lookback window. This improves the execution efficiency as we avoid daily calls to the History method." Yet it's probably best to wait for Derek's comment since there may be specifics concerning live trading.

Elsid Aliaj: Just attaching here the latest version of the In & Out that I got. Much indebted to Derek and other contributors. It includes the consolidator approach for greater efficiency and code that ensures that sells are submitted/filled before buys. Can't promise that this will address the live trading issue but the features may be useful nonetheless for further testing and live trading.

Peter Guenther thanks i'll give it a go let you know if any issues arise

Strategy when back into equities today (12/18).  Prior trade was 10/6.

First crack on disambiguating the DEBT signal
Following up on DEBT (SHY) possibly being an ambiguous signal; see above the notion of an expected economic rebound causing investors to anticipate the Fed to increase rates at some stage, raising cost of capital expectations. We then said “if we could disentangle investors' improving economic outlook, we might have a counter-signal that could be used to mute the SHY signal in certain situations in which it should actually be interpreted as something that is the symptom of a positive development”.

Attached is an implementation that uses increasing input prices (METL and NRES, above median) to mute the DEBT signal.

Chak: Good set of questions and insights there, keep on hacking ;)
Nathan Swenson: Thanks for the live trading update! Fingers crossed that the market does not go down, now that the algo has decided to go back in. Then it would have gotten the rhythm of this cycle wrong.

Nice work Peter!  This new version especially excels with 3X funds.  I see 71% YoY return using TMF and TQQQ.

In_out_flex_v5_try

an implementation that uses increasing input prices (METL and NRES, above median) to mute the DEBT signal.
Excellent example of a curve fitting.

I found this thread a few days back and have spent several hours since then poring over the archived Quantopian thread and all 250+ replies in this thread. As someone new to quant finance coming from a software engineering background, the discussion alone has taught me so much. Thanks Peter Guenther for having this conversation open-sourced :) and I've learned a lot from Menno Dreischor's discussion about out-of-sample testing.

I'll be following this discussion and hope to contribute, though I'd want to start by validating the algorithm vs hacking returns assuming the alpha is valid. Kevin Davey's book has some interesting directions to take out-of-sample testing that might supplement the work Menno has already done. Cheers!

Austin: Welcome on board and good effort working your way through the archive and the thread here! Will be great to hear about your insights.

Additional musing about signal ambiguity
I reckon for the DEBT signal we were a bit too late in determining its ambiguity. So, I was wondering whether it might be possible to get ahead of the curve by directly musing about additional signal ambiguity. Below is a first brainstorming regarding the signals, their interpretation in the algo, and thoughts concerning possible ambiguity (i.e. that something that is interpreted as a bad thing may sometimes be a symptom of a positive development). The table below suggests three ambiguous signals, see the note "ambiguity, address".
For those looking for a slimmer algo, this might be an opportunity to try and drop these signals.
For others, this might be an opportunity to think about ways to disambiguate the signals.

Table. Signal ambiguity

Note: If your main concern is overfitting at this stage (and before), that is fine. Skip this post and focus on posts concerning testing and distilling signals. There will be posts coming up along these lines and we may achieve a certain level of convergence again later.

Second crack at disambiguating signals: the GOLD pair
We noted that gold can be an inflation hedge, such that the gold price increases when inflation expectations increase (see Table. “Signal ambiguity” above). However, inflation expectations tend to rise when the economy is expected to do well, making a rising gold price a symptom of a positive development in this scenario. This makes the GOLD pair signal ambiguous, since the algo interprets a rising gold price as a bad thing (= flight to safe haven = “market jitter”). The attached algo uses an ETF measuring inflation expectations (RINF, see line 45) to re-assess the GOLD pair signal (see lines 121-125). The logic: To rule out that the gold price increase is due to rising inflation expectations, we only react to the GOLD pair signal when changes in inflation expectations are below median.
Since RINF only started in Jan 2012, I had to catch this via a try ... except construction.
However, RINF could be manually constructed by comparing treasuries with inflation-protected treasuries. I have to explore that option a bit more and may give it a go later unless somebody else might be faster (?) ...

   Looks like it entered QQQ on 12/18 is this correct? Nathan Swenson  Peter Guenther

Yes, Elsid, entered QQQ on 12/18.

Yes that is a very very late entry into QQQ after such major move up.  I would just like to add that I have seen a lot of variations of the In/Out algo and the one change that really makes the algo react better without any significant increased drawdown (not using leveraged TQQQ, only QQQ or stocks universe) is to change the self.INI_WAIT_DAYS to 2 as pointed out by Guy Fleury 

self.INI_WAIT_DAYS = 2

This makes the algo in the March 2020 covid crash re-enter and stay in since about 4/6. And while it does get a little whip-sawed during the crash, the drawdown was only 15.5% I think one of the lowest drawdowns that I have seen in all the variations of the algo published here.  Also, it leaves in tact the SHY indicator as is.

Goldie, in that last run example you have SHY commented out.  

I agree that the entry for IN was far too late.  The algo needs to react more quickly.  There's no way I'm holding TQQQ this late in the move.  I am holding some regular equity funds and will follow algo with those. 

 Nathan Swenson thanks for picking that up.

Still, I have corrected that  (commented it back in) and it now stays in the market from Aug - Dec in the QQQ's... as long as 

self.INI_WAIT_DAYS = 2

I don't know what you guys are really trying to achieve, but when changing it to 2 days from 15, sharpe ratio plummets. CAGR drops and DD goes from 16.5 to about 22%

Wow....quite a bit of progress from Quanopian on this one. I am still getting my head around QC framework.

Can a kind soul care to explain below lines of code:

Goldie Yalamanchi , Nathan Swenson , and Elsid Aliaj: I tend to agree with Elsid and would not reduce the wait days too much. What I take away from Guy's table (reported elsewhere) is that the performance is relatively stable from 5-15 wait days, so there might be some scope to play around with reducing the wait days in this range. I would not go down to the 1 or 2 days, although the returns in the backtest look better. I would rather go to where we achieve some stability in the sense that results look relatively similar over a range of a parameter's values. I would also assume that by reducing the wait days we become more dependent on the return flips and that they provide precise signals in terms of pulling the wait days up when it is beneficial to do so. When these signals are less precise in the future compared to the backtest, we might be ending up with too few wait days. I will also try and run some tests to get a feel for what the new dependencies are ...

Performance from a full 2008 backtest actually goes down. And DD sky rockets to 24%. 

Maybe if you backtest just 2020 the numbers look better but the whole picture looks much worst. 

"What I take away from Guy's table (reported elsewhere) is that the performance is relatively stable from 5-15 wait days, so there might be some scope to play around with reducing the wait days in this range. I would not go down to the 1 or 2 days, although the returns in the backtest look better."

I agree, if the QC Optimizer wasn't pay-as-you-go I'd have run a couple of optimizations over the many parameters involved in this algo to look at stability and out-of-sample viability. As far as goals to measure against, parameter stability seems MUCH more important than backtest return % over many years. That is, as long as one's goal is to create a model that can describe the future, not soley describe the past. Jared mentioned in a different thread that QC is working to make the Optimizer feature work with nodes you're already paying for, and after that feature is pushed in ~10 days I'll run more tests and share my findings here...

Peter Guenther, if memory serves I remember you asking what market phenomonon this In/Out algo appears to be harnessing to capture such nice backtest results. Did the community ever come up with a market phenomonon this algo is exploiting, or is this experiment soley focused on the backtest?

In the mean time I'm working on a futures strategy, and since I've loved the discussion around this algo so much, I'm planning on sharing it if I can get it working / add value to the community :) these open sourced conversations are great!

Austin: Looking forward to your analysis results! This will be very interesting in terms of seeing the sensitivities and where there are any flimsy parts that need fixing, further development, or a more substantial overhaul. The futures version that you are working on also sounds very exciting and thanks for planning to share it! I am sure the community will love it and I fully subscribe to your positive note regarding having these open algo conversations, they are great.

Yes, I believe that you are right: currently, the source of the returns is not fully understood regarding its specific mechanics. I reckon a working hypothesis is that the algo signals pick up ‘jitter’ in the equity market that translates into a boost in bond prices. What was driving investors’ jitter often may not eventuate. Nevertheless, ‘riding the jitter wave’ in bonds vs remaining invested in equity seems to pays off.

Peter Gunther,

I'm sure you won't answer this, my last post on this thread, just as you haven't answered
many of my previous ones.
I showed you where are the flimsy parts that need fixing, further development,
or a more substantial overhaul.
In the United States, only one person has veto power.
You have given this right to any of the 8 sources who are not fully understood regarding its specific mechanics  and are therefore unreliable.

Happy curvefitting!

Rock247it 

Can someone get my name off this??? How do I remove my name???

On the upper right corner of the thread there is Follow Discussion bell, click on it.

I tried and it's working for me.

Impressive burst of passion there, always great to see, and, I reckon, a useful reminder of the several streams/camps we have in this thread with, sometimes, very different views of the world. For further reference, a list (~ chronological order):
1. Engineers – type “Rich data modelists”
2. Engineers – type “Coding experts”
3. Engineers – type “Parameter minimalists”
4. Live traders
5. Sensitivity assessors
Looking forward to these varied streams/camps’ further contributions, although I must say that I am a bit worried about our “parameter minimalists” fraction, but we shall see.

Stream: Parameter minimalists

Since much of the discussion lately has been focussed in the Rich data modelists stream, here is a contribution to our Parameter minimalists stream. Specifically, this is a great algo that Dan Whitnable posted on 22 Oct 2020 (see the Quantopian archive). Dan noted that it builds on ideas by Tentor Testivis and others (see his original notes). The algo distils a bear signal by comparing three return pairs. Let’s call the algo the “Distilled Bear” in & out algo; bear in a positive sense = strong. The algo is quite efficient in terms of total return per parameters used.

The Distilled Bear in & out algo
Holdings: In = QQQ, Out = TLT/IEF
Backtesting period: Jan 1 2008 to Dec 18 2020
Total return: 1843%
Sharpe ratio: 1.77; Max drawdown: 14.1%; Compounding annual return: 25.7%

Peter Guenther -

What would be interesting is to see the distilled bear combined with the Momentum Strategy with Market Cap and EV/EBITDA' strategy introduced by Jing Wu, 6 Feb 2018. 

 Peter Guenther have you tried to backtest the strategy between 1/1/2012 and 31/12/2019? What I found is that the strategy shines between 2008 - end 2012, and then again in 2020. Otherwise underperforms a simple buy-and-hold of QQQ (or SPY). The main reason for the great returns over the 2008-2020 timeframe seems to be those two periods....

I think it depends what you hold during that period Damiano Bolzoni.  If you hold 2x Treasury 7-10 yrs and 20 years, you'd outperform. The interesting aspect of 2x or3x treasury holdings is simply, we don't ever expect the price to goto zero, much like TQQQ, SSO, or SPXL would.

Thunder Chicken I'm running the strategy as posted here, so I will hold whatever the strategy tells to hold...the eyepopping results posted here are backed by 2 timeframes, and otherwise the strategy underperforms a simple buy-and-hold (with the same asset)

Damiano Bolzoni: Thanks for sharing these results regarding testing subsections of the backtesting period. Great stuff! I reckon you tested the In & Out, latest version? I recall that Radu Spineanu, back then on Quantopian, was running tests from 2012 onwards and also mentioned subdued performance. Not sure whether you have these readily available, but if so, would you be happy to share some numbers so that people can get a feel for the performance differences (e.g. the period you mentioned, 1/1/2012 and 31/12/2019, and then: In & Out (e.g. holding QQQ) +XXX%, SPY +XXX%, QQQ +XXX%)? That would be very cool.

Irrespectively, I see it the following way: For me, the In & Out is an insurance package that I take out to protect my nicely curated equity stock portfolio from steep drawdowns and then hopefully even benefit from a sell high-buy low type of scenario. Yet, such drawdowns are rare. During calm times, I pay an ‘insurance premium’ that, like any insurance, doesn’t really pay off but instead costs money when nothing super-bad is happening. The insurance premium are losses and/or missed gains due to nervous moves in an out of the market as soon as signs of market stress occur. The algo has multiple radars running (signals) to assess market stress and, hopefully, capture it early via the multiple radars.

The problem is, and this could be an interesting path to explore, that we do not know when to switch our radars on versus off and just be invested in equity without the insurance package and associated costs.

It’s also important to note that, at least in my personal view, the In & Out is not necessarily intended to make money in and of itself as a standalone strategy. This is where a great stock selection strategy is required which hopefully generates returns above the market in times without steep drawdowns. The In & Out is a (possibly costly) guard of our amazing stock selection’s returns. Your data will be useful to assess how costly this guard is. Maybe others are cheaper, for example The Distilled Bear? This will be very interesting to explore further.

Damiano Bolzoni - You'd be correct.  It is near impossible to beat a bull market. Most hedge funds and mutual funds do not beat a straight bull market, by any appreciable amount. I have attach the backtest for the period you indicated.

Strategy Net Profit:  314.710%

Strategy Probabilistic Sharpe Ratio: 1.728

Strategy Beta: .328

Please note the Sharpe Ratio and Beta.

-TC

Nathan Swenson - Are you just generating the signal daily, around 11:30, on QC, in order to determine if you have a signal or not?

Thunder Chicken,

Yes, I have my live algo setup to paper trade.  I just check it daily for trades and trade my accounts manually.  Currently in equities. I still have flex v4 running.  I'm still not sure of the best version to run.  The new bear distill looks interesting.  

Peter Guenther,
Can you share a secret how you pick up symbols for signals in your In_out_flex_v5_disambiguate_v2?
The last one was "RINF".
Have you ever looked at it's intraday chart?

For the last week there were only 3 trades 500, 200, 200 shares.
Last year average daily volume of 100 shares per day.
Here is its monthly chart.

Do you think this is a reliable source for TQQQ trading?
 

Stream: Rich data modelists

Thanks for your question, Vovik!

I had hoped that it’s no “secret” why indicators are in the algo since these are individually introduced (use Ctrl + F “RINF”) plus there is an explanation in the code. What sometimes happens is that people cut away the in-code explanations to “save words” in the code; not a good idea, look at a code in a few months’ or years’ time from now and you know what I mean.
The RINF is there to gauge inflation expectations.

Now, how is the indicator specifically used in the code? Check out self.INFL in the code. We want to know, on a particular day, whether inflation expectations are above or below their median. Below is the ~three-month chart for RINF. Does this chart help me to know, on a particular day, whether expectations are above or below their median? I would say yes. I do not need a minute-sharp high-frequency trade price to make that call. Note that close to the median you will always have noise, frequently updated prices won’t make that go away and could even aggravate the issue. It’s also important to note: we do not trade RINF itself (in that case, we would be more highly concerned about liquidity/volume). What we are after via RINF is a coarse indication: above or below median inflation expectations.

However, I think a real downside of RINF is that it only started in Jan 2012 (see my original post). Therefore, the post notes that: “However, RINF could be manually constructed by comparing treasuries with inflation-protected treasuries. I have to explore that option a bit more and may give it a go later unless somebody else might be faster (?) ...”

Vovik, are you maybe that somebody who might be faster and thereby contribute to our Rich data modelists stream?

Peter Guenther,,

self.INFL is a member of self.SIGNALS

self.SIGNALS = [self.PRDC, self.METL, self.NRES, self.DEBT, self.USDX, self.INFL]

symbols = self.SIGNALS + [self.MRKT] + self.FORPAIRS

returns_sample = (self.history / self.history_shift - 1)

extreme_b = returns_sample.iloc[-1] < pctl_b

if (extreme_b[self.SIGNALS + self.pairlist]).any(): self.be_in = False

As I understand it, if any of [self.SIGNALS self.pairlist] falls below pctl_b, 
self.HLD_OUT will be generated.

Do you think "RINF" hitting lowest low after lowest low is a reliable source for TQQQ trading?

I've been readng through this entire thread and I find it very interesting! I'm a little confused on one part though.

mom = (self.history / self.history_shift_mean - 1)

What is this exactly caclulating? I don't understand the -1 in there. Is it some rule in matrices?

Vovik you have a very good eye there! In the Initialize part, self.INFL should be in the self.FORPAIRS (line 51) bracket instead of the self.SIGNALS bracket (line 52). The intention actually is not to use self.INFL as a signal by itself but to disambiguate the GOLD pair according to the table in an earlier post (Additional musing about signal ambiguity). Updated version attached.

Regarding TQQQ, I reckon trading 3 x leveraged ETFs is anyway a hell of a wild rodeo ride. I would take that risk only with a very measured investment (maybe <10-15% wealth or less). In my view, no matter which in & out strategy you use, or indicators you have included or have not included, drawdowns can be brutal in 3 x leverage ETFs and the in & out precision won't be sufficiently accurate to completely prevent these. One then hopes for a stellar increase in the TQQQ to more than make up for the drawdown.

Updated version:
Total returns: 2092% (worse)
Sharpe ratio: 1.87 (worse); Max drawdown: 13.6% (better); Compounding annual return: 26.8% (worse)

Frank DiGiacomo thanks for joining in and no worries. The code snippet calculates a return. self.history contains day-specific prices. self.history_shift_mean contains past averaged prices (i.e. for a particular stock and day, it would be the stock's average price in the range 55 days ago to 65 days ago). For an example stock, say the day-specific price is 110, the shifted_mean price is 100, then we calculate 110/100 - 1 = 10% return (the -1 is required to calculate the return).

Peter Guenther,

Do you think "RINF" hitting lowest low after lowest low is a reliable source for QQQ trading?

Vovik , thanks for your follow up. My personal view: Yes, I think RINF is reliable. It measures what I conceptually want it to measure, that is inflation expectations. It does so with sufficient accuracy, giving me a daily gauge on where we are at in terms of (changes in) the market’s inflation expectations. Would I use it in the In & Out to trade equity (QQQ, SPY etc.)? Yes, I reckon this is the whole point regarding further developing the In & Out, isn’t it?

Note that RINF does not continuously create new lows, at least not in terms of how the algo uses it. Below is a picture of RINF’s values in the return sample. Currently, it is actually relatively high which I reckon reflects the market’s expectation of an economic rebound.

A call for additional in & out-type algos

We have several in & out algos in this thread and beyond, in multiple variants. I am sure that there will be additional exciting stuff coming, including the use of futures and sensitivity testing via QC’s Cloud Parameter Optimization / Sensitivity Testing.

However, reading the rationales, discussions, etc. you might say “nah, I would have done this completely differently, like so ...” I recall people mentioning chart-analytical techniques like breakouts in certain markets and other interesting ideas. If you have something that you are happy to share with the community, probably the best way is via a new QC thread, but do leave a post in this thread as well, just so that everybody who is interested in these kinds of tactics (including myself) is aware. It would be really cool to see some (fundamentally) different approaches to tackle the in & out problem. Keep on coding!

New Chart: The Annual Saw Tooth Returns - In & Out versus Always In
This time, no innovation on the algo itself but instead a new chart that might be useful. We have noted here and elsewhere that the In & Out is not directly meant to replace a stock selection strategy but can be an optional complement. In other words, the choice would be to run your stock selection strategy with an In & Out logic instead of running your stock selection strategy without this logic (= being always in). You would hope that the In & Out component acts like an insurance package against steep drawdowns in your strategy. However, we were wondering what the implicit costs of this insurance package are.

The chart takes an annual view on this question by comparing the returns of the In & Out execution of the stock selection strategy (light blue line) versus the always-in execution (black line) of the stock selection strategy (= QQQ in the attached algo). The returns are reset at the beginning of each year, hence the saw tooth pattern. The chart may be useful to understand the implicit costs of the In & Out in terms of underperformance during certain times. For instance, the below chart shows that the relative performance in 2013 is not great and there is a stretched out soft patch from 2017 to 2019.

PS: If someone knows an easier implementation to create the chart, I would be very interested. However, note that I have implemented the calculations for the chart so that the stock selection dictionary could consist of more than one stock (e.g. holding QQQ and SPY together) and then the chart should still be created correctly.
The attached algo also creates a chart to compare the three-month (60 days) returns of the In & Out versus always in.

The code for the calculations and charts are in lines 220-237. I initialize a few variables for the charts in lines 72-78.

Hey Peter,

You can consolidate the huge chunks of code if you incorporated a few if and elif statements. For example, X-month and Annual can fall under the same plot, if you're comfortable with clicking to highlight and unhighlight the additional labels. Alot of the code looks like copy paste. Try looking on stackoverflow for some hints.

Thanks, Chak, for your insights/guidance, much appreciated!

Saw Tooth Chart comparison: The Distilled Bear (top) and The In & Out (bottom)
Now that we have the Saw Tooth Chart, what we can do is to compare different in & outs regarding the distribution of their returns. Below is a pasted together view of the returns based on The Distilled Bear (top) and The In & Out (bottom) algos. The comparatively strong sections of The Distilled Bear seem to be in 2008, 2013 (substantial), 2016, and 2017. The In & Out performs comparatively better in 2011, 2012, and 2020 (substantial).

Implications
- can one in & out ‘learn’ from the other, resulting in a ‘super in & out’?
- should one mix the logics and simultaneously bet on both horses, e.g. 50-50?
- ...

 Hey Peter Guenther how strong is the hypothesis with comparing industrials, gold, cash and their correlations? I'm sure a quick study can be done going back to almost 50-100 years on these broad markets? 

My concern is if these correlations hold up or if they possibly break meaning the algo's in / out goes totally out the windows. 

Also your thoughts on self stock selection vs ETF (stock selection chosen for you) 

Elsid Aliaj, great questions! Regarding your first point, Menno did a test a while back, looking at a momentum strategy switching between equity, gold and bonds. The test was from 1875 to 2020 and the strategy seemed to generate systematic returns over this substantial time period. The strategy builds on a certain correlation between the involved assets. So we could argue that if the (negative) correlation was there the past 150 years, it is likely to be there in the future, although there might be phases when it's more muted versus stronger. For these situations where the correlation is muted, I definitely see the benefit of your earlier point that sometimes we might want to go into all cash.

Regarding the selection of individual stocks versus ETFs, ETFs can be efficient in terms of saving on commissions. So that is generally attractive. However, I reckon in the backtests we tend to select ETFs that we know had a good run in the past, creating a lookback bias. When we have a logic that systematically selects individual stocks from the universe of all stocks, e.g. based on fundamentals, the backtest results could be more valid. What are your thoughts?

Stream: Rich data modelists

Bringing it all together
As you know (see my earlier post), I was musing for some time now about whether it could be beneficial to combine the two in & out algos to create a ‘super’ in & out. The attached algo is a first attempt into this direction. There might be different ways how this could be done, e.g. see Derek Melchin’s suggestion to use correlations in the Amazing returns = superior stock selection strategy + superior in & out strategy thread.

The general idea of the attached implementation is that we track both algos and see how they are going return-wise and then increasingly switch to the overperforming one. A key variable is self.weight_inout_vs_dbear which is a number between 1 (fully on In & Out) and 0 (fully on Distilled Bear). It is determined (lines 263-272) via comparing the in & outs’ returns over a tight timeframe (10 days; see self.io_mom_lookback in line 112) so that we quickly capture comparative overperformance and switch.
The algo creates a plot for the weight towards the In & Out vs Distilled Bear (line 342) in the “In Out” chart.

The performances / total returns:
In & Out alone: 2,090%
Distilled Bear alone: 1,890%
New combo: 2,800%

The algo has a few features which are included because of other issues I want to explore but which have no relevance in the current setting, so feel free to ignore. For example, adding equities and alternative assets via ticker string (lines 45-59), including OnSecutiriesChanged (lines 149-164), and determining equities/alternative assets with the best momentum (lines 286-300). You also have a chart that tracks the cash level (line 331) and the Annual Saw Tooth chart (lines 334-337).

I noticed that the recent algos now use "TIP" instead of "RINF" as a disambiguation signal.  Is that better?

Hi Peter

Good job!

I just run a back-test for the same period

of v1.0 "Intersection of ROC comparison using the OUT_DAY approach" by Vladimir

It only has two variables, and more economical code (three times fewer lines), and it seem to get better results than yours.

What do you think?

Thanks for your post and positive note, Liam, appreciated!
Also thanks for sharing these results, I think that is fantastic.

With my ‘rich data modelist’ hat on (see the different camps/streams mentioned earlier), I would tend to think, let’s combine the three in & outs into one. It’s a fundamentally different mindset to the features you highlighted in your post (few parameters / lines of code) and that is OK. We can think about it this way: there is a continuum and you can allocate the following people on it in the following way:
[parameter minimalists]     Menno  <----  Vladimir  ---->  Peter     [rich data modelists]

Now, just to clarify: the idea of combining the in & outs is not necessarily to boost performance, although this might happen in the process. It is rather about combining different logics, so that we do not have to pick one logic (‘a winner’) beforehand. Instead, we let the relative weights of the logics being determined empirically by the (future) data and how events unfold--this is the gist of what the post above tries to bring to the table. The weights are determined by the logics’ relative short-term performance. So, with this in mind, when we look at total return and number of parameters/lines of code of individual in & outs, I am not sure that we correctly capture what we might be able to develop here.

Liam, Peter

I also see that v2.5 “Dual Momentum with Out Days” by Vladimir looks much better than InOut_DBear_v1.

Jon Quant, sorry almost missed your question there. Well spotted! It could be a bit better but I am not even sure. The rationale for replacing RINF is just that the ETF is only available from Jan 2012 onwards and I wanted to have something that I could build on from Jan 2008. So I am using the bonds - tip difference.