XGBoost Regression Model Prediction of Future Returns based on Indicator and Market Percent Change Data

Here is the article about this algorithm that got published on the Medium platform. 

Hey Dan,

Thanks for sharing this and it looks like a great first attempt.

Couple of thoughts here:

• In the model training, it looks like you are using only the last 24 hourly indicator values as input features. (In other words your input features are shape (24, 3) with columns [ROCP, RSI, ATR]?) What about market data, and/or lagged features? This seems like a very small amount of data to be able to train the ML model effectively without overfitting.
• Shouldn't the target variable be future returns? It looks like the target here is `scaled_rocp` which means you are learning the mapping from indicators at time t to the return between [t-24, t].
• With 10 symbols in the universe (refreshed monthly), you are training 10 separate models for each symbol. This seems unnecessarily computationally expensive if the universe was larger, and you would also be discarding potentially very useful information in the cross-section.
• The v2 and v3 difference in results might be because the random operations weren't seeded, try adding `random_state` in the `RandomizedSearchCV`

Hello Adam,

First, thank you for the comment!

Okay so, the Market ROCP Data is one of the indicators? In the SymbolData class the market_rocp is created with only data on the SPY index. Also, the 24 shape is deceiving as that is the same size I made my rolling windows, so that is 24 windows of 24 hours. This means there is quite a bit of input data. You can see that in the SymbolData class as well.

indicator_df = pd.DataFrame(np.hstack((scaled_market_rocp, scaled_rsi, scaled_atr)))

This is a simple ML algorithm designed to help understand the fundamentals of implementing ML on the QC platform and is not intended to be a production-ready algorithm. Also, I am posting this as a community challenge to see the different and modified examples of this strategy that other quants choose to post. As for the low data, as I mentioned above I think 24 windows of 24 hours is sufficient for this simple strategy.  I was experimenting with using the target ROCP or a scaled ROCP and decided on the latter due to the performance increase. It has a quantile effect on the targets only trading the ones that are valued higher or lower after scaling. Then, I did some tuning and exploring which IMO is part of the experience and in this case, I preferred to use the scaled_rocp as the target. As for the seeding the random state options, I just tried this after reading your reply with no effect. Same results with the two environments behaving the same on the backtests. Thanks for the suggestion though! If you figure it out let me know. I appreciate the thoughtful feedback!  I would love to see a modified version from you posted with your other changes, as the point of this is to make this algorithm better with the help community as a shared learning exercise. So, posting shared results is always encouraged! Happy coding! 😊

Hey Dan, totally get that it's a simple algo for learning/getting familiar with QC and I'm always glad to see more people working with ML on here! 

So I think there might be a small misunderstanding here of how `RollingWindow` works in QC, or perhaps an unintended bug in the code? You can run for instance:

indicator_df = pd.DataFrame(np.hstack((scaled_market_rocp, scaled_rsi, scaled_atr)))
self.Log(f'Data Shape: {indicator_df.shape}')

x_train, x_valid, y_train, y_valid = train_test_split(indicator_df, scaled_rocp, test_size=0.35, random_state=42)
self.Log(f'X (Train): {x_train.shape}')
self.Log(f'X (val): {x_valid.shape}')

>>> Data Shape: (24, 3)
>>> X (Train): (15, 3)
>>> X (val): (9, 3)

So the RSI rolling window you defined would represent the previous 12 hr RSI at timepoints { t-24, t-23, …, t-1, t } where t is the current time that you are training the model.

Now for the target, you define it as (and possibly scaled) `rocp = algorithm.ROCP(symbol, 24)` which is the %-return over the period [t-24, t]. In other words, this is ( price_t - price_{t-24} ) / price_{t-24}. I am guessing that what you intended is future returns, i.e. the %-return over the period [t, t+24] or something like that, since in principle price_t is already known at time t.

Also one thing I missed initially - in the Predict function you generally wouldn't want to be scaling the data differently from the model training. If the RSI during training is [ 0.45, 0.5, 0.55 ], this gets scaled to [ -1, 0, 1 ]. If the new live RSI is [ 0.05, 0.1, 0.15 ], this also gets scaled to [ -1, 0, 1 ] if you re-fit the scaler, so the ML model will treat these two sets of features as identical when they are not. 

Hey Alex,

Thank you for the constructive criticism!

The splitting of data in training and not in the prediction function makes sense. Guess, that I assumed the eval_set would fix that, but that makes sense to me for the shape issues. That is a good example of a bug that needs to be squashed, and I love to see quants like yourself finding them! Good catch! As for the other recommendations and fixes. I am only using the last item from the scaled indicator_df when making a prediction, so just the prior day predicting the next day. Using the .pct_change() method could be a better implementation though. The scaling happens the same on the predicted indicators as the trained ones, and they are scaled using MinMaxScaler() to a range of -1 to 1 for each value in the numpy array. I understand that on a small length of input features that scaling issue could and would occur, but for features with a length of 24 wouldn't it be a very rare issue? Also, this seems like a drawback of scaling data in general. I do see your point though! This is exactly why you should post an example of a modified version. I would love to see the improvements to help my understanding of your feedback! It would also help illustrate your recommendations and fixes better for anyone reading. Thanks for the valuable input either way! 😊

Correction, sorry I meant Adam not Alex on that last message. 😊

This used to run and up till recently there must've been a change to Quantconnect as the algo seems to hang indefinitely after training the model during the backtest.  Is this occurring for anyone else?

Hi Axist,

The algorithm doesn't hang if we reduce the training parameters. For example:

parameters = {
    'n_estimators': [2],
    'learning_rate': [0.001],
    'max_depth': [2],
    'gamma': [0.001],
    'random_state': [42]
}

Best,
Derek Melchin