I recently read through Rob Carver's Leveraged Trading book and am implementing his Simple System.  It's pretty basic and the alpha's are simple (MA crossover, breakout, carry), but I did like the idea of using weighted alphas together rather than a single alpha.

This idea married with something else I've been thinking a lot about.  My alphas need to be adaptive, i.e. they need to change as the market changes. 

Here's an example.  In a recent project I trained  n number of Extra Trees Classifiers with 3 years of data and about 50 complex features. I then held out the most recent 60 days of data for testing.  I kept all the classifiers rather than selecting the best and then weighted their predictions based on their performance on the test set. The results were clearly better than my prior results.  This works because the different classifiers randomly select their features and some work better in certain conditions that others don't.  So in this case, the alpha became adaptive because we weighted the classifiers on their recent performance.

So now I want to extend this idea to multiple AlphaModels in QC.  Let's say I take my MA Crossover model and instantiate n versions of the model with different parameters for MA Type and slow/fast periods.  Then we build a function into our alpha model to create the historical daily returns for that model and all of our selected securities over some time period.   What this would do is effectively find which parameters had performed better most recently for each given security. We would then use that recent performance score to weight the alpha's insights.

My thoughts and questions on this:
1. Is it overfitting?  Yes on each individual alpha, but the hope is the weighting balances some of the overfitting out.
2. This seems very akin to Portfolio Construction by looking at returns by security and using that to weight the securities in the portfolio.

I assume something like this has been done, for example using Factors probably?  Would love any insight the more experience folks might have to share on this topic.