Hi All,
I'm trying to understand the KPIs from the Key Performance table of an Alpha. I read the documentation but still not clear. In particular, I'd like to understand Mean Insight Value and Estimated Alpha Value. If I get a 0 Mean Insight Value, but a decently positive Estimated Alpha Value, how can I interpret that? It seems to depend on the timeframe chosen for the backtest. I guess what I'm trying to understand is how to get an estimated monthly value of my algorithms before submitting them.
Thank yiou very much for the amazing work.
Emilio
LukeI
From what I can gather from the documentation, the Estimated alpha value is based on how much volume was traded on that symbol. If a symbol has a lot of volume your insights are worth more because an institution could make larger allocations without affecting the market. I wouldn't use it as a true estimate of your alpha's value, only a theoretical maximum. The mean insight value is probably just the average, on a per-insight basis. I would imagine a coin-flip would have a mean insight value of 0 as it's 50/50 chance to be right it won't earn any money on average.
From my tests it seems that a (mean insight valuel)*(total number of insights) = Estimated Alpha value. In theory, if you put out 10 insights a month and the mean insight value is $100 then you could charge $1000 a month for that alpha. In practice it probably won't be so simple.
The KPI i'm trying to figure out is the magnitude and direction score. What drives those calculations? I can seemingly change the magnitude to an arbitrarily small value and it seems to give me a score of ~50. raising the magnitude almost always gives me a very low score. Direction is easier to figure out buy if my algorithm has a win rate of 60% does that mean my direction scores are 60% accurate and does that correspond to a score of 60?
Emilio Freire
Hi Lukel,
Thanks a lot for sharing your knowledge on this. I also thought the Estimated Value was basically the mean value times the total number of insights for the period selected, Then I guess the mean value is the one affected by volume and so on, since it's the KPI telling you on average how much money someone could potentially make from each insight.
Regarding the Direction Score, my understanding is it measures how many times you're right with your predictions. However, if you read the documentation it doesn't seem to be so simple as it depends on the "prediction time interval" that your insights object provides. For example, if you predict price will go up within the next 2 hours, if the price does go up immediately after you make the prediction then you get 100% score for that insight, but the price goes up later on within those 2 hours then you get a lower score (between 100% and 0%). And of course you get 0% if it doesn't go up within that time interval. Based on my results however, if I go to the "Inisghts" tab of the backtests I only see 0% or 100%!
This is of course just my understanding and probably it's not correct but hopefully someone from the team will help us figure this out!
One thing is clear, it's not all about the direction score as I can get over 60% accuracy and still my insights are not that valuable because I mostly predict small movements when the volume is also lower...
It would be amazing to have full documentation on Alpha Streams!
Emilio
Jared Broad
Hey! Glad you guys are excited about it -- we are as well! =) The framework scoring distills massively complex quant concepts down to a few scores. It is a challenging project. We had a previous direction score algorithm which was too complex so we recently simplified it - I updated the documentation to reflect it.
For your scores, we only review the moment of the insight completion giving you a binary score of 1 or 0 if you correctly predicted the direction. The sum of the insights for the backtest is averaged to give your overall direction score as a percentage.
i.e.
Direction:
You predicted up relative to starting price; if its > starting price you get 1. If price < start: 0. Sum(Backtest) => Average Score.
Magnitude:
You predicted up 10%; price moves up 5% at insight end time. You get magnitude score of 50%. Sum(Backtest) => Average Score.
Magnitude is a harder one to set; though arguably more valuable as most firms want an expected return curve. With a universe of magnitude scores, they can build this expected return.
The average insight value being $0 means winning insights earned the same as losing ones lost. This balance of signals nets $0. It is possible to have a near $0 algorithm and still have a positive return curve! We're going to keep improving the analysis but I think this setup helps you focus on optimizing each part of your algorithm separately. Once the Alpha is strong; you can work on portfolio construction to apply those signals.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Emilio Freire
Hi Jared,
Thank you very much for taking the time to explain. It's starting to become clearer.
Best,
Emilio
Emilio Freire
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!