QuantConnect Community

From what I can gather from the documentation, the Estimated alpha value is based on how much volume was traded on that symbol. If a symbol has a lot of volume your insights are worth more because an institution could make larger allocations without affecting the market. I wouldn't use it as a true estimate of your alpha's value, only a theoretical maximum. The mean insight value is probably just the average, on a per-insight basis. I would imagine a coin-flip would have a mean insight value of 0 as it's 50/50 chance to be right it won't earn any money on average.

From my tests it seems that a (mean insight valuel)*(total number of insights) = Estimated Alpha value. In theory, if you put out 10 insights a month and the mean insight value is $100 then you could charge $1000 a month for that alpha. In practice it probably won't be so simple.

The KPI i'm trying to figure out is the magnitude and direction score. What drives those calculations? I can seemingly change the magnitude to an arbitrarily small value and it seems to give me a score of ~50. raising the magnitude almost always gives me a very low score. Direction is easier to figure out buy if my algorithm has a win rate of 60% does that mean my direction scores are 60% accurate and does that correspond to a score of 60?

Hi Lukel,

Thanks a lot for sharing your knowledge on this. I also thought the Estimated Value was basically the mean value times the total number of insights for the period selected, Then I guess the mean value is the one affected by volume and so on, since it's the KPI telling you on average how much money someone could potentially make from each insight.

Regarding the Direction Score, my understanding is it measures how many times you're right with your predictions. However, if you read the documentation it doesn't seem to be so simple as it depends on the "prediction time interval" that your insights object provides. For example, if you predict price will go up within the next 2 hours, if the price does go up immediately after you make the prediction then you get 100% score for that insight, but the price goes up later on within those 2 hours then you get a lower score (between 100% and 0%). And of course you get 0% if it doesn't go up within that time interval. Based on my results however, if I go to the "Inisghts" tab of the backtests I only see 0% or 100%!

This is of course just my understanding and probably it's not correct but hopefully someone from the team will help us figure this out!

One thing is clear, it's not all about the direction score as I can get over 60% accuracy and still my insights are not that valuable because I mostly predict small movements when the volume is also lower...

It would be amazing to have full documentation on Alpha Streams!

Emilio

Hey! Glad you guys are excited about it -- we are as well! =) The framework scoring distills massively complex quant concepts down to a few scores. It is a challenging project. We had a previous direction score algorithm which was too complex so we recently simplified it - I updated the documentation to reflect it.

For your scores, we only review the moment of the insight completion giving you a binary score of 1 or 0 if you correctly predicted the direction. The sum of the insights for the backtest is averaged to give your overall direction score as a percentage.

i.e. 

Direction:
You predicted up relative to starting price; if its > starting price you get 1. If price < start: 0. Sum(Backtest) => Average Score.

Magnitude:
You predicted up 10%; price moves up 5% at insight end time. You get magnitude score of 50%. Sum(Backtest) => Average Score.

Magnitude is a harder one to set; though arguably more valuable as most firms want an expected return curve. With a universe of magnitude scores, they can build this expected return. 

The average insight value being $0 means winning insights earned the same as losing ones lost. This balance of signals nets $0. It is possible to have a near $0 algorithm and still have a positive return curve! We're going to keep improving the analysis but I think this setup helps you focus on optimizing each part of your algorithm separately. Once the Alpha is strong; you can work on portfolio construction to apply those signals. 

Hi Jared,

Thank you very much for taking the time to explain. It's starting to become clearer.

Best,

Emilio