Discussion Forum - QuantConnect.com

Ab DeWeese INVESTOR

| Reward Discussion

I'm trying to calculate the Mean and StDev of the TrueRange. I was using something like:

            TrueRange trueRange = TR(symbol, Resolution.Daily);

            SimpleMovingAverage trueRangeMean = SMA(symbol, VOLATILITY_SAMPLE_SIZE, Resolution.Daily).Of(trueRange);

            StandardDeviation trueRangeStDev = STD(symbol, VOLATILITY_SAMPLE_SIZE, Resolution.Daily).Of(trueRange);

Then moved to something like:

            AverageTrueRange atr = ATR(symbol, VOLATILITY_SAMPLE_SIZE, MovingAverageType.Simple, Resolution.Daily);

To grab the mean.

I found that the trueRangeMean and trueRangeStDev calculations above where wildly off. Conversely atr returns something that makes sense for the mean when you look at the data. But I do not trust the StDev calculation. I could be doing it wrong but... I read the API source code and the documentation AND I tried doing a reverse incation like this:

            StandardDeviation veDtSegnaReurt = trueRange.Of(STD(symbol, VOLATILITY_SAMPLE_SIZE, Resolution.Daily));

But it does not compile (while jives with the documentation and my understanding of the API source code).

So... what am I missing? I would like to create an indicator that is dynamically updated which computes the StDev of the TrueRange. Any suggestions?

Update Backtest

Ab DeWeese

Accepted Answer

Update Backtest

Notebook

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Ab DeWeese

151 ,

Anyone have insights here's? I get reasonable values for StDev when I calculate it myself. But when I do:

var indicator = STD().Of(trueRange);

I get bogus numbers for the stdev that don't line up with the underlying trueRange time series...

Shile Wen

63.5k ,

Hi Ab,

Please attach a backtest that demonstrates this issue.

Best,
Shile Wen

Ab DeWeese

151 ,

I stripped down the code and reproduced the issue. Pasted below is the code and the logs that demonstrate the issue.

** Note how the mean emitted by the composite indicator does not agree with its counterpart emitted by the ATR indicator

** Note how the StDev emitted by the composit indicator does not agree with the StDev calculated by hand

** Bonus: The reverse incantation: //veDtSegnaReurt = TrueRange.Of(STD(MySymbol, NUMBER_OF_SAMPLES, Resolution.Daily)); does not compile so I'm pretty sure I'm not getting the A.Of(B) out of order in my STD().Of(TR) call.

Can you please reproduce the issue on your side? Or look at my code and tell me what I am doing wrong...?

using QuantConnect.Data;

using QuantConnect.Data.Market;

using QuantConnect.Indicators;

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Threading.Tasks;



namespace QuantConnect.Algorithm.CSharp

{

    public class _StdMathErrorAlgorithm : QCAlgorithm

    {

        private DateTime Start = new DateTime(2019, 1, 1);

        private DateTime End = new DateTime(2019, 12, 31);



        private Symbol MySymbol { get; set; }

        private const int NUMBER_OF_SAMPLES = 20;



        private TrueRange TrueRangeIndicator { get; set; }

        private SimpleMovingAverage TrueRangeMeanFromComposite { get; set; }

        private StandardDeviation TrueRangeStDevFromComposite { get; set; }

        private AverageTrueRange MeanFromAtrIndicator { get; set; }

        private RollingWindow<decimal> TrueRangeWaveform = new RollingWindow<decimal>(NUMBER_OF_SAMPLES);

        private decimal TrueRangeStDevByHand => TrueRangeWaveform.StandardDeviationOfTheSample(SomeMath.HowToFail.FailWide);



        public override void Initialize()

        {

            Debug($"StDev test {DateTime.UtcNow}");

            Debug(Header);



            SetStartDate(Start);

            SetEndDate(End);

            SetCash(100000);

            MySymbol = AddEquity("SPY", Resolution.Daily).Symbol;



            TrueRangeIndicator = TR(MySymbol, Resolution.Daily);

            TrueRangeMeanFromComposite = SMA(MySymbol, NUMBER_OF_SAMPLES, Resolution.Daily).Of(TrueRangeIndicator);

            TrueRangeStDevFromComposite = STD(MySymbol, NUMBER_OF_SAMPLES, Resolution.Daily).Of(TrueRangeIndicator);

            //veDtSegnaReurt = TrueRange.Of(STD(MySymbol, NUMBER_OF_SAMPLES, Resolution.Daily));

            MeanFromAtrIndicator = ATR(MySymbol, NUMBER_OF_SAMPLES, MovingAverageType.Simple, Resolution.Daily);

        }



        public override void OnData(Slice slice)

        {

            TrueRangeWaveform.Add(TrueRangeIndicator.Current.Value);

            if (TrueRangeWaveform.IsReady)

                PrintDebugString(slice[MySymbol]);

        }



        internal static string Header = "{Spy}, {Date}, {TrueRange}, {StDevFromComposite}, {StDevByHand}, {MeanFromComposite}, {MeanFromAtr}";

        private void PrintDebugString(TradeBar tradebar) => Debug($"{MySymbol}, {tradebar.EndTime.Date:yy/MM/dd}, {TrueRangeIndicator.Current.Value:N2}, {TrueRangeStDevFromComposite.Current.Value:N2}, {TrueRangeStDevByHand:N2}, {TrueRangeMeanFromComposite.Current.Value:N2}, {MeanFromAtrIndicator.Current.Value:N2}");

    }



    internal static class SomeMath

    {

        internal enum HowToFail { FailWide, FailNarrow }

        internal static decimal VarianceOfTheSample(this IEnumerable<decimal> samples, HowToFail howToFail)

        {

            decimal result;

            decimal onFailure;

            switch (howToFail)

            {

                case HowToFail.FailNarrow:

                    onFailure = 0;

                    break;

                case HowToFail.FailWide:

                default:

                    onFailure = samples.Max() - samples.Min();

                    break;

            }

            if (samples.NMinusOne() > 0)

                result = samples.SumOfQuadranceFromMean() / samples.NMinusOne();

            else

                result = onFailure;

            return result;

        }



        internal static decimal Mean(this IEnumerable<decimal> samples)

        {

            int count = samples.Count();

            if (0 == count)

                throw new InvalidOperationException("Mean not defined for empty sample.");

            else

                return samples.Sum() / count;

        }



        internal static decimal StandardDeviationOfTheSample(this IEnumerable<decimal> samples, HowToFail howToFail) => (decimal)Math.Sqrt((double)samples.VarianceOfTheSample(howToFail)); // rounding error is in the noise

        private static decimal NMinusOne(this IEnumerable<decimal> samples) => samples.Count() - 1;

        internal static decimal SumOfQuadranceFromMean(this IEnumerable<decimal> samples) => samples.Sum(observation => Quadrance(observation, samples.Mean()));

        internal static decimal Quadrance(decimal x1, decimal x2) => (x1 - x2).Squared();

        internal static decimal Squared(this decimal x) => x * x;

    }

}

*** LOGS *** here are the 1st 50 logs:

2019-01-01 00:00:00 :   Launching analysis for 614b417c6fcaa390a8e781af8d4f6716 with LEAN Engine v2.4.0.0.9950
2019-01-01 00:00:00 :   StDev test 12/9/2020 9:56:34 PM
2019-01-01 00:00:00 :   {Spy}, {Date}, {TrueRangeIndicator}, {StDevFromComposite}, {StDevByHand}, {MeanFromComposite}, {MeanFromAtrIndicator}
2019-01-30 00:00:00 :   SPY, 19/01/30, 2.00, 125.78, 1.86, 129.03, 3.65
2019-01-31 00:00:00 :   SPY, 19/01/31, 4.94, 126.04, 1.69, 129.47, 3.77
2019-02-01 00:00:00 :   SPY, 19/02/01, 3.02, 126.41, 1.66, 129.97, 3.66
2019-02-02 00:00:00 :   SPY, 19/02/02, 1.94, 126.85, 1.58, 130.22, 3.44
2019-02-05 00:00:00 :   SPY, 19/02/05, 2.57, 127.18, 1.02, 130.42, 3.14
2019-02-06 00:00:00 :   SPY, 19/02/06, 1.50, 127.87, 1.06, 130.73, 3.01
2019-02-07 00:00:00 :   SPY, 19/02/07, 1.37, 128.45, 1.12, 131.05, 2.92
2019-02-08 00:00:00 :   SPY, 19/02/08, 4.30, 128.65, 1.16, 131.49, 3.01
2019-02-09 00:00:00 :   SPY, 19/02/09, 2.59, 128.90, 1.16, 131.69, 2.96
2019-02-12 00:00:00 :   SPY, 19/02/12, 1.40, 129.35, 1.18, 131.90, 2.94
2019-02-13 00:00:00 :   SPY, 19/02/13, 3.77, 129.77, 1.19, 132.51, 3.00
2019-02-14 00:00:00 :   SPY, 19/02/14, 1.77, 130.29, 1.22, 132.71, 2.93
2019-02-15 00:00:00 :   SPY, 19/02/15, 2.68, 130.52, 1.18, 132.90, 2.98
2019-02-16 00:00:00 :   SPY, 19/02/16, 2.92, 130.83, 1.17, 133.31, 2.94
2019-02-20 00:00:00 :   SPY, 19/02/20, 2.04, 131.14, 1.16, 133.56, 2.85
2019-02-21 00:00:00 :   SPY, 19/02/21, 1.62, 131.40, 1.05, 133.83, 2.67
2019-02-22 00:00:00 :   SPY, 19/02/22, 1.99, 131.59, 1.01, 134.08, 2.57
2019-02-23 00:00:00 :   SPY, 19/02/23, 1.87, 132.15, 1.01, 134.40, 2.56
2019-02-26 00:00:00 :   SPY, 19/02/26, 2.10, 132.61, 1.01, 134.81, 2.51
2019-02-27 00:00:00 :   SPY, 19/02/27, 1.34, 133.03, 0.99, 135.23, 2.39
2019-02-28 00:00:00 :   SPY, 19/02/28, 2.04, 133.36, 0.99, 135.39, 2.39
2019-03-01 00:00:00 :   SPY, 19/03/01, 1.09, 133.57, 0.83, 135.53, 2.20
2019-03-02 00:00:00 :   SPY, 19/03/02, 2.12, 133.89, 0.81, 135.80, 2.15
2019-03-05 00:00:00 :   SPY, 19/03/05, 4.87, 133.89, 1.01, 135.99, 2.30
2019-03-06 00:00:00 :   SPY, 19/03/06, 1.31, 133.98, 1.03, 136.01, 2.24
2019-03-07 00:00:00 :   SPY, 19/03/07, 2.12, 133.91, 1.02, 135.99, 2.27
2019-03-08 00:00:00 :   SPY, 19/03/08, 3.15, 133.73, 1.01, 135.93, 2.35
2019-03-09 00:00:00 :   SPY, 19/03/09, 2.50, 133.48, 0.90, 135.73, 2.26
2019-03-12 00:00:00 :   SPY, 19/03/12, 5.99, 133.23, 1.23, 135.88, 2.43
2019-03-13 00:00:00 :   SPY, 19/03/13, 1.58, 133.23, 1.22, 135.90, 2.44
2019-03-14 00:00:00 :   SPY, 19/03/14, 2.78, 133.30, 1.19, 136.04, 2.39
2019-03-15 00:00:00 :   SPY, 19/03/15, 1.13, 133.42, 1.21, 136.16, 2.36
2019-03-16 00:00:00 :   SPY, 19/03/16, 2.21, 133.52, 1.21, 136.27, 2.34
2019-03-19 00:00:00 :   SPY, 19/03/19, 1.31, 133.90, 1.22, 136.29, 2.26
2019-03-20 00:00:00 :   SPY, 19/03/20, 2.86, 134.05, 1.23, 136.59, 2.30
2019-03-21 00:00:00 :   SPY, 19/03/21, 3.09, 134.26, 1.23, 136.90, 2.37
2019-03-22 00:00:00 :   SPY, 19/03/22, 4.45, 134.73, 1.31, 137.50, 2.50
2019-03-23 00:00:00 :   SPY, 19/03/23, 5.39, 134.87, 1.45, 137.94, 2.67
2019-03-26 00:00:00 :   SPY, 19/03/26, 2.48, 135.13, 1.44, 137.85, 2.69
2019-03-27 00:00:00 :   SPY, 19/03/27, 3.05, 135.19, 1.41, 138.06, 2.78
2019-03-28 00:00:00 :   SPY, 19/03/28, 3.71, 135.12, 1.41, 138.09, 2.86
2019-03-29 00:00:00 :   SPY, 19/03/29, 2.08, 135.11, 1.37, 138.17, 2.91
2019-03-30 00:00:00 :   SPY, 19/03/30, 2.07, 135.18, 1.37, 138.22, 2.91
2019-04-02 00:00:00 :   SPY, 19/04/02, 3.58, 135.24, 1.30, 138.50, 2.84
2019-04-03 00:00:00 :   SPY, 19/04/03, 1.10, 135.50, 1.31, 138.59, 2.83
2019-04-04 00:00:00 :   SPY, 19/04/04, 1.94, 135.80, 1.32, 138.77, 2.82
2019-04-05 00:00:00 :   SPY, 19/04/05, 1.41, 136.07, 1.35, 138.74, 2.74

Ab DeWeese

151 ,

I figured it out.

            //TrueRangeMeanFromComposite = SMA(MySymbol, NUMBER_OF_SAMPLES, Resolution.Daily).Of(TrueRangeIndicator);

            TrueRangeMeanFromComposite = new SimpleMovingAverage(NUMBER_OF_SAMPLES).Of(TrueRangeIndicator);

            //TrueRangeStDevFromComposite = STD(MySymbol, NUMBER_OF_SAMPLES, Resolution.Daily).Of(TrueRangeIndicator);

            TrueRangeStDevFromComposite = new StandardDeviation(NUMBER_OF_SAMPLES).Of(TrueRangeIndicator);

Subtle but interesting. Looks like the data wants to be passed by value instead of passing the indicator by reference. Also further testing showed that the StDev calculation is actually the StDev of the POPULATION not the StDev of a SAMPLE. The StDev indicator class documentation suggests it the StDev of a sample while the underlying variance object it leverages has documentation that suggests its the variance of a population. I ran some tests and verified its using the population version of the math vs the sample version.

FYI to future googlers.

Ab DeWeese INVESTOR

Update Backtest

Notebook

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!

Platform

QuantConnect Community

Alpha League Competition: $1,000 Weekly Prize Pool

Filter Discussion by Tags

240000 Quants.

Online Now

Stay Connected

STD().Of(TR())

Started By:

Join QuantConnect for Free Sign Up

Platform

SIGN IN

QuantConnect Community

Alpha League Competition: $1,000 Weekly Prize Pool

Filter Discussion by Tags

240000 Quants.

Online Now

Stay Connected

STD().Of(TR())

Started By:

Join QuantConnect for Free Sign Up