QuantConnect Community

How do I decipher Lean chart data?

Back

I want to grab chart data from a local Lean backtest and show in a Form using a charting library I have.

Researching chart data in BacktestResultPacket got me really confused. So I created a dump from chart data to make things a bit clearer. But unfortunately that didn't make things any clearer at all. So now I need info on how to interpret chart data so I can display my own pretty charts.

There are what to me looks like multiple anomalies in the chart data:

  1. Datapoints for saturdays are included in the data for all series, and the value for saturday is the same as for the next day with tradebars data.
  2. For the series with type Candle, there are 2 exactly identical data points for each date.
  3. For series with type Candle, there seems to be an extra data point at the end of the list, in addition to the  extra datapoint for each day mentioned above.
  4. The series 'Daily Performance', shows a performance percentage for saturdays, but 0 performance for mondays (or first day with tradebars after saturday).

Maybe the problem is in my dump code, which I have included after the dump itself below. Otherwise, you might want to investigate this further.

I have the following questions I need help with:

Q1: Can the ChartPoint.x timestamp be translated to a DateTime by using Time.UnixTimeStampToDateTime() as I have done it in TimestampAndDateTimeToString() and then directly use this DateTime as x-axis in my charts. Or does this have to be adjusted for timezone or something else?

Q2: How can I generate OHLC Candles using the data in a Candle series? (I realize two datapoints for each day can generate a kind of candle with Open and Close. But thats really a bar, not a candle. And the values should not both be the same, right?)

Q3: How should I handle data for saturdays in all the series?

Q4: How should I handle the extra day at the end of Candle series?

Q5: What to do about performance for saturdays but none for mondays?

 

Lean Chart data dump:

2016-06-17 14:59:21 - LeanChartsDumper.Dump() to file: LeanChartsDump.txt
==============================================================
Algorithm: BasicTemplateAlgorithm
Symbol: SPY
PeriodStart: 2015-01-02 00:00:00
PeriodFinish: 2015-12-31 23:59:59
---------------------------------------
Chart name: Benchmark
Series name:Benchmark Index:0 Type:Line Unit:$ Values.Count:308
[0] x:1420156800 = 2015-01-02 00:00:00 (Friday) y:0
[1] x:1420243200 = 2015-01-03 00:00:00 (Saturday) y:203,5083
[2] x:1420416000 = 2015-01-05 00:00:00 (Monday) y:203,5083
[3] x:1420502400 = 2015-01-06 00:00:00 (Tuesday) y:199,8332
[4] x:1420588800 = 2015-01-07 00:00:00 (Wednesday) y:197,9509
[5] x:1420675200 = 2015-01-08 00:00:00 (Thursday) y:200,4175
[6] x:1420761600 = 2015-01-09 00:00:00 (Friday) y:203,974
[7] x:1420848000 = 2015-01-10 00:00:00 (Saturday) y:202,3395
[8] x:1421020800 = 2015-01-12 00:00:00 (Monday) y:202,3395
[9] x:1421107200 = 2015-01-13 00:00:00 (Tuesday) y:200,7544
[10] x:1421193600 = 2015-01-14 00:00:00 (Wednesday) y:200,1898
[11] x:1421280000 = 2015-01-15 00:00:00 (Thursday) y:198,9812
[12] x:1421366400 = 2015-01-16 00:00:00 (Friday) y:197,1584
[13] x:1421452800 = 2015-01-17 00:00:00 (Saturday) y:199,744
[14] x:1421712000 = 2015-01-20 00:00:00 (Tuesday) y:199,744
[15] x:1421798400 = 2015-01-21 00:00:00 (Wednesday) y:200,1699
[16] x:1421884800 = 2015-01-22 00:00:00 (Thursday) y:201,1804
[17] x:1421971200 = 2015-01-23 00:00:00 (Friday) y:204,1722
[18] x:1422057600 = 2015-01-24 00:00:00 (Saturday) y:203,0528
[19] x:1422230400 = 2015-01-26 00:00:00 (Monday) y:203,0528
[20] x:1422316800 = 2015-01-27 00:00:00 (Tuesday) y:203,5282
[21] x:1422403200 = 2015-01-28 00:00:00 (Wednesday) y:200,8436
[22] x:1422489600 = 2015-01-29 00:00:00 (Thursday) y:198,2678
[23] x:1422576000 = 2015-01-30 00:00:00 (Friday) y:200,1006
[24] x:1422662400 = 2015-01-31 00:00:00 (Saturday) y:197,5843
[25] x:1422835200 = 2015-02-02 00:00:00 (Monday) y:197,5843
[26] x:1422921600 = 2015-02-03 00:00:00 (Tuesday) y:200,0312
[27] x:1423008000 = 2015-02-04 00:00:00 (Wednesday) y:202,9239
[28] x:1423094400 = 2015-02-05 00:00:00 (Thursday) y:202,1512
[29] x:1423180800 = 2015-02-06 00:00:00 (Friday) y:204,1919
[30] x:1423267200 = 2015-02-07 00:00:00 (Saturday) y:203,6273
[31] x:1423440000 = 2015-02-09 00:00:00 (Monday) y:203,6273
[32] x:1423526400 = 2015-02-10 00:00:00 (Tuesday) y:202,7159
[33] x:1423612800 = 2015-02-11 00:00:00 (Wednesday) y:204,8754
[34] x:1423699200 = 2015-02-12 00:00:00 (Thursday) y:204,9943
[35] x:1423785600 = 2015-02-13 00:00:00 (Friday) y:206,9657
[36] x:1423872000 = 2015-02-14 00:00:00 (Saturday) y:207,8177
[37] x:1424131200 = 2015-02-17 00:00:00 (Tuesday) y:207,8177
[38] x:1424217600 = 2015-02-18 00:00:00 (Wednesday) y:208,1447
[39] x:1424304000 = 2015-02-19 00:00:00 (Thursday) y:208,1645
[40] x:1424390400 = 2015-02-20 00:00:00 (Friday) y:208,0158
[41] x:1424476800 = 2015-02-21 00:00:00 (Saturday) y:209,2641
[42] x:1424649600 = 2015-02-23 00:00:00 (Monday) y:209,2641
[43] x:1424736000 = 2015-02-24 00:00:00 (Tuesday) y:209,2344
[44] x:1424822400 = 2015-02-25 00:00:00 (Wednesday) y:209,8287
[45] x:1424908800 = 2015-02-26 00:00:00 (Thursday) y:209,6505
[46] x:1424995200 = 2015-02-27 00:00:00 (Friday) y:209,4028
[47] x:1425081600 = 2015-02-28 00:00:00 (Saturday) y:208,6895
[48] x:1425254400 = 2015-03-02 00:00:00 (Monday) y:208,6895
[49] x:1425340800 = 2015-03-03 00:00:00 (Tuesday) y:210,0071
:
[258] x:1446595200 = 2015-11-04 00:00:00 (Wednesday) y:211
[259] x:1446681600 = 2015-11-05 00:00:00 (Thursday) y:210,36
[260] x:1446768000 = 2015-11-06 00:00:00 (Friday) y:210,1499
[261] x:1446854400 = 2015-11-07 00:00:00 (Saturday) y:210,0399
[262] x:1447027200 = 2015-11-09 00:00:00 (Monday) y:210,0399
[263] x:1447113600 = 2015-11-10 00:00:00 (Tuesday) y:208,08
[264] x:1447200000 = 2015-11-11 00:00:00 (Wednesday) y:208,5599
[265] x:1447286400 = 2015-11-12 00:00:00 (Thursday) y:207,74
[266] x:1447372800 = 2015-11-13 00:00:00 (Friday) y:204,8399
[267] x:1447459200 = 2015-11-14 00:00:00 (Saturday) y:202,5399
[268] x:1447632000 = 2015-11-16 00:00:00 (Monday) y:202,5399
[269] x:1447718400 = 2015-11-17 00:00:00 (Tuesday) y:205,6199
[270] x:1447804800 = 2015-11-18 00:00:00 (Wednesday) y:205,47
[271] x:1447891200 = 2015-11-19 00:00:00 (Thursday) y:208,7299
[272] x:1447977600 = 2015-11-20 00:00:00 (Friday) y:208,55
[273] x:1448064000 = 2015-11-21 00:00:00 (Saturday) y:209,3099
[274] x:1448236800 = 2015-11-23 00:00:00 (Monday) y:209,3099
[275] x:1448323200 = 2015-11-24 00:00:00 (Tuesday) y:209,07
[276] x:1448409600 = 2015-11-25 00:00:00 (Wednesday) y:209,35
[277] x:1448496000 = 2015-11-26 00:00:00 (Thursday) y:209,32
[278] x:1448582400 = 2015-11-27 00:00:00 (Friday) y:209,32
[279] x:1448668800 = 2015-11-28 00:00:00 (Saturday) y:209,5599
[280] x:1448841600 = 2015-11-30 00:00:00 (Monday) y:209,5599
[281] x:1448928000 = 2015-12-01 00:00:00 (Tuesday) y:208,69
[282] x:1449014400 = 2015-12-02 00:00:00 (Wednesday) y:210,6799
[283] x:1449100800 = 2015-12-03 00:00:00 (Thursday) y:208,5299
[284] x:1449187200 = 2015-12-04 00:00:00 (Friday) y:205,61
[285] x:1449273600 = 2015-12-05 00:00:00 (Saturday) y:209,6199
[286] x:1449446400 = 2015-12-07 00:00:00 (Monday) y:209,6199
[287] x:1449532800 = 2015-12-08 00:00:00 (Tuesday) y:208,35
[288] x:1449619200 = 2015-12-09 00:00:00 (Wednesday) y:206,9499
[289] x:1449705600 = 2015-12-10 00:00:00 (Thursday) y:205,3399
[290] x:1449792000 = 2015-12-11 00:00:00 (Friday) y:205,8699
[291] x:1449878400 = 2015-12-12 00:00:00 (Saturday) y:201,88
[292] x:1450051200 = 2015-12-14 00:00:00 (Monday) y:201,88
[293] x:1450137600 = 2015-12-15 00:00:00 (Tuesday) y:202,8999
[294] x:1450224000 = 2015-12-16 00:00:00 (Wednesday) y:205,0299
[295] x:1450310400 = 2015-12-17 00:00:00 (Thursday) y:208,0299
[296] x:1450396800 = 2015-12-18 00:00:00 (Friday) y:204,86
[297] x:1450483200 = 2015-12-19 00:00:00 (Saturday) y:200,02
[298] x:1450656000 = 2015-12-21 00:00:00 (Monday) y:200,02
[299] x:1450742400 = 2015-12-22 00:00:00 (Tuesday) y:201,6699
[300] x:1450828800 = 2015-12-23 00:00:00 (Wednesday) y:203,5
[301] x:1450915200 = 2015-12-24 00:00:00 (Thursday) y:206,02
[302] x:1451001600 = 2015-12-25 00:00:00 (Friday) y:205,6799
[303] x:1451260800 = 2015-12-28 00:00:00 (Monday) y:205,6799
[304] x:1451347200 = 2015-12-29 00:00:00 (Tuesday) y:205,21
[305] x:1451433600 = 2015-12-30 00:00:00 (Wednesday) y:207,3999
[306] x:1451520000 = 2015-12-31 00:00:00 (Thursday) y:205,9299
[307] x:1451624400 = 2016-01-01 05:00:00 (Friday) y:203,8699

---------------------------------------
Chart name: Strategy Equity
Series name:Equity Index:0 Type:Candle Unit:$ Values.Count:617
[0] x:1420174800 = 2015-01-02 05:00:00 (Friday) y:100000
[1] x:1420174800 = 2015-01-02 05:00:00 (Friday) y:100000
[2] x:1420261200 = 2015-01-03 05:00:00 (Saturday) y:100000
[3] x:1420261200 = 2015-01-03 05:00:00 (Saturday) y:100000
[4] x:1420434000 = 2015-01-05 05:00:00 (Monday) y:100000
[5] x:1420434000 = 2015-01-05 05:00:00 (Monday) y:100000
[6] x:1420520400 = 2015-01-06 05:00:00 (Tuesday) y:98805,8958
[7] x:1420520400 = 2015-01-06 05:00:00 (Tuesday) y:98805,8958
[8] x:1420606800 = 2015-01-07 05:00:00 (Wednesday) y:97881,722
[9] x:1420606800 = 2015-01-07 05:00:00 (Wednesday) y:97881,722
[10] x:1420693200 = 2015-01-08 05:00:00 (Thursday) y:99092,8275
[11] x:1420693200 = 2015-01-08 05:00:00 (Thursday) y:99092,8275
[12] x:1420779600 = 2015-01-09 05:00:00 (Friday) y:100839,0296
[13] x:1420779600 = 2015-01-09 05:00:00 (Friday) y:100839,0296
[14] x:1420866000 = 2015-01-10 05:00:00 (Saturday) y:100036,5062
[15] x:1420866000 = 2015-01-10 05:00:00 (Saturday) y:100036,5062
[16] x:1421038800 = 2015-01-12 05:00:00 (Monday) y:100036,5062
[17] x:1421038800 = 2015-01-12 05:00:00 (Monday) y:100036,5062
[18] x:1421125200 = 2015-01-13 05:00:00 (Tuesday) y:99258,2059
[19] x:1421125200 = 2015-01-13 05:00:00 (Tuesday) y:99258,2059
[20] x:1421211600 = 2015-01-14 05:00:00 (Wednesday) y:98981,0024
[21] x:1421211600 = 2015-01-14 05:00:00 (Wednesday) y:98981,0024
[22] x:1421298000 = 2015-01-15 05:00:00 (Thursday) y:98387,5855
[23] x:1421298000 = 2015-01-15 05:00:00 (Thursday) y:98387,5855
[24] x:1421384400 = 2015-01-16 05:00:00 (Friday) y:97492,5961
[25] x:1421384400 = 2015-01-16 05:00:00 (Friday) y:97492,5961
[26] x:1421470800 = 2015-01-17 05:00:00 (Saturday) y:98762,1191
[27] x:1421470800 = 2015-01-17 05:00:00 (Saturday) y:98762,1191
[28] x:1421730000 = 2015-01-20 05:00:00 (Tuesday) y:98762,1191
[29] x:1421730000 = 2015-01-20 05:00:00 (Tuesday) y:98762,1191
[30] x:1421816400 = 2015-01-21 05:00:00 (Wednesday) y:98971,2256
[31] x:1421816400 = 2015-01-21 05:00:00 (Wednesday) y:98971,2256
[32] x:1421902800 = 2015-01-22 05:00:00 (Thursday) y:99467,4097
[33] x:1421902800 = 2015-01-22 05:00:00 (Thursday) y:99467,4097
[34] x:1421989200 = 2015-01-23 05:00:00 (Friday) y:100936,3597
[35] x:1421989200 = 2015-01-23 05:00:00 (Friday) y:100936,3597
[36] x:1422075600 = 2015-01-24 05:00:00 (Saturday) y:100386,7195
[37] x:1422075600 = 2015-01-24 05:00:00 (Saturday) y:100386,7195
[38] x:1422248400 = 2015-01-26 05:00:00 (Monday) y:100386,7195
[39] x:1422248400 = 2015-01-26 05:00:00 (Monday) y:100386,7195
[40] x:1422334800 = 2015-01-27 05:00:00 (Tuesday) y:100620,1463
[41] x:1422334800 = 2015-01-27 05:00:00 (Tuesday) y:100620,1463
[42] x:1422421200 = 2015-01-28 05:00:00 (Wednesday) y:99302,0312
[43] x:1422421200 = 2015-01-28 05:00:00 (Wednesday) y:99302,0312
[44] x:1422507600 = 2015-01-29 05:00:00 (Thursday) y:98037,3236
[45] x:1422507600 = 2015-01-29 05:00:00 (Thursday) y:98037,3236
[46] x:1422594000 = 2015-01-30 05:00:00 (Friday) y:98937,2258
[47] x:1422594000 = 2015-01-30 05:00:00 (Friday) y:98937,2258
[48] x:1422680400 = 2015-01-31 05:00:00 (Saturday) y:97701,7026
[49] x:1422680400 = 2015-01-31 05:00:00 (Saturday) y:97701,7026
:
[567] x:1449118800 = 2015-12-03 05:00:00 (Thursday) y:103075,9993
[568] x:1449205200 = 2015-12-04 05:00:00 (Friday) y:101642,3284
[569] x:1449205200 = 2015-12-04 05:00:00 (Friday) y:101642,3284
[570] x:1449291600 = 2015-12-05 05:00:00 (Saturday) y:103611,1893
[571] x:1449291600 = 2015-12-05 05:00:00 (Saturday) y:103611,1893
[572] x:1449464400 = 2015-12-07 05:00:00 (Monday) y:103611,1893
[573] x:1449464400 = 2015-12-07 05:00:00 (Monday) y:103611,1893
[574] x:1449550800 = 2015-12-08 05:00:00 (Tuesday) y:102987,6684
[575] x:1449550800 = 2015-12-08 05:00:00 (Tuesday) y:102987,6684
[576] x:1449637200 = 2015-12-09 05:00:00 (Wednesday) y:102300,2193
[577] x:1449637200 = 2015-12-09 05:00:00 (Wednesday) y:102300,2193
[578] x:1449723600 = 2015-12-10 05:00:00 (Thursday) y:101509,7093
[579] x:1449723600 = 2015-12-10 05:00:00 (Thursday) y:101509,7093
[580] x:1449810000 = 2015-12-11 05:00:00 (Friday) y:101769,9393
[581] x:1449810000 = 2015-12-11 05:00:00 (Friday) y:101769,9393
[582] x:1449896400 = 2015-12-12 05:00:00 (Saturday) y:99810,8984
[583] x:1449896400 = 2015-12-12 05:00:00 (Saturday) y:99810,8984
[584] x:1450069200 = 2015-12-14 05:00:00 (Monday) y:99810,8984
[585] x:1450069200 = 2015-12-14 05:00:00 (Monday) y:99810,8984
[586] x:1450155600 = 2015-12-15 05:00:00 (Tuesday) y:100311,6693
[587] x:1450155600 = 2015-12-15 05:00:00 (Tuesday) y:100311,6693
[588] x:1450242000 = 2015-12-16 05:00:00 (Wednesday) y:101357,4993
[589] x:1450242000 = 2015-12-16 05:00:00 (Wednesday) y:101357,4993
[590] x:1450328400 = 2015-12-17 05:00:00 (Thursday) y:102830,4993
[591] x:1450328400 = 2015-12-17 05:00:00 (Thursday) y:102830,4993
[592] x:1450414800 = 2015-12-18 05:00:00 (Friday) y:101274,0784
[593] x:1450414800 = 2015-12-18 05:00:00 (Friday) y:101274,0784
[594] x:1450501200 = 2015-12-19 05:00:00 (Saturday) y:98897,6384
[595] x:1450501200 = 2015-12-19 05:00:00 (Saturday) y:98897,6384
[596] x:1450674000 = 2015-12-21 05:00:00 (Monday) y:98897,6384
[597] x:1450674000 = 2015-12-21 05:00:00 (Monday) y:98897,6384
[598] x:1450760400 = 2015-12-22 05:00:00 (Tuesday) y:99707,7393
[599] x:1450760400 = 2015-12-22 05:00:00 (Tuesday) y:99707,7393
[600] x:1450846800 = 2015-12-23 05:00:00 (Wednesday) y:100606,3184
[601] x:1450846800 = 2015-12-23 05:00:00 (Wednesday) y:100606,3184
[602] x:1450933200 = 2015-12-24 05:00:00 (Thursday) y:101843,6384
[603] x:1450933200 = 2015-12-24 05:00:00 (Thursday) y:101843,6384
[604] x:1451019600 = 2015-12-25 05:00:00 (Friday) y:101676,6493
[605] x:1451019600 = 2015-12-25 05:00:00 (Friday) y:101676,6493
[606] x:1451278800 = 2015-12-28 05:00:00 (Monday) y:101676,6493
[607] x:1451278800 = 2015-12-28 05:00:00 (Monday) y:101676,6493
[608] x:1451365200 = 2015-12-29 05:00:00 (Tuesday) y:101445,9284
[609] x:1451365200 = 2015-12-29 05:00:00 (Tuesday) y:101445,9284
[610] x:1451451600 = 2015-12-30 05:00:00 (Wednesday) y:102521,1693
[611] x:1451451600 = 2015-12-30 05:00:00 (Wednesday) y:102521,1693
[612] x:1451538000 = 2015-12-31 05:00:00 (Thursday) y:101799,3993
[613] x:1451538000 = 2015-12-31 05:00:00 (Thursday) y:101799,3993
[614] x:1451624400 = 2016-01-01 05:00:00 (Friday) y:100787,9393
[615] x:1451624400 = 2016-01-01 05:00:00 (Friday) y:100787,9393
[616] x:1451624400 = 2016-01-01 05:00:00 (Friday) y:100787,9393

Series name:Daily Performance Index:1 Type:Bar Unit:% Values.Count:308
[0] x:1420156800 = 2015-01-02 00:00:00 (Friday) y:0
[1] x:1420243200 = 2015-01-03 00:00:00 (Saturday) y:0
[2] x:1420416000 = 2015-01-05 00:00:00 (Monday) y:0
[3] x:1420502400 = 2015-01-06 00:00:00 (Tuesday) y:-1,194104
[4] x:1420588800 = 2015-01-07 00:00:00 (Wednesday) y:-0,9353428
[5] x:1420675200 = 2015-01-08 00:00:00 (Thursday) y:1,237315
[6] x:1420761600 = 2015-01-09 00:00:00 (Friday) y:1,762188
[7] x:1420848000 = 2015-01-10 00:00:00 (Saturday) y:-0,795846
[8] x:1421020800 = 2015-01-12 00:00:00 (Monday) y:0
[9] x:1421107200 = 2015-01-13 00:00:00 (Tuesday) y:-0,7780163
[10] x:1421193600 = 2015-01-14 00:00:00 (Wednesday) y:-0,2792752
[11] x:1421280000 = 2015-01-15 00:00:00 (Thursday) y:-0,599526
[12] x:1421366400 = 2015-01-16 00:00:00 (Friday) y:-0,9096568
[13] x:1421452800 = 2015-01-17 00:00:00 (Saturday) y:1,302174
[14] x:1421712000 = 2015-01-20 00:00:00 (Tuesday) y:0
[15] x:1421798400 = 2015-01-21 00:00:00 (Wednesday) y:0,2117274
[16] x:1421884800 = 2015-01-22 00:00:00 (Thursday) y:0,5013417
[17] x:1421971200 = 2015-01-23 00:00:00 (Friday) y:1,476815
[18] x:1422057600 = 2015-01-24 00:00:00 (Saturday) y:-0,5445414
[19] x:1422230400 = 2015-01-26 00:00:00 (Monday) y:0
[20] x:1422316800 = 2015-01-27 00:00:00 (Tuesday) y:0,2325276
[21] x:1422403200 = 2015-01-28 00:00:00 (Wednesday) y:-1,309991
[22] x:1422489600 = 2015-01-29 00:00:00 (Thursday) y:-1,273597
[23] x:1422576000 = 2015-01-30 00:00:00 (Friday) y:0,9179179
[24] x:1422662400 = 2015-01-31 00:00:00 (Saturday) y:-1,248795
[25] x:1422835200 = 2015-02-02 00:00:00 (Monday) y:0
[26] x:1422921600 = 2015-02-03 00:00:00 (Tuesday) y:1,229688
[27] x:1423008000 = 2015-02-04 00:00:00 (Wednesday) y:1,436061
[28] x:1423094400 = 2015-02-05 00:00:00 (Thursday) y:-0,3781745
[29] x:1423180800 = 2015-02-06 00:00:00 (Friday) y:1,00256
[30] x:1423267200 = 2015-02-07 00:00:00 (Saturday) y:-0,2746056
[31] x:1423440000 = 2015-02-09 00:00:00 (Monday) y:0
[32] x:1423526400 = 2015-02-10 00:00:00 (Tuesday) y:-0,4445216
[33] x:1423612800 = 2015-02-11 00:00:00 (Wednesday) y:1,057978
[34] x:1423699200 = 2015-02-12 00:00:00 (Thursday) y:0,05763025
[35] x:1423785600 = 2015-02-13 00:00:00 (Friday) y:0,9551512
[36] x:1423872000 = 2015-02-14 00:00:00 (Saturday) y:0,4088735
[37] x:1424131200 = 2015-02-17 00:00:00 (Tuesday) y:0
[38] x:1424217600 = 2015-02-18 00:00:00 (Wednesday) y:0,1563018
[39] x:1424304000 = 2015-02-19 00:00:00 (Thursday) y:0,009455188
[40] x:1424390400 = 2015-02-20 00:00:00 (Friday) y:-0,07095448
[41] x:1424476800 = 2015-02-21 00:00:00 (Saturday) y:0,5960908
[42] x:1424649600 = 2015-02-23 00:00:00 (Monday) y:0
[43] x:1424736000 = 2015-02-24 00:00:00 (Tuesday) y:-0,01410742
[44] x:1424822400 = 2015-02-25 00:00:00 (Wednesday) y:0,2821411
[45] x:1424908800 = 2015-02-26 00:00:00 (Thursday) y:-0,08437137
[46] x:1424995200 = 2015-02-27 00:00:00 (Friday) y:-0,1173466
[47] x:1425081600 = 2015-02-28 00:00:00 (Saturday) y:-0,3383553
[48] x:1425254400 = 2015-03-02 00:00:00 (Monday) y:0
[49] x:1425340800 = 2015-03-03 00:00:00 (Tuesday) y:0,6271393
:
[258] x:1446595200 = 2015-11-04 00:00:00 (Wednesday) y:0,2880673
[259] x:1446681600 = 2015-11-05 00:00:00 (Thursday) y:-0,3013171
[260] x:1446768000 = 2015-11-06 00:00:00 (Friday) y:-0,09921569
[261] x:1446854400 = 2015-11-07 00:00:00 (Saturday) y:-0,05199698
[262] x:1447027200 = 2015-11-09 00:00:00 (Monday) y:0
[263] x:1447113600 = 2015-11-10 00:00:00 (Tuesday) y:-0,9269263
[264] x:1447200000 = 2015-11-11 00:00:00 (Wednesday) y:0,2290902
[265] x:1447286400 = 2015-11-12 00:00:00 (Thursday) y:-0,3905016
[266] x:1447372800 = 2015-11-13 00:00:00 (Friday) y:-1,386673
[267] x:1447459200 = 2015-11-14 00:00:00 (Saturday) y:-1,115202
[268] x:1447632000 = 2015-11-16 00:00:00 (Monday) y:0
[269] x:1447718400 = 2015-11-17 00:00:00 (Tuesday) y:1,510243
[270] x:1447804800 = 2015-11-18 00:00:00 (Wednesday) y:-0,0724082
[271] x:1447891200 = 2015-11-19 00:00:00 (Thursday) y:1,575814
[272] x:1447977600 = 2015-11-20 00:00:00 (Friday) y:-0,08561336
[273] x:1448064000 = 2015-11-21 00:00:00 (Saturday) y:0,3619419
[274] x:1448236800 = 2015-11-23 00:00:00 (Monday) y:0
[275] x:1448323200 = 2015-11-24 00:00:00 (Tuesday) y:-0,1138528
[276] x:1448409600 = 2015-11-25 00:00:00 (Wednesday) y:0,133035
[277] x:1448496000 = 2015-11-26 00:00:00 (Thursday) y:-0,01423482
[278] x:1448582400 = 2015-11-27 00:00:00 (Friday) y:0
[279] x:1448668800 = 2015-11-28 00:00:00 (Saturday) y:0,1138473
[280] x:1448841600 = 2015-11-30 00:00:00 (Monday) y:0
[281] x:1448928000 = 2015-12-01 00:00:00 (Tuesday) y:-0,4123516
[282] x:1449014400 = 2015-12-02 00:00:00 (Wednesday) y:0,9471617
[283] x:1449100800 = 2015-12-03 00:00:00 (Thursday) y:-1,013765
[284] x:1449187200 = 2015-12-04 00:00:00 (Friday) y:-1,390887
[285] x:1449273600 = 2015-12-05 00:00:00 (Saturday) y:1,937048
[286] x:1449446400 = 2015-12-07 00:00:00 (Monday) y:0
[287] x:1449532800 = 2015-12-08 00:00:00 (Tuesday) y:-0,6017892
[288] x:1449619200 = 2015-12-09 00:00:00 (Wednesday) y:-0,6675062
[289] x:1449705600 = 2015-12-10 00:00:00 (Thursday) y:-0,7727354
[290] x:1449792000 = 2015-12-11 00:00:00 (Friday) y:0,2563597
[291] x:1449878400 = 2015-12-12 00:00:00 (Saturday) y:-1,92497
[292] x:1450051200 = 2015-12-14 00:00:00 (Monday) y:0
[293] x:1450137600 = 2015-12-15 00:00:00 (Tuesday) y:0,5017197
[294] x:1450224000 = 2015-12-16 00:00:00 (Wednesday) y:1,042581
[295] x:1450310400 = 2015-12-17 00:00:00 (Thursday) y:1,453272
[296] x:1450396800 = 2015-12-18 00:00:00 (Friday) y:-1,513579
[297] x:1450483200 = 2015-12-19 00:00:00 (Saturday) y:-2,346543
[298] x:1450656000 = 2015-12-21 00:00:00 (Monday) y:0
[299] x:1450742400 = 2015-12-22 00:00:00 (Tuesday) y:0,8191307
[300] x:1450828800 = 2015-12-23 00:00:00 (Wednesday) y:0,901213
[301] x:1450915200 = 2015-12-24 00:00:00 (Thursday) y:1,229863
[302] x:1451001600 = 2015-12-25 00:00:00 (Friday) y:-0,1639662
[303] x:1451260800 = 2015-12-28 00:00:00 (Monday) y:0
[304] x:1451347200 = 2015-12-29 00:00:00 (Tuesday) y:-0,2269163
[305] x:1451433600 = 2015-12-30 00:00:00 (Wednesday) y:1,059915
[306] x:1451520000 = 2015-12-31 00:00:00 (Thursday) y:-0,7040205
[307] x:1451624400 = 2016-01-01 05:00:00 (Friday) y:-0,9935815

LeanChartsDumper.cs

using System;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using QuantConnect.Packets;

namespace QuantConnect.Views
{
public class LeanChartsDumper {
private const string DTFORMAT = "yyyy-MM-dd HH:mm:ss";

public static void Dump(BacktestResultPacket packet, string filename, string symbol) {
var sb = new StringBuilder();
sb.AppendLine(string.Format("{0} - LeanChartsDumper.Dump() to file: {1}",
DateTime.Now.ToString(DTFORMAT), filename));
sb.AppendLine("==============================================================");
sb.AppendLine("Algorithm: " + packet.BacktestId);
sb.AppendLine("Symbol: " + symbol);
sb.AppendLine("PeriodStart: " + packet.PeriodStart);
sb.AppendLine("PeriodFinish: " + packet.PeriodFinish);

const string indent = " ";
const int n = 50;

// Dump all charts
foreach (var chartKvp in packet.Results.Charts) {
sb.AppendLine("---------------------------------------");
sb.AppendLine("Chart name: " + chartKvp.Value.Name);

// Dump first and last n data points in all series in this chart
foreach (var seriesKvp in chartKvp.Value.Series) {
var s = seriesKvp.Value;
var valuesCount = s.Values.Count;
sb.AppendLine(indent + string.Format("Series name:{0} Index:{1} Type:{2} Unit:{3} Values.Count:{4}",
new object [] { s.Name, s.Index, s.SeriesType, s.Unit, valuesCount }));

// First n points
for (var i = 0; i < n && i < valuesCount; i++) {
var istr = "[" + i + "]";
// Get timestamp and convert to datetime
var xstr = TimestampAndDateTimeToString(s.Values[i].x);
sb.AppendLine(indent + indent + string.Format("{0,-7}x:{1, -48} y:{2}",
istr, xstr, s.Values[i].y));
}

sb.AppendLine(indent + indent + ":");

// Last n points
for (var i = valuesCount - n; i < valuesCount; i++)
{
var istr = "[" + i + "]";
// Get timestamp and convert to datetime
var xstr = TimestampAndDateTimeToString(s.Values[i].x);
sb.AppendLine(indent + indent + string.Format("{0,-7}x:{1, -48} y:{2}",
istr, xstr, s.Values[i].y));
}

sb.AppendLine("");
}
}

// Save to file
File.WriteAllText(filename, sb.ToString());
}

private static string TimestampAndDateTimeToString(long timestamp) {
var dt = Time.UnixTimeStampToDateTime(timestamp);
return string.Format("{0} = {1} ({2})",
timestamp, dt.ToString(DTFORMAT), dt.ToString("dddd", CultureInfo.InvariantCulture));
}

}
}

Where I called the dumper in QuantConnect.Views.WinForms.LeanWinForm.cs

private void MessagingOnBacktestResultEvent(BacktestResultPacket packet)
{
if (packet.Progress != 1) return;

LeanChartsDumper.Dump(packet, "LeanChartsDump.txt", "SPY");

 

 

 

Update Backtest



4 Comments
 
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

The "X" is unix time, GMT -- which is 4-5 hours ahead which is why it might look like Saturday's for you. In reality we only sample during market hours (market open and close). We're looking at changing this soon to generate much more data per backtest but that will happen in the next 4-6 weeks.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Hi Jared,

Thanks for your answer, but I dont think your explanation computes.

Take this example from 'Equity' series:        [14] x:1420866000 = 2015-01-10 05:00:00 (Saturday)

GMT is 5h (or 4h when DST active) ahead of New York, which is local time for SPY.  (oh, I am already getting a headache)

So the timestamp in local NY time is 2015-01-10 00:00:00, which still was a saturday.

And this still doesn't explain why all chart types contain data for 6 days per week, when there are only a maximum of 5 traded days in a week.

It also doesn't explain why there are 2 identical datapoint for each day in Candle series.

Further, If the timestams are in GMT and should be adjusted -5h for NY time, that would mean series 'Benchmark' and 'Daily Performance' are timestamped at 7 or 8 pm in the evening and contain data for sundays. This doesn't sound logical to me.

I still think you need to take a really close look at this data and verify it is correct. This is important!

When you change and expand the data generated, could you please give us full candles data with OHLC?

And please, could you document how to interpret all the data? Because it is very difficult, time consuming and error prone to try to find this out from the code.

 

 

 

 

 

0
For your understanding the equity sampling is done in the algorithm manager

loop. If youre curious about it you can read that file closely and put some

breakpoints in LEAN (AlgorithmManager.cs).
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

I have had a look at AlgorithmManager and found what to me looks like a bug (but maybe it's a feature, what do I know?)

In order to understand AlgorithmManager.Run(), I simplified it and kept only the most important code related to timestamps and generating chart data:

Run() {
foreach (var timeSlice in Stream(job, algorithm, feed, results, token)) { // Line 198
var time = timeSlice.Time; // Line 238
if (backtestMode) {
if (_previousTime.Date != time.Date) {
SampleBenchmark(algorithm, results, _previousTime.Date); // Line 249
results.SampleEquity(_previousTime, Math.Round(algor... // Line 252
results.SamplePerformance(_previousTime.Date, Mat... // Line 261
}
}
_previousTime = time; // Line 582
}
results.SampleEquity(_previousTime, Math.Round(algorithm.Portf... // Line 637
SampleBenchmark(algorithm, results, _previousTime);
results.SamplePerformance(_previousTime, Math.Round((algorithm.Port...
}

It is obvious that if Stream returns a TimeSlice with a time that is not a traded day or a day for which there is no tradebar data in the disk file, there will be problems. And it does!

 

What happens is the following:

time is updated from timeSlice.Time (which is a saturday)  on Line 238

Samples are made for Benchmark, Equity and Performance on Lines 249, 252 and 261 using _previousTime (a friday, OK)

_previousTime is updated from time (so _previousTime is now a saturday)

Next loop, Samples are made for Benchmark, Equity and Performance using _previousTime (a saturday, >>> BUG <<<)

 

Drilling down into Stream, it seems the problem originates in SubscriptionSynchronizer.Sync(), or maybe in .MoveNext() or one of the other methods called by it.

I am not sure yet where the extra data sample each day for Equity data, or the extra sample at the end ([616] in the dump), comes from. But that is probably relatively easy to find once a problem with this code is recognized and dealt with.

Exactly hos to cure this I leave as an exercise for someone with a deep understanding of how this code is supposed to work. It's a bit complicated and I wouldn't dare touch this code which is at the heart of Lean. Good luck!

 

0
Update Backtest



0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Loading...

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!

Discussion Tags