I've created a custom data class for Python for Minute Resolution data, which takes in local files saved in the QuantConnect format of YYYYMMDD_tickType.zip/YYYYMMDD_symbol_resolution_tickType.csv.


My problem is that I am missing data for UTC 00:00 of every day. Not adding a timedelta(minutes=1) to the EndTime of the CustomData Class shifts the data by one minute with nothing missing, but when I do add the timedelta, the data is properly aligned except that it is missing the data for 00:00.


Am I missing something in the Reader method? It seems that setting the Time and EndTime to the start time and end time of the bar is not working, so perhaps it is because I can't set the Period in Python. Another possibility is that the data is created for the DATE given to the Reader method, and when the date changes it resets the data for 00:00. However, the data for 00:00 comes from the previous date's 23:59.

Attached is a backtest that replicates my custom data class and uses "local" BTCUSDT data from the cloud. The DataFrame in the log shows the missing data for the custom data class vs. the standard AddCrypto method. Logs in the Reader method for the data class show that the data file is being read:

 

2022-08-01 23:21:00 :	Time of data: 23:59:00
2022-08-01 23:26:00 :	Data.Time, Data.EndTime, Close, OHLV: (datetime.datetime(2022, 8, 2, 0, 0), datetime.datetime(2022, 8, 2, 0, 0), 23268.01, 23287.83, 23289.68, 23260.0, 54.61836)
2022-08-02 22:44:00 :	Time of data: 23:59:00
2022-08-02 22:44:00 :	Data.Time, Data.EndTime, Close, OHLV: (datetime.datetime(2022, 8, 3, 0, 0), datetime.datetime(2022, 8, 3, 0, 0), 22987.79, 22986.03, 22993.03, 22980.0, 70.51278)
2022-08-03 23:11:00 :	Time of data: 23:59:00
2022-08-03 23:11:00 :	Data.Time, Data.EndTime, Close, OHLV: (datetime.datetime(2022, 8, 4, 0, 0), datetime.datetime(2022, 8, 4, 0, 0), 22818.37, 22846.28, 22848.8, 22816.04, 62.90323)
2022-08-04 23:10:00 :	Time of data: 23:59:00
2022-08-04 23:10:00 :	Data.Time, Data.EndTime, Close, OHLV: (datetime.datetime(2022, 8, 5, 0, 0), datetime.datetime(2022, 8, 5, 0, 0), 22622.98, 22603.02, 22625.53, 22600.28, 78.06183)
2022-08-06 00:00:00 :	Time: Custom Data Close, QC AddCrypto Close
2022-08-06 00:00:00 :	2022-08-01 23:59:00: 23290.56, 23290.56
2022-08-06 00:00:00 :	2022-08-02 00:00:00: nan, 23268.01
2022-08-06 00:00:00 :	2022-08-02 23:59:00: 22987.0, 22987.0
2022-08-06 00:00:00 :	2022-08-03 00:00:00: nan, 22987.79
2022-08-06 00:00:00 :	2022-08-03 23:59:00: 22846.28, 22846.28
2022-08-06 00:00:00 :	2022-08-04 00:00:00: nan, 22818.37
2022-08-06 00:00:00 :	2022-08-04 23:59:00: 22604.02, 22604.02
2022-08-06 00:00:00 :	2022-08-05 00:00:00: nan, 22622.98
2022-08-06 00:00:00 :	2022-08-05 23:59:00: nan, 22622.98
2022-08-06 00:00:00 :	2022-08-06 00:00:00: nan, 22622.98
2022-08-06 00:00:00 :	Algorithm Id:(1418982f48e7b82c3b4113db8df1fbc8) completed in 1.20 seconds at 23k data points per second. Processing total of 27,520 data points.