I have created custom data sets for Lean based on the same forex data format, but using data from an external source. Data is 1min and each file has been broken into 1 day from 00:00 through 23:59, and zipped. File formatting and naming is the same (other than the inclusion of volume in my data).

Sample data files are available here.

I have found that UTC+2 works best for this data (FXCM sourced via IQFeed) as I don't get any Sat/Sun data files.

There are sometimes data files that don't a complete 24 hour period, and certain days that don't exist (ie. Sat/Sun). There will be other times when the first trading day of the week doesn't kick off right at 00:00. My assumption is that using Fill Forward sorts out any errors relating to these issues, and this seems to be correct based on the limited understanding of QC.

My issue is when I run a backtest against my data, there are times when the algorithm never finishes, eg: running the MovingAverageCross algorithm with default settings other than:
private const string Symbol = "AUDUSD";
AddSecurity(SecurityType.Forex, Symbol, Resolution.Minute, true);

In this example, the End Date is 1 Jan 2015. When running this against my data (which has not file for this date), the following output is received:


What looks to be occuring is that the algorithm is waiting for data from the FileSystemDataFeed that never arrives (and never will since it doesn't exist). The Isolator.ExecuteWithTimeLimit() message occurs regularly (every 60 seconds).

If I change the End Data to a day when there is a data file available, the algorithm seems to execute (and end) correctly.

Am I missing something here? Is there a way to Fill Forward across days to resolve this issue?

I don't really want to add a check for file existence into the algo as this means nothing other than in backtesting.

Regards

Adam