Reading data from multiple local files

Hi Notta,

Yep, it's possible. Here is the rationale:

1- The param “date” is the datetime that LEAN is looking data for (depending on resolution - it can be just a date (Daily) or a date and time (Intraday).

2- When the source inside the SubscriptiongDataSource object you return changes, LEAN gets that it needs to load that new source. Then it reads the first line and send to Reader.

3- If the source is the same, it reads the next line and sends to Reader.

So, in a nutshell, return a new source when the month in date changes. Naming your files with years and months would do it and using the year and month in date to coerce the right filename…

The lines in your monthly files need to be in-sync with the resolution you use (seconds, minutes, hours, whatever).

Be careful around Time and EndTime/Period in Reader / wrt what you have in your file or you won't receive the right data at the right time and most people don't even notice since the algo runs…

Fred

Fred Painchaud thank you for the prompt answer. Ok so using strftime(FILE_NAMING_SCHEME) I can call the correct file to be read. As a concrete example: With data of resolution of 1 minute, I can put a month's worth of data into a single file with the filename XXX-[MONTH].csv and then just do 

f"XXX-{date.strftime('%m')}.csv"

to read it within GetSource?

Could you also elaborate on “Be careful around Time and EndTime/Period in Reader / wrt what you have in your file or you won't receive the right data at the right time and most people don't even notice since the algo runs…” ? 

I have not been using EndTime for some of the data sources I had since sometimes the data is irregular. Do I have to specify EndTime for all classes inheriting from TradeBar class? 

Also for those datasets that I can specify an EndTime, should I be wary of the timezone of the data? Is that what you are referring to in your comment about endTime/Period? 

Hi Notta,

Here is an example.

class LocalSPYMinute(PythonData):
    def GetSource(self, config: SubscriptionDataConfig, date: datetime, isLive: bool) -> SubscriptionDataSource:
        source = f"C:\\Users\\Fred\\Desktop\\tmp\\{date.date().strftime('%Y%m%d')}_spy_minute_trade.csv"
        return SubscriptionDataSource(source, SubscriptionTransportMedium.LocalFile)

    def Reader(self, config: SubscriptionDataConfig, line: str, date: datetime, isLive: bool) -> "LocalSPYMinute":
        data = line.split(",")
        local_spy = LocalSPYMinute()
        local_spy.Symbol = config.Symbol
        cur_time = datetime.datetime.strptime(data[0], "%H%M%S")
        cur_delta = datetime.timedelta(hours=cur_time.hour, minutes=cur_time.minute)
        local_spy.Time = date + cur_delta - datetime.timedelta(minutes=1)
        local_spy.EndTime = date + cur_delta
        local_spy.Value = float(data[4])
        local_spy["Open"] = float(data[1])
        local_spy["High"] = float(data[2])
        local_spy["Low"] = float(data[3])
        local_spy["Close"] = float(data[4])
        local_spy["Volume"] = float(data[5])

        return local_spy

Those files are divided into days but you get the idea. Your line of code is correct.

Note that my files contain end time at minute resolution but .Time is start time. For equity, the time must be in the time zone of the market. If you set .Time and .EndTime, you do not need to set .Period. Or you can set .Time and .Period.

Don't forget to set .Value if you want to trade that custom data.

Fred

Fred Painchaud I see, thank you for the detailed answer!