How shall I deploy live trade on local LEAN by custom data?

Hi Frank,

I recommend you comma-separate your values (CSV = Comma-Separated Values).

For the Reader code, you do it like this:

    def Reader(self, config, line, date, isLive):
        data = line.split(",") # Hence, commas between values - will help a lot for parsing
        coin = CustomData()
        coin.Symbol = config.Symbol
        cur_time = datetime.strptime(data[0], "%Y%m%d %H:%M")
        coin.Time = cur_time - timedelta(hours=1)
        coin.EndTime = cur_time
        coin.Value = float(data[4])
        coin["Open"] = float(data[1])
        coin["High"] = float(data[2])
        coin["Low"] = float(data[3])
        coin["Close"] = float(data[4])
        coin["Volume"] = float(data[5])

        return coin

Fred

Hi Fred

Thanks for your reply.

In the fact, I have save data file as .csv file, so the values are comma-separate;

In addition, the values saved in .csv file are many lines and many columns; and in Reader, ‘line’ means the latest line when the mode is ‘isLive’, right?

Hi Frank,

Sorry but the data in your image, as shown in the image, is not formatted as comma-separated values. I am referring to this image:

Comma-separated values are separated by commas. You can see examples here:

For instance, this example:

Year,Make,Model,Description,Price
1997,Ford,E350,"ac, abs, moon",3000.00
1999,Chevy,"Venture ""Extended Edition""","",4900.00
1999,Chevy,"Venture ""Extended Edition, Very Large""",,5000.00
1996,Jeep,Grand Cherokee,"MUST SELL!
air, moon roof, loaded",4799.00

In Reader, the parameter ‘line’ is a string that has been read by LEAN after the call to GetSource from the returned source.

LEAN calls GetSource to know where to find the data it needs, for the date and time passed as ‘date’ to GetSource. When the ‘source’ returned in the SubscriptionDataSource object did not change from the previous call to GetSource, LEAN concludes that the data is in the next line. It simply reads the next line and pass it to Reader for parsing. If ‘source’ changes, then LEAN open that new source and reads the first line in that new source which it passes to Reader.

You are not using ‘date’ in GetSource, which can be ok but you need to understand the above paragraph - for the current implemented logic to work, your CSV file updates need to be in-sync with LEAN.

Trade data is usually divided in daily files in LEAN, so ‘date’ is used to identify the needed day and thus the needed file. So for each day, a new file is used, and each line correspond to one slice of the resolution for the data.

Finally, the parameter ‘isLive’ is independent. It tells you if the algo is live or not because in live mode, you could want to use another source for the data, such as a live data feed. And when ‘isLive’ is False, you would then use local files of historical data.

Fred

Hi Fred

I am so thankful for your detailed explaination. Above data ‘as shown in the image’ is loaded by pandas and if opened by txt editor, it is shown as below:
 

I have undertand most of the logic of Custom data because of your detailed explaination except below one:

When paper trading or live trading, for example Resolution.Hour, in which one new bar would be generated per hour.

All 1 hour new bars are generated by an independent script which would run at each ‘xx:00:00’ o'clock and the running time of this script is about 10 seconds. so quanct connect would not get the newest bars data at ‘xx:00:00’ but at ‘xx:00:10’, I want to know whether this would make some trouble?

when paper trading / live trade, what does it means as below picture:

Hi Frank,

😊

Very nice screenshot of a CSV file there! Hehehe. Thumbs up - we now know this is CSV. It looks like quote data, not trade, but let's pass.

So you are saying that, say, the 1-hour bar from 13:00 to 14:00 will be ready at 14:00:10, approx, for LEAN to use? Is that what you are saying?

When you log during live/paper trading, what do you see? Do you see that the bar, say, from 13:00 to 14:00, emitted at approx 14:00:10 by your script, is indeed consumed at the right time (14:00) by LEAN?

If you do, all is good.

If you don't, you probably don't see any data at all. Then, it means that your hour bars arrive too late for LEAN to considered them. At 14:00, LEAN is looking for hour data. For data tagged with an endtime of 14:00. If it is not there, it might just go “there's no data”, and move to the next hour, 1 hour later…

I am not sure about the interpretation to give to this:

“All 1 hour new bars are generated by an independent script which would run at each ‘xx:00:00’ o'clock and the running time of this script is about 10 seconds. so quanct connect would not get the newest bars data at ‘xx:00:00’ but at ‘xx:00:10’”

It sounds like your script creates hour bars at midnight every day. But then, it means it creates yesterday's bars, as it cannot predict the future. But you are talking about live/paper trading. So do you want to deploy live/paper trade not with real-time data but with the past day's data?

Fred

Hi Fred

Thank you very much for all your replies which make me undertand lean more and more clear.

Maybe my last expression confuses you a little, so now I clear it as below:

Let's say that I trade ‘BTCUSDT’ only(trade frequence is 1-hour, ‘Resolution.Hour’) and it is 2022-03-25 00:05:00 now; I have prepared 1-hour history bars for ‘BTCUSDT’ from 2022-03-24 00:00:00(total 24 bars) which is saved in ‘btcusdt.csv’ file. Then time comes to 2022-03-25 01:00:10, my script which is independent from LEAN would generates a new 1-hour bar and write into ‘btcusdt.csv’ file(now total 25 bars);

Yes just as you said, at  2022-03-25 01:00:00 LEAN can't get new data, so do nothing; and then about 20 minutes later, LEAN would look for new data again and this time it gets a new bar and threw it ‘OnData’. This is what I confirm when I test this process.

I guess if we want to make paper/live trading on local LEAN, we can only use ‘custom data’ which is loaded by function ‘AddData’ rather than ‘AddCrypto’. 

Hi Frank,

Re your last sentence, yes, custom data is only loaded with AddData. You can mix and match custom data and data loaded with AddCrypto (if the latter is available).

Your script seems just a bit too slow at producing the 1-hour bars. I don't know what is the exact tolerance for LEAN when it requests data, I mean, there is certainly some tolerance for intra-day data. 10 seconds to produce one 1-hour bar seems like an eternity to me (no offense/nothing personal). I would review my algorithm/technique there. Say, if you use tick data to produce your 1-hour bars, and you accumulate all ticks, and then, only after the hour, you go through all ticks during the hour to produce the bar, it is kinda normal that it takes some (too much) time. Rather, I would construct the 1-hour bar bit by bit, like not tick by tick, but like every minute or so, I would consolidate the last minute's ticks into the 1-hour bar I am currently building. And thus, at the hour, say 14:00, the only work left to have the 1-hour bar is aggregate the last 1-minute of ticks, which should only take a couple of milliseconds, if not a couple of microseconds.

Writing to the CSV file is also something to look at relatively closely to get the best performance possible.

So, I believe that you need to make the 1-hour bars available in the CSV file like only some milliseconds after the hour, not 10 seconds… Then, my guess is that LEAN will see them the first time it seeks for them…

I mean, when LEAN is using a live data feed to paper trade or live trade, 1-hour bars cannot be available before the hour is finished. So they can only be ready shortly after. How shortly it must be, I don't know (the tolerance above), but some milliseconds after the hour seems reasonable. It is certainly possible to provide 1-hour bars much much faster than that - 1-hour bars can be produced from all ticks within that hour in just a few SIMD CPU instructions if the tick data is laid out appropriately (according to those SIMD instructions specs), which in total would take just a few microseconds to execute. But I am guessing this speed is not necessary for LEAN.

Fred