Need help with using zip file for custom data

Hey Kyrillos Boshra,

you will have to do the followings things to make it work:

1. extract file from zip
2. make sharable link from extracted file

Hope it helps ;)

Best,

Nico

I found in the documents that I can use zip file, I have no problem using a csv but the file size is too large and the backtesting takes so much time, can you elaborate more because I don't understand your proposed solution or you completely misunderstood what I'm asking for

Hey Kyrillos Boshra,

could you please provide the page were you read that? I think  that the only way to load custom data was trough a csv file.

Best,

Nico

https://www.quantconnect.com/docs/v2/writing-algorithms/importing-data/streaming-data/key-concepts#04-Set-Data-Sources

this part

Hey Kyrillos Boshra,

sorry for the long wait. I think you have to add the csv file ("SNAP.csv") to the file format:

def GetSource(
        self, config: SubscriptionDataConfig, date: datetime, isLive: bool
    ) -> SubscriptionDataSource:
        return SubscriptionDataSource(
            "https://www.dropbox.com/s/brwzhl2wgl13qx0/SNAP.zip?dl=1",
            SubscriptionTransportMedium.RemoteFile,
            FileFormat.ZipEntryName, "SNAP.csv"
        )

Best,

Nico

Ak Trading,

would you mind sending the code?

Hey Ak Trading and Kyrillos Boshra,

sorry I dont know how to solve this I tried multiple things but apparently these doesnt work. 

It does work though only using the csv file…

# region imports
from AlgorithmImports import *
# endregion

class CustomDataBitstampAlgorithm(QCAlgorithm):

    def Initialize(self):
        self.SetStartDate(2022, 11, 21)
        self.SetEndDate(datetime.now())
        self.SetCash(100000)

        # Define the symbol and "type" of our generic data:
        self.custom_data_symbol = self.AddData(MyCustomDataType, "SNAP").Symbol

        history = self.History(MyCustomDataType, self.custom_data_symbol, 200, Resolution.Minute)
        self.Debug(f"We got {len(history)} items from historical data request of {self.custom_data_symbol}.")


    def OnData(self, slice):
        history = self.History(MyCustomDataType, self.custom_data_symbol, 200, Resolution.Minute)
        self.Debug(f"We got {len(history)} items from historical data request of {self.custom_data_symbol}.")
        if self.custom_data_symbol not in slice:
            return

        data = slice[self.custom_data_symbol]
        self.Log(f'{data.EndTime}: Close: {data.Close}')
        self.Plot(self.custom_data_symbol, 'Price', data.Close)


class MyCustomDataType(PythonData):
    def GetSource(
        self, config: SubscriptionDataConfig, date: datetime, isLive: bool
    ) -> SubscriptionDataSource:
        return SubscriptionDataSource(
            "https://www.dropbox.com/s/v3bqwv8xd18dazn/SNAP.csv?dl=1",
            SubscriptionTransportMedium.RemoteFile
        )

    def Reader(
        self, config: SubscriptionDataConfig, line: str, date: datetime, isLive: bool
    ) -> BaseData:

        if not (line.strip() and line[0].isdigit()):
            return None

        index = MyCustomDataType()

        index.Symbol = config.Symbol

        # Example File Format:
        #                Date        Open       High        Low       Close     Volume      Turnover
        # 2023-01-17 11:30:00-05:00  7792.9    7799.9     7722.65    7748.7    116534670    6107.78

        data = line.split(",")
        index.Time = datetime.strptime(data[0][:-6], "%Y-%m-%d %H:%M:%S")
        index.EndTime = index.Time + timedelta(seconds=1)
        index.Value = float(data[4])
        index["Open"] = float(data[1])
        index["High"] = float(data[2])
        index["Low"] = float(data[3])
        index["Close"] = float(data[4])
        index["Volume"] = float(data[5]) * 100

        return index