Need help with importing custom data

Hello Ziyad,

Firstly, don't worry about sounding like a "noob". Everyone starts somewhere, and it's great that you're taking the initiative to learn and apply these concepts.

From the code you've provided, it seems like you're on the right track. You've correctly defined a custom data class (MASIIndexData) and are attempting to load data from a local file. However, there are a few things that might be causing issues:

1. Data File Path: In your GetSource method, you've provided a path to a directory, not a specific file. You need to provide the full path to the actual CSV file you're trying to load. For example, if your file is named MASI_data.csv, your path should look something like r'C:\users\hp\OneDrive\Documents\Moroccan All Shares Historical Data\MASI_data.csv'.

2. Data File Format: Your Reader method assumes a specific format for your data file. Make sure your CSV file has the same format. The expected format based on your code is: Date, Price, Open, High, Low, Volume, Change.

3. Error Handling: In your Reader method, you're using a try/except block but not doing anything in the except clause. This means if there's an error in reading a line, your program will silently ignore it and move on. This could be why you're not seeing any results. Consider logging the error or the problematic line to help debug issues.

4. Data Availability: You're using a RollingWindow of 50 periods and also setting a warm-up period of 50 days. Make sure your data file has at least this many entries, otherwise, your algorithm might not start trading.

5. Model Training: You're checking if self.model is None and then initializing it. However, it's not clear where and how this model is trained. Make sure your model is properly trained before you try to get allocations from it.

6. Data Access: In your OnData method, you're adding the entire slice object to your RollingWindow. If MASI is the only data you're interested in, consider adding just that data to the window: self.data.Add(slice["MASI"]).

Remember, you can always use the self.Debug or self.Log methods to print out information and help debug your algorithm.

Lastly, it's important to note that QuantConnect does not currently support direct data import from investing.com. You'll need to manually download your data, save it in the correct format, and then provide the path to this file in your algorithm.

I hope this helps! Let me know if you have any other questions.

Hi,

Thank you for your detailled comment.

Here is the updated code that unfortunately doesn't work either, i have fixed some of the issues you mentionned previously. Please let me know what i should do next.

from AlgorithmImports import *
import pandas as pd

class MASIIndexData(PythonData):
   '''MASI Custom Data Class'''

   def GetSource(self, config: SubscriptionDataConfig, date: datetime, isLiveMode: bool) -> SubscriptionDataSource:
       # Provide the full path to your CSV file
       file_path = r'C:\users\hp\OneDrive\Documents\Moroccan All Shares Historical Data.csv'
       return SubscriptionDataSource(file_path, SubscriptionTransportMedium.LocalFile)

   def Reader(self, config: SubscriptionDataConfig, line: str, date: datetime, isLiveMode: bool) -> BaseData:
       try:
           data = line.split(',')
           masi_data = MASIIndexData()
           masi_data.Symbol = config.Symbol
           masi_data.Time = datetime.strptime(data[0], "%Y-%m-%d")
           masi_data.EndTime = masi_data.Time + timedelta(days=1)
           masi_data.Value = float(data[1])
           masi_data["Open"] = float(data[2])
           masi_data["High"] = float(data[3])
           masi_data["Low"] = float(data[4])
           masi_data["Volume"] = float(data[5])
           masi_data["Change"] = float(data[6])
           return masi_data
       except:
           # Log and handle any errors
           self.Debug(f'Error reading line: {line}')
           return None

class MyAlgorithm(QCAlgorithm):
   def Initialize(self):
       self.SetStartDate(2012, 1, 1)  # Set Start Date
       self.SetEndDate(2023, 9, 7)  # Set End Date
       self.SetCash(100000)  # Set Strategy Cash

       # Add the custom data source MASIIndexData
       self.AddData(MASIIndexData, "MASI")

       n_periods = 50

       self.data = RollingWindow[Slice](n_periods)

       self.Train(self.DateRules.MonthStart("MASI"), self.TimeRules.Midnight, self.Rebalance)

       self.model = None

       self.SetWarmup(n_periods)

       self.prev_day = -1

   def OnData(self, slice):
       # Add just the MASI data to the rolling window
       if "MASI" in slice.Keys:
           self.data.Add(slice["MASI"])

   def Rebalance(self):
       if not self.data.IsReady:
           return

       try:
           # Reverse RollingWindow to have older data at the top
           data = self.PandasConverter.GetDataFrame(self.data).iloc[::-1]
       except:
           return

       if self.model is None:
           self.model = Model()

       allocations = self.model.get_allocations(data)
       self.Log(f'Portfolio Allocations: {allocations}')

       # Set holdings based on the custom data
       self.SetHoldings("MASI", allocations[0])  # Assuming MASI is the first and only custom data in the list
 

Hello Ziyad,

Your updated code looks much better and you've addressed many of the issues I previously mentioned. However, there are still a few things that might be causing problems:

1. Indentation: Python is sensitive to indentation, and it seems like there might be some issues with your code's indentation. For example, the methods GetSource and Reader should be indented to be part of the MASIIndexData class. Similarly, the lines of code after self.AddData(MASIIndexData, "MASI") should be indented to be part of the Initialize method.

2. Model Training: You're still initializing your model in the Rebalance method, but it's not clear where and how this model is trained. The Model class and its get_allocations method are not defined in the provided code. Make sure your model is properly trained before you try to get allocations from it.

3. Data Access: In your OnData method, you're checking if "MASI" is in slice.Keys before adding it to your RollingWindow. This is a good practice, but remember that slice.Keys are symbols, not strings. You should compare it to the symbol of your custom data, not the string "MASI". You can save the symbol when you add the data: self.masi = self.AddData(MASIIndexData, "MASI").Symbol and then check if self.masi in slice.Keys.

4. Setting Holdings: In your Rebalance method, you're trying to set holdings for "MASI". However, SetHoldings expects a symbol, not a string. You should use the symbol you saved when you added the data: self.SetHoldings(self.masi, allocations[0]).

Remember, you can always use the self.Debug or self.Log methods to print out information and help debug your algorithm.

I hope this helps! Let me know if you have any other questions.