Hi All,

I've been reviewing everything I can find regarding the AddData/PythonData class.  I have made a number of unsuccessful attempts to pull in some FRED data regarding daily federal funds rates.  My desire is for either or both SPY and DFF data to appear in the OnData method call.  I think that sometimes there might be SPY only or DFF only day but mostly they would appear on the same day.  

If so, I have a number of questions:

  • First of all, is that realistic?  Will QuantConnect synthesize one stream of data where there might be gaps?
  • I don't see the DFF data in the dozens and dozens of FRED symbols.  That's not a big problem as I am able to call their API directly. 
    • Perhaps I've overlooked the DFF data source?
  • In the attached backtest, I've defined a PythonData class and there are a number of issues with it.
    • GetSource doesn't seem to have a way to accept the API key and observation start date.
    • Where is the documentation for FileFormat.UnfoldingCollection?  I am guessing that it does what I want but perhaps that's one issue?
    • How can I pass in API key and start date?
    • The Reader method is never called.  Why?
    • Most of the AddData examples appear to expect to read a line of data, like a line from a CSV file.  The FRED DFF is a json file and looks like so:
{
   "count" : 1,
   "file_type" : "json",
   "limit" : 100000,
   "observation_end" : "9999-12-31",
   "observation_start" : "2022-09-01",
   "observations" : [
      {
         "date" : "2022-09-01",
         "realtime_end" : "2022-09-06",
         "realtime_start" : "2022-09-06",
         "value" : "2.33"
      }
   ],
   "offset" : 0,
   "order_by" : "observation_date",
   "output_type" : 1,
   "realtime_end" : "2022-09-06",
   "realtime_start" : "2022-09-06",
   "sort_order" : "asc",
   "units" : "lin"
}

Here's the beginning part of my PythonData class:

# Doesn't work.  
# - Never calls the Reader method, even when supplied with a valid key.  Why?
# - need to find a way to pass in the API key and not hard code it in the GetSource call.
# - need to find a way to pass in the start date and not hard code it in the GetSource call.
# 
class DailyFedFundRates2(PythonData):

    def GetSource(self, config, date, isLiveMode):        
        source = f"https://api.stlouisfed.org/fred/series/observations?series_id=DFF&api_key=REDACTED&file_type=json&observation_start=2022-01-01"
        return SubscriptionDataSource(source, SubscriptionTransportMedium.RemoteFile, FileFormat.UnfoldingCollection)

    def Reader(self, config, line, date, isLiveMode):
        if not (self.line.strip()):
            raise Exception(f'Cannot read FRED DFF data.  Aborting.' )           

I've attached the code.  I'm guessing there's some obvious thing I've missed?  Advice welcome.

Thank you.

Regards,

Rich