Import data from Nasdaq data link/Qundl

Hey G Tam,

here's the backtest with the data you need 😊

Best, 
Nico

Nico Xenox Thank you.

ISM is monthly data so I guess imported data resolution is daily so it will be filled with pevious month value.
I want to get 3MA and 12MA, I used following code.

       self.SetStartDate(2008,3,28)
       self.SetEndDate(2023,1,1)
       self.ism = self.AddData(NasdaqCustomColumns, self.nasdaqCode, Resolution.Daily, TimeZones.NewYork).Symbol
       self.ism_3ma = self.SMA(self.nasdaqCode, 63, Resolution.Daily)
       self.ism_12ma = self.SMA(self.nasdaqCode,  252, Resolution.Daily)

It worked but not sure if this is the correct way to do it.

Also, data did not change after 2016-05-27, it stayed at 51.3, are there limits on the data that can be imported?
 

Hey G Tam,

yes, the code that you provided looks good. You asked for the dataset from nasdaq(quandl) which you can find here. The data goes from 1948 to 2016. 
You will have to work with this timespan if you want to request data from quandl. If you want to have data that is up to date and you have a source that you trust you can try to do some custom importing.

Data chart from 1948 to 2016:

Ah, I see I thought full data is available.

Thank you. I'll look to import data from yfinance.

Hi Nico Xenox-

I'm having trouble getting FRED/UNRATE data. I used G Tam's example but it's not working. Any way you could debug this code?

Also, in FRED/UNRATE, the data isn't refreshed on NASDAQ Datalink (only up to Jan. 2022) even though the ‘Validate’ link gives me a CSV that has current data all the way to 2023. Am I forced to use what's refreshed on Datalink or do I get access to the data in the ‘Validate’ source? https://data.nasdaq.com/data/FRED/UNRATE-civilian-unemployment-rate 

# region imports

from AlgorithmImports import *
from QuantConnect.DataSource import *
from statistics import stdev
import time
# endregion

class MeasuredTanTermite(QCAlgorithm):

    def Initialize(self):
        self.SetStartDate(2021, 6, 10)
        self.SetCash(100000)
        self.rebalanceTime = datetime.min
        self.AddUniverse(self.CoarseSelectionFilter, self.FineSelectionFilter)
        self.UniverseSettings.Resolution = Resolution.Daily #resolution of the securities in the universe, default is minute
        self.Data = {}
        self.ValueCheck= {}
        self.banned_tickers = {}

        self.spy = self.AddEquity("SPY", Resolution.Daily).Symbol

        #initializes bond yeild data from FRED
        self.fred_peak_to_trough = self.AddData(Fred, Fred.ICEBofAML.USHighYieldMasterIIOptionAdjustedSpread, Resolution.Daily).Symbol

        self.nasdaqCode = "FRED/UNRATE"
        self.unrate = self.AddData(NasdaqDataLink, self.nasdaqCode, Resolution.Daily).Symbol

    def OnData(self, data):

        for symbol in self.Data.keys():
            symbolData = self.Data[symbol]

            if not symbolData.IsReady: 
               continue

            
            if data.ContainsKey(self.nasdaqCode):
                self.Plot("Unemployment Rate", "Value", self.unrate.Value)

            #self.Log("Symbol " + str(symbol))

            #checks for availability of FRED data by using SPY trading days, plots spread
            if data.ContainsKey(self.fred_peak_to_trough) and data.ContainsKey(self.spy):
                peak_to_trough = data.Get(Fred, self.fred_peak_to_trough).Value
                self.Plot('FRED HY Spread', 'Bond Spread', peak_to_trough)

 # The CoarseFundamental class provides access to the following information that you can base your selection upon: self.Market, self.DollarVolume (for the day), self.Volume (for the day), self.HasFundamentalData, self.PriceFactor, self.SplitFactor, self.PriceScaleFactor, self.AdjustedPrice, self.EndTime.
 
    def CoarseSelectionFilter(self, coarse): #updates at midnight during backtesting, and around 6-7am in live trading, based on previous day data
        
        if self.Time <= self.rebalanceTime: #checks if it has been 1 day since rebalancing
            return self.Universe.Unchanged #if not do not change the universe
        self.rebalanceTime = self.Time + timedelta(35) #set rebalance time in days

        symbols_by_price = [c for c in coarse if c.HasFundamentalData and len(c.Symbol.Value) <= 4]
        self.filteredByPrice = symbols_by_price[:10]

        #for c in symbols_by_price[:30]: 
	    #    self.Log(",Symbol," + str(c.Symbol.Value) + ",DollarVolume," + str(c.DollarVolume) +  ",Price," + str(c.Price) + ",Volume," + str(c.Volume))
        
        return [c.Symbol for c in self.filteredByPrice]

    def FineSelectionFilter(self, fine):
        sortedByMarketCap = sorted(fine, key=lambda c: c.MarketCap)
        symbols_by_marketcap = [c for c in sortedByMarketCap if c.MarketCap > 0 and c.CompanyReference.IsREIT is False]
        self.filteredBymarketcap = symbols_by_marketcap

        #return self.filteredBymarketcap
        return [c.Symbol for c in self.filteredBymarketcap]

    def OnSecuritiesChanged(self, changes):
        for security in changes.AddedSecurities:
            symbol = security.Symbol
            if symbol not in self.Data:
                self.Data[symbol] = SymbolData(self, symbol)
        
        for security in changes.RemovedSecurities:
            symbol = security.Symbol
            if symbol in self.Data:
                symbolData = self.Data.pop(symbol, None)
                self.SubscriptionManager.RemoveConsolidator(symbol, symbolData.consolidator)

class SymbolData:
    def __init__(self, algorithm, symbol):
        self.algorithm = algorithm
        self.symbol = symbol
        self.Bars = RollingWindow[TradeBar](7)
        self.consolidator = TradeBarConsolidator(timedelta(days=1))
        self.consolidator.DataConsolidated += self.OnDataConsolidated
        algorithm.SubscriptionManager.AddConsolidator(symbol, self.consolidator)
    
    def OnDataConsolidated(self, sender, bar):
        self.Bars.Add(bar)
    
    @property
    def IsReady(self):
        return self.Bars.IsReady

Hey Justin E,

if I change the code we can see that it only has data till 2022-1-22. This means that you will probably have to import the CSV if you want to have it up to date.

Hope it helps ;)

I was able to import data from Googlespread sheet!!