Hi everyone, 

I've been a Quantopian user for the past years. Now I'm currently testing QuantConnect. 

The main features that I liked there and I think would be a big plus for QuantConnect: 

  • The ability to select easily through all the universe (similar pipelines)
  • The ability to debug backtesting and run commands at the breakpoints
  • The ability to debug locally a backtesting (with zipline, I was able to debug Factor Calculations or other locally)
For the first issue with the universe selection, let me share with you what I'm trying to accomplish:
  • At EOD, I want to go through all the universe and calculate the average daily volume for the past 52 w, the high of the past 52w and the resistance on the last 3 years => Select only a few securities that meet specific conditions with these stocks 
  • At every minute, get the price, the total daily volume then compare and select securities that verify specific conditions with this information (not in the below code)
I've followed the guide for migrating the code, but the code below fails at the simple step of loading the full daily history for all the securities over the last three years as it takes more than 10 minutes.   Do you have any idea how I can make it work?  ThanksAyoubfrom clr import AddReference AddReference("System.Core") AddReference("QuantConnect.Common") AddReference("QuantConnect.Algorithm") from System import * from QuantConnect import * from QuantConnect.Algorithm import QCAlgorithm from QuantConnect.Data.UniverseSelection import * from QuantConnect.Orders import OrderStatus from QuantConnect.Orders.Fees import InteractiveBrokersFeeModel LAST_TRADED_PRICE_MIN = 0.85 LAST_TRADED_PRICE_MAX = 10 WEEKS = 52 DAILY_VOLUME_MIN = 3000000 DAILY_VOLUME_MAX = 1000000000 MIN_DAILY_RETURNS = 8 MAX_DAILY_RETURNS = 100 WEEK_HIGH_MIN_FACTOR = 0.01 WEEK_HIGH_MAX_FACTOR = 0.07 WEEKS_PREVIOUS_RESISTANCE = 156 WEEKS_PREVIOUS_RESISTANCE_FACTOR = 0.23 TARGET_PERCENTAGE = 0.5 LOSS_THRESHOLD = .07 PROFIT_THRESHOLD = -.03 class Factors: def __init__(self, weeks_high=52, weeks_resistance=156): self.days_high = (weeks_high + 1) * 5 self.days_resistance = (weeks_resistance + 1) * 5 def process(self, element): """ :param element: dict(str, pd.DataFrame) :type element: :return: :rtype: """ element['week_high'] = element['high'].rolling( window=self.days_high).max() element['dollar_volume'] = element['volume'] * element['close'] element['average_daily_volume'] = element['dollar_volume'].rolling( window=self.days_high).mean() element['previous_resistance'] = element['high'].rolling( window=self.days_resistance).max() element['is_tradable'] = element['previous_resistance'] >= ( 1 + WEEKS_PREVIOUS_RESISTANCE_FACTOR) * element['week_high'] element.dropna(inplace=True) return element[element['is_tradable']] class RawPricesCoarseUniverseAlgorithm(QCAlgorithm): def Initialize(self): '''Initialise the data and resolution required, as well as the cash and start-end dates for your algorithm. All algorithms must initialized.''' # what resolution should the data *added* to the universe be? self.UniverseSettings.Resolution = Resolution.Daily self.SetStartDate(2020,11,1) #Set Start Date self.SetEndDate(2020,12,1) #Set End Date self.SetCash(10000) #Set Strategy Cash # Set the security initializer with the characteristics defined in CustomSecurityInitializer self.SetSecurityInitializer(self.CustomSecurityInitializer) # this add universe method accepts a single para&é@a meter that is a function that # accepts an IEnumerable<CoarseFundamental> and returns IEnumerable<Symbol> self.AddUniverse(self.CoarseSelectionFunction) # Taking profit self.take_profit_active = True self.profit_threshold = .07 self.profit_logging = True # Stop Loss self.stoploss_active = True self.loss_threshold = -.03 self.stoploss_logging = True def CustomSecurityInitializer(self, security): '''Initialize the security with raw prices and zero fees Args: security: Security which characteristics we want to change''' security.SetFeeModel(InteractiveBrokersFeeModel()) # sort the data by daily dollar volume and take the top 'NumberOfSymbols' def CoarseSelectionFunction(self, coarse): df = self.History([s.Symbol for s in coarse], (WEEKS_PREVIOUS_RESISTANCE + 1) * 5, Resolution.Daily) self.Log(f"Processing Data") self.data = Factors(weeks_high=WEEKS, weeks_resistance=WEEKS_PREVIOUS_RESISTANCE).process(df) self.Log(f"Total symbols found {len(self.data)}") x = len(self.data) return [s for s in coarse if str(s) in self.data]