I notice that sometimes my backtest runs very quickly and the server statistics show >100% CPU usage, but other times it runs extremely slowly with <10% CPU usage. This is for the exact same algorithm. Is it because I am using a Community B-MICRO instance? Or is it being bottlenecked when trying to load data? Will subscribing to QC and getting a beefier backtesting node help with this?
Thanks,
Karthik
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Hi Karthik. Correct the free tier is a shared-CPU, the speed achieved will depend entirely on other free user loads. If you subscribe to a paid node it's a dedicated CPU.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Perfect, that is exactly the information I was looking for. Thank you!
I am re-running my backtest on a dedicated node but it is still running very slowly with low CPU utilization. Is there something else that could be the bottleneck? Is the first run generally slower while the data populates? My backtest is running on SPY with minute resolution, and after 15 minutes it has only covered 4 months of simulation time.
Also, are free tier research nodes also shared CPU?
Yes, free tier research nodes are also shared CPU's. Please post some code to review; it might be a bug in the algorithm causing it to slow down but it's hard to suggest solutions without knowing what you're trying to do.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Jared Broad thanks for the clarification. I've pasted my code below. Now that I am on a dedicated node, it occasionally runs slow but now if I kill the backtest and start a new one the new one will usually runs at the expected fast level. Whereas on the free node the new test would usually be just as slow.
Start New Discussion 
Back
Reward Discussion 
7
from Execution.ImmediateExecutionModel import ImmediateExecutionModel
import pandas as pd
class MyOptionMarginModel:
option = None
originalMarginModel = None
def __init__(self, option, originalMarginModel):
self.option = option
self.originalMarginModel = originalMarginModel
def GetLeverage(self, security):
return self.originalMarginModel.GetLeverage(security)
def SetLeverage(self, security, leverage):
self.originalMarginModel.SetLeverage(security, leverage)
def HasSufficientBuyingPowerForOrder(self, parameters):
return self.originalMarginModel.HasSufficientBuyingPowerForOrder(parameters)
def GetMaximumOrderQuantityForTargetBuyingPower(self, parameters):
return self.originalMarginModel.GetMaximumOrderQuantityForTargetBuyingPower(parameters)
def GetMaximumOrderQuantityForDeltaBuyingPower(self, parameters):
return self.originalMarginModel.GetMaximumOrderQuantityForDeltaBuyingPower(parameters)
def GetReservedBuyingPowerForPosition(self, params):
if self.option.IsDelisted:
return ReservedBuyingPowerForPosition(0)
else:
return self.originalMarginModel.GetReservedBuyingPowerForPosition(params)
def GetBuyingPower(self, parameters):
return self.originalMarginModel.GetBuyingPower(parameters)
class CalibratedUncoupledRadiator(QCAlgorithm):
'''
If SPY is within a certain % of its 52W high, write a call option that is a certain $ above the opening price.
If the option gets assigned, cover ASAP
'''
# Paramaters
callOtmAmount = 6
withinYearlyHighPct = 10
spy = None
spyOption = None
spyCoverOrderTicket = None
yearlyHigh = None
days = 0
def Initialize(self):
self.SetStartDate(2007, 1, 1)
self.SetEndDate(2020, 11, 19)
self.SetCash(100000)
#self.callOtmAmount = int(self.GetParameter("callOtmAmount")) # TODO deal with missing parameter
self.spy = self.AddEquity("SPY", Resolution.Minute)
self.spy.SetDataNormalizationMode(DataNormalizationMode.Raw) # Required for working with options
self.spy.SetLeverage(4)
self.UniverseSettings.MinimumTimeInUniverse = 10
self.yearlyHigh = self.MAX(self.spy.Symbol, 252, Resolution.Daily)
history = self.History(self.spy.Symbol, 252, Resolution.Daily)
for bar in history.itertuples():
self.yearlyHigh.Update(bar.Index[1], bar.high)
self.Debug("SPY yearly high warmed up, value = " + str(self.yearlyHigh.Current.Value))
self.Schedule.On(self.DateRules.Every(DayOfWeek.Monday, DayOfWeek.Wednesday, DayOfWeek.Friday), self.TimeRules.AfterMarketOpen("SPY", 1), self.OpeningBar)
self.Schedule.On(self.DateRules.Every(DayOfWeek.Monday, DayOfWeek.Wednesday, DayOfWeek.Friday), self.TimeRules.BeforeMarketClose("SPY", 0), self.ClosingBar)
self.SetExecution(ImmediateExecutionModel())
def OpeningBar(self):
self.spyOption = None
if not self.IsMarketOpen(self.spy.Symbol):
self.Debug("Market not open, returning")
return
self.LogPortfolio()
if self.Portfolio[self.spy.Symbol].Invested:
self.Liquidate(self.spy.Symbol)
'''
if self.spyCoverOrderTicket is None:
self.Debug("Assigned, placing limit order to cover")
self.spyCoverOrderTicket = self.LimitOrder(self.spy.Symbol, -1 * self.Portfolio["SPY"].Quantity, self.Portfolio["SPY"].AveragePrice)
return
'''
high = self.yearlyHigh.Current.Value
if 100 * (high - self.Securities[self.spy.Symbol].Close)/high > self.withinYearlyHighPct:
self.Debug("Too far away from yearly high: " + str(self.Securities[self.spy.Symbol].Close) + " vs " + str(high))
return
options = self.OptionChainProvider.GetOptionContractList(self.spy.Symbol, self.Time)
if options is None or len(options) == 0:
self.Error("No options found")
return
callsExpiringToday = [o for o in options if o.ID.Date.date() == self.Time.date() and o.ID.OptionRight == OptionRight.Call]
sortedByStrike = sorted(callsExpiringToday, key = lambda o: o.ID.StrikePrice)
for option in sortedByStrike:
id = option.ID
if id.StrikePrice >= self.Securities[self.spy.Symbol].Open + self.callOtmAmount:
self.spyOption = self.AddOptionContract(option, Resolution.Minute)
mm = MyOptionMarginModel(self.spyOption, self.spyOption.MarginModel)
self.spyOption.SetMarginModel(mm)
self.Debug(str(id.Date) + " " + str(id.StrikePrice) + " " + str(id.OptionRight))
break
'''
if self.days > 30:
self.Quit()
self.days = self.days+1
'''
def ClosingBar(self):
self.spyOption = None
def OnData(self, slice):
for kvp in slice.Delistings:
symbol = kvp.Key
value = kvp.Value
if value.Type == DelistingType.Warning:
self.Debug("OnData(Delistings): {0}: {1} will be delisted at end of day today.".format(self.Time, symbol))
if value.Type == DelistingType.Delisted:
self.Debug("OnData(Delistings): {0}: {1} has been delisted.".format(self.Time, symbol))
if self.spyOption is not None and self.Securities[self.spyOption.Symbol].Price != 0:
self.MarketOrder(self.spyOption.Symbol, -5)
self.spyOption = None
return
def OnOrderEvent(self, orderEvent):
self.Debug(str(orderEvent))
oe = orderEvent
#self.Debug("OrderID: " + str(oe.OrderId) + " EventID: " + str(oe.Id) + " Symbol: " + str(oe.Symbol) + " Status: " + str(oe.Status) + " FillPrice: " + str(oe.FillPrice) + " FillQuantity: " + str(oe.FillQuantity) + " IsAssignment: " + str(oe.IsAssignment) + " Message: " + oe.Message)
if not self.spyCoverOrderTicket is None and orderEvent.OrderId == self.spyCoverOrderTicket.OrderId and self.spyCoverOrderTicket.Status == OrderStatus.Filled:
self.Debug("covered!")
self.spyCoverOrderTicket = None
def LogPortfolio(self):
if len(self.Portfolio.Keys) > 0:
self.Debug("Portfolio TotalMarginUsed=" + str(self.Portfolio.TotalMarginUsed))
for symbol in self.Portfolio.Keys:
holding = self.Portfolio[symbol]
if holding.Invested:
self.Debug(str(symbol) + " " + str(holding.Quantity) + " @ " + str(holding.AveragePrice) + " Delisted=" + str(self.Securities[holding.Symbol].IsDelisted))
Hi Karthik, Jared,
I often experienced the same problem recently, and today it seems systematic. Grateful for Karthik raising it.
Exactly as described above: " sometimes my backtest runs very quickly and the server statistics show >100% CPU usage, but other times it runs extremely slowly with <10% CPU usage ".
I experienced the issue randomly (apparently), with different algos. Most of the time, stopping / launching the backtest again (and/or sign out / sign in in between) was sufficient to get a normally running backtest. But not today.
I am not on a premium subscription (yet) and understand the consequences, but still I would not expect such a dramatic drop in available CPU - which really makes backtesting unbearably slow?
Thanks in advance for any guidance!
Pierre Vidal you're on the free tier - this is a shared CPU. The speed depends on the number of clients using the free tier. If you'd like a predictable dedicated node, which is also 40% faster in clock speed, I highly recommend upgrading =). This also supports the platform to keep us sustainable.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can
continue your Boot Camp training progress from the terminal. We
hope to see you in the community soon!
QuantConnect™ 2023. All Rights Reserved
|