Back

Debugging universe vs traded equities

Code below, but quick walkthrough of it.  Right now I'm just testing building the universe, later I'll be adjusting the long/short trades, but I have an issue where 1 symbol in the universe is not traded.  I used this code as my base and have been modifying it.

I've got a unverse built with 4 underlytings: DIA, IWM, QQQ, and SPY, that are pulled from a csv file. These underlyings are parsed from the columns of a pandas Dataframe so I can adjust which underlyings are traded by adding a column for it.  These are then added to the self.current_universe variable and returned by selector().  During debug logging I can see all 4 underlyings are added to self.current_universe, but when OnData() is processing SPY is nevertraded, so the portfolio is split between DIA, IWM, and QQQ.

Any thoughts about how to determine or debug why SPY is ignored?

Also, what is the best way to liquidate all holdings at the close of trading?

from clr import AddReference
AddReference("System")
AddReference("QuantConnect.Algorithm")
AddReference("QuantConnect.Common")

from System import *
from QuantConnect import *
from QuantConnect.Algorithm import *
from QuantConnect.Algorithm import QCAlgorithm
from QuantConnect.Data.UniverseSelection import *
from datetime import datetime
import decimal as d
import pandas as pd


class DropboxUniverseSelectionAlgorithm(QCAlgorithm):

def Initialize(self):
self.Debug("Execution Flow: Entering Initialize")

self.SetStartDate(2012,12,4)
self.SetEndDate(2017,10,13)

self.backtestSymbolsPerDay = None
self.current_universe = []

self.UniverseSettings.Resolution = Resolution.Daily;
self.AddUniverse("mxyzptlk", self.selector)
#self.Debug("Initializing, universe = " + ', '.join(self.current_universe))

def selector(self, data):
self.Debug("Execution Flow: Entering Selector")
##Fetch the csv file from dropbox and convert it to a pandas dataframe
# handle live mode file format
if self.LiveMode:
self.Debug("Execution Flow: Entering Selector::LivLiveMode")
# fetch the file from dropbox
url = ""
df = pd.read_csv(url, header = None)
# if we have a file for today, return symbols
if not df.empty:
self.current_universe = df.iloc[0,:].tolist()
# no symbol today, leave universe unchanged
return self.current_universe

# backtest - first cache the entire file
if self.backtestSymbolsPerDay is None:
self.Debug("Execution Flow: Entering Selector::backtestSymbolsPerDay")
url = "<REDACTED>"
#Create DataFrame from CSV
self.backtestSymbolsPerDay = pd.read_csv(url, header=[0,1], index_col=0)

index = str(data.strftime("%Y/%m/%d"))

if index in self.backtestSymbolsPerDay.index:
self.Debug("Execution Flow: selector::indexing")
self.current_universe = []
for item in self.backtestSymbolsPerDay.loc[index,:].dropna().index.tolist():
self.current_universe.append(item[0])

self.Debug("Selector, current_universe = " + str(self.current_universe))
return self.current_universe


def OnData(self, slice):
self.Debug("Execution Flow: Entering OnData")

if slice.Bars.Count == 0: return
if self.changes == None: return

# start fresh
self.Debug("Execution Flow: OnData::Liquidating")
self.Liquidate()

percentage = 1 / d.Decimal(slice.Bars.Count)
self.Debug("OnData/for(TradeBar), set holdings: slice.Bars.Keys = " + str(slice.Bars.Keys))

for tradeBar in slice.Bars.Values:
self.SetHoldings(tradeBar.Symbol, percentage)
self.Debug("tradeBar.Symbol = " + str(tradeBar.Symbol) + " @ \npercentage = " +str(percentage))

# reset changes
self.changes = None

def OnSecuritiesChanged(self, changes):
self.Debug("Execution Flow: Entering OnSecuritiesChanged")
self.changes = changes

 

2012-12-04 00:00:00 : Launching analysis for 3e0de60cbaa318a4e77bb7899d70a926 with LEAN Engine v2.4.0.1.2319
2012-12-04 00:00:00 : Execution Flow: Entering Initialize
2012-12-04 00:00:00 : Execution Flow: Entering Selector
2012-12-04 00:00:00 : Execution Flow: Entering Selector::backtestSymbolsPerDay
2012-12-04 00:00:00 : Execution Flow: selector::indexing
2012-12-04 00:00:00 : Selector, current_universe = ['DIA', 'IWM', 'QQQ', 'SPY']
2012-12-04 00:00:00 : Execution Flow: Entering OnSecuritiesChanged
2012-12-04 00:00:00 : Execution Flow: Entering OnData
2012-12-04 00:00:00 : Execution Flow: OnData::Liquidating
2012-12-04 00:00:00 : OnData/for(TradeBar), set holdings: slice.Bars.Keys = System.Collections.Generic.Dictionary`2+KeyCollection[QuantConnect.Symbol,QuantConnect.Data.Market.TradeBar]
2012-12-04 00:00:00 : Converted OrderID: 1 into a MarketOnOpen order.
2012-12-04 00:00:00 : tradeBar.Symbol = IWM RV0PWMLXVHPH @ percentage = 0.3333333333333333333333333333
2012-12-04 00:00:00 : Converted OrderID: 2 into a MarketOnOpen order.
2012-12-04 00:00:00 : tradeBar.Symbol = QQQ RIWIV7K5Z9LX @ percentage = 0.3333333333333333333333333333
2012-12-04 00:00:00 : Converted OrderID: 3 into a MarketOnOpen order.
2012-12-04 00:00:00 : tradeBar.Symbol = DIA R7KVSI4AAX5X @ percentage = 0.3333333333333333333333333333
2012-12-05 00:00:00 : Execution Flow: Entering Selector
2012-12-05 00:00:00 : Execution Flow: selector::indexing
2012-12-05 00:00:00 : Selector, current_universe = ['DIA', 'IWM', 'QQQ', 'SPY']
2012-12-05 00:00:00 : Execution Flow: Entering OnData
Update Backtest








Just to check, i adjusted the input data to place SPY as the first underlying with no success, it's still ignored but I have no clue why.  I even hardcoded the universe and still SPY was skipped.

self.current_universe = ['SPY', 'DIA', 'IWM', 'QQQ']2012-12-04 00:00:00 : Launching analysis for 6cc2a1f675755ece76fb8503d7a2748b with LEAN Engine v2.4.0.1.2324
2012-12-04 00:00:00 : Execution Flow: Entering Initialize
2012-12-04 00:00:00 : Execution Flow: Entering Selector
2012-12-04 00:00:00 : Execution Flow: Entering Selector::backtestSymbolsPerDay
2012-12-04 00:00:00 : Execution Flow: selector, setting index
2012-12-04 00:00:00 : Execution Flow: selector::indexing
2012-12-04 00:00:00 : Selector, current_universe = ['SPY', 'DIA', 'IWM', 'QQQ']
2012-12-04 00:00:00 : Execution Flow: Entering OnSecuritiesChanged
2012-12-04 00:00:00 : Execution Flow: Entering OnData
2012-12-04 00:00:00 : Execution Flow: OnData::Liquidating
2012-12-04 00:00:00 : OnData/for(TradeBar), set holdings: slice.Bars.Keys = System.Collections.Generic.Dictionary`2+KeyCollection[QuantConnect.Symbol,QuantConnect.Data.Market.TradeBar]
2012-12-04 00:00:00 : Converted OrderID: 1 into a MarketOnOpen order.
2012-12-04 00:00:00 : tradeBar.Symbol = IWM RV0PWMLXVHPH @ percentage = 0.3333333333333333333333333333
2012-12-04 00:00:00 : Converted OrderID: 2 into a MarketOnOpen order.
2012-12-04 00:00:00 : tradeBar.Symbol = QQQ RIWIV7K5Z9LX @ percentage = 0.3333333333333333333333333333
2012-12-04 00:00:00 : Converted OrderID: 3 into a MarketOnOpen order.
2012-12-04 00:00:00 : tradeBar.Symbol = DIA R7KVSI4AAX5X @ percentage = 0.3333333333333333333333333333
2012-12-05 00:00:00 : Execution Flow: Entering Selector
2012-12-05 00:00:00 : Execution Flow: selector, setting index
2012-12-05 00:00:00 : Execution Flow: selector::indexing
2012-12-05 00:00:00 : Selector, current_universe = ['SPY', 'DIA', 'IWM', 'QQQ']
0

As another test I added data for TLT and it was traded, but SPY is still persona nongrata.

0

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed