Back

[Python] How to get portfolio and stock data into classes that are in other files?

I'm currently porting over my algorithm from Quantopian and having some trouble. I use a framework for trading multiple algorithms on the same account simultaniously, using classes for each algo and some execution/decision classes tying them all together. Currently my algorithm is >1500 lines of code and seeing as how QC has a nice file system I figured I could move some of my utility funcions or even the algorithm classes over into separate files. I currently have 6 algorithm classes, each one is generating a buys/sells on different ETF's using price history data. Then there are some classes that make a decision on all those buy/sell signals, combining them into one portfolio. Then I have utility funcions such as a stop loss that also need price history data. It would be very inefficient/confusing to have them all running in OnData() all the time.

Here is my current understanding, maybe you can help with the gaps:

Quantopian: (this might not be ideal but it's how the framework that I'm using is set up, I didn't write it originally, still learning)

execution_manager = ExecutionHandler_Market()
def handle_data(context, data):
target = desired_allocation
execution_manager.execute_orders(context, data, target)
def utility(context, data):
#do some stuff with data
#Define the Execution Handler module type.
class ExecutionHandler(object):
def execute_orders(self, context, data, target_portfolio):
raise NotImplementedError()
#ExecutionHandler_Market is an ExecutionHandler object that makes market orders
class ExecutionHandler_Market(ExecutionHandler):
def execute_orders(self, context, data, target_portfolio):
utility(context, data):
#do some other stuff with data

QC:

?

I understand the general part of how to get functions/classes working across multiple files but I don't understand how to get account/stock/hisotry data into those files. They all need to access variables in Main.py, portfolio objects such as account cash value, holdings, price data, etc.

I've tried looking for examples of what I need but most algorithms on QC do all the data handling in OnData() and only put minor extranious functions in other files.

Update Backtest








Hmm the formatting on the code is hard to read, let me try again:

 

execution_manager = ExecutionHandler_Market()

def handle_data(context, data):
target = desired_allocation
execution_manager.execute_orders(context, data, target)

def utility(context, data):
#do some stuff with data

#Define the Execution Handler module type.
class ExecutionHandler(object):
def execute_orders(self, context, data, target_portfolio):
raise NotImplementedError()

#ExecutionHandler_Market is an ExecutionHandler object that makes market orders
class ExecutionHandler_Market(ExecutionHandler):
def execute_orders(self, context, data, target_portfolio):
utility(context, data):
#do some other stuff with data
0

I'd try to cut the framework out.
Since you are running multiple algorithms over the same account, try implementing each algorithm as it's own, independent QCAlgorithm class.
Take the securities you want to trade as inputs to a constructor or otherwise make them settable from the outside.

Then you can do a main QCAlgorithm class that delegates to the others.

Also note that onData is merely the most talked about why to trigger trade decisions in QC land, schedule functions still work about the same as in Quantopian and you can even get your "data" object back by caching it in an instance variable.

public override void Initialize() {
SetBenchmark(this.benchmark.Symbol);
Schedule.On(DateRules.EveryDay(this.benchmark.Symbol),
TimeRules.AfterMarketOpen(this.benchmark.Symbol, 60),
rebalance);
Schedule.On(this.DateRules.EveryDay(this.benchmark.Symbol),
this.TimeRules.AfterMarketOpen(this.benchmark.Symbol, 120),
update);
}

public override void OnData(Slice data) {
this.data = data;
}
0

Wow Thanks, you really got me thinking and after about 2 hours of failing I think I figured out a workable solution. I wasn't able to figure out what you meant by making multiple independent QCAlgorithm classes but I did figure out how to pass the OCAlgorithm parts that I need (data etc.) back and forth between the two files.

You can see what I did in the attached backtest. If you can describe in more detail the way to implement what you suggested I'd be open to that as well.

0


Lukel, like you I am converting a multi-strategy algo (800+ code lines) with sub-strategies trading groups of ETFs from Quantopian. Your post was a breakthrough for me. If I understand your QC solution, each sub-strategy gets its own file.py and the master algo calls each sub from main.py? Would you share more on your latest progress coding a multi-strat? What were the major stumbling blocks as you got further along and how did you resolve?

Everyone else, I am looking for more examples in Python so I understand QC structure and syntax correct.  If you have resources beyond the posted github examples, please point me in the right direction! Trying to reach the promised land by Sep 29, which is approaching all too quickly.

0

I have learned a lot since the post above, and achieved success in coding a multi-strategy algorithm. 

I left out a lot of the ordering stuff, and classes that the algorithms inherit from but that's all pretty unnessesary. here is just the bare bones:

1


Lukel, Great work! I will look into your multi-algo structure. Thanks :)

0

Lukel, many thanks. I appreciate your willingness to share.



If I understand the structure you're adopting, the symbols for all algos

get aggregated into Initialize in main.py. Per QC Docs, indicators eg EMA,

RSI also sit inside Initialize in main.py. So the structure works great

with only a couple indicators. For a multi-strat algo with 10 symbols and 2

unique indicators, the IDE pulls data from 10 x 2 = 20 feeds in main.py.

However, for a strategy with 10 symbols and 10 indicators, the algo now

pulls data from 10 x 10 = 100 feeds. The difference widens considerably

with more symbols and more indicators. Over multi-year backtests I suspect

the processing speed will be unacceptably low. Would also worry about

processing speed in a live algo environment, even using .SetWarmup. Am I

thinking about this correctly?



I am considering a list format all in main.py with separate lists for each

strategy's symbols and indicators similar to my Quantopian structure. That

way I'm not pulling every indicator for every symbol in my algo, which

would be hundreds of data feeds.



Would like to hear your thoughts. It's quite possible I'm misunderstanding

something. Thanks. -Brett
0

Brett,

I won't pretend to know the intricacies of how the data is pulled from QC but i think you have a slight misunderstanding. The indicators for each strategy are within each strategies .py file, in the compute_allocation functions. The indicators only use the self.stocks of that strategy. In return it fills up a dictionary (allocation) that the main.py or another file with execution logic can call individually and merge with all the other allocations.

 

Main.py only has the complete list so that it can call the data and have a record of which stocks belong to the strategies so that they can be used for execution in conjunction with the allocation.

 

I am probably not explaining it well but for example: algorithm1.py uses its self.stocks, SPY and TLT, with a SMA cross and creates an allocation. Algorithm2.py uses its self.stocks, SPY and GLD, with an EMA cross and generates an allocation. Each indicator only sees 2 stocks. Main.py iterates through the allocations to generate a combined allocation, probably combined_allocation[stock]+=algo.allocation[stock]*1.0/n where n is the number of algorithms. Then it uses the unique combined securities list (spy,tlt,gld)  to place orders based on the combined allocation without having any repeats (which would be problematic).

You really don't need to use the combined list a while lot. I use it for execution and checking to make sure the stocks in my universe thats also in the algorithm aren't part of the other strategies before purchase.

 

Sorry for typos I'm on mobile right now

1

Makes sense. I tried moving the indicators out of Initialize and, yes, it

works fine. Are you up and running with live trading, or at least live

paper trading now?
0

I got the basic implementation done, now I'm double checking to make sure every function behaves in exactly the same way as I expect. I'm finding that there's a big difference between successfully executing code and getting the same behavior out of a signals and execution on both platforms.

The only thing I'm worried about is the memory limit, I can run a backtest for about 3 months before hitting it, I just don't know what/how to optimize if I hit it on live trading.

I think I will be ready to begin paper trading by the time Quantopian ends their live trading on the 29th

0

Luke,

You give me hope! Just curious, are you running on 1 account only ? I am RIA but trying to run 10-15 instances of same algo (with slightly different var settings on each to reflect risk tolerances). Still hoping to know what that will cost me from QC but no response yet.

0

LukeI, Well done! After testing your paper (or live) trading with QC, can you let us know how it went compared to Quantopian? Thanks :) 

1

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed