QuantConnect Community

Hi Jing, this is really great. Thanks for sharing your work. I am looking forward to exploring the math further. Your explanation seemed to really stream-line the difficult/important concepts into a more understandable framework, which I appreciated. Is there anything you were able to identify for why the strategy's activity slowed down from October 2014 to December 2015?

Hi FX trader,  For this pairs trading method, the chosen pair is “GLD” & “DGL”. In comparison to the profitable period from October 2012 to Feb 2014, if we take a close look at the historical log returns from October 2014 to December 2015,  the returns are less volatile and the 2 curves almost overlap with each other. Not much price divergence for pairs traing profits. Although we have trades during this period, the profits offset from longing and shorting this pair simultaneously 

It would be a good project to try copula pairs trading on not just ETF but stock pairs because of the volatility of stock market. It is promising to try high-frequency pairs trading with copula method. See Xiaowei's great example for High Frenquency Pairs Trading for cointegration Method Based on Stock Market 

 -- sorry for my late reply. Thanks for taking the time to look in to this and get back to me. This makes me interested in having a sort of dormant tag, that attempts to measure and perceive how similar current market conditions are with historical conditions that have been identified to place ideal trades. More for a reference framework. And thanks for sharing this strategy too

This is the new version of pairs trading algorithm with copula method. I fixed the bug of the unsupported decimal and float multiplication.

Hi Manoj Gupta, try to clone the new version algorithm in my last comment or check tutorial page here https://www.quantconnect.com/tutorials/pairs-trading-strategy/ for the updated algorithm

Hi Hugh Donnelly, it can be amended to trade multiple pairs at the same time, but the performance based on the candidate pairs you choose. It's kind of difficult because the copula parameter estimation process has to be adjusted for multiple pairs in Algorithm initialization and then the Ondata trading part accepts different input and trigger the different signals. Almost all the parts have to be amended. It might be more efficient to just change the candidate pairs in algorithm if you want to see the performance of copula method for other pairs.

Jing Wu -- Thanks, I will take a look.

I am very new here, and not sure yet how to do many even simple tasks, so apologize for not doing the work!  In this instance, visually speaking, it appears the algo is correlated with SPY, but I am not sure how to numerically prove this claim: i see beta=0.57, and given that volatility of your portfolio is most certainly lower than that of SPY itself, it means the correlation with SPY is definitely above that number. That should not be the case for a pair trade between SPY and DIA - but it seems there is a long market bias in the strategy right now.  Maybe there is something in portfolio construction that needs to be addressed?

Hi,

Thanks for the post and introducing me to this novel technique. I am wondering why you use the "hedge ratio" as below:

Shouldn't the portfolio be dollar neutral? That is equal amount of both pairs with one long and one short?

Best regards,

Pravin

Hi Satyapravin, dollar neutral is one of the methods to decide the hedging ratio in pairs trading. Here I use the beta neutral hedge ratio Price(Stock A) - beta * Price(Stock B) (beta is the coefficient in simple linear regression) not the dollar neutral ratio. Then I purchase x shares of stock A and short x*beta shares of stock B.

Hi Karen Chaltikian Thank you for your question! It's a very interesting perspective.

I understand what you are saying, and I test the market volatility within the same period, which is 0.14, and the algorithm's volatility is 0.104. In this case, the correlation coefficient between the market and the algorithm must be a number higher than 0.57. 

I will try to look into the portfolio construction! Hopefully we can figure it out.

Hi Jingw, I have some questions:

1). Is this supposed to pick only one pair out of 8 pairs?  2). Can I put 100 pairs in your code 'def _pair_selection()'?  3). I cloned the fixed version but it showed different result from the one you posted. What am I missing here? (See attached backtest.)  4). How does this algo control leverage? In the code, the quantity of leg1 is set up always 0.4. I am curious about total maximum leverage.  5). In your tutorial, you said "...We use the first 3 years of data to choose the best fitting copula and asset pair ("training formation period"). Next, we use a period of 5 years from 2011 to 2017 ("the trading period"), to execute the strategy...." What does this mean? Is this sort of Machine Learning algo? What does exactly mean by 'training formation period' and 'trading period'? 

Hi 

I changed a little: trading everyday after 15 min. after market open, and added 'QQQ', 'XLK'. I got en error message as below around November 7~14, 2016:

Runtime Error: System.Exception: BacktestingRealTimeHandler.Run(): There was an error in a scheduled event XLK: EveryDay: XLK: 15 min after MarketOpen. The error was Python.Runtime.PythonException: KeyNotFoundException : 'QQQ' wasn't found in the Slice object, likely because there was no-data at this moment in time and it wasn't possible to fillforward historical data. Please check the data exists before accessing it with data.ContainsKey("QQQ")
at QuantConnect.Data.Slice.get_Item (QuantConnect.Symbol symbol) [0x0002d] in <8f9e699bbc3f46739adc6b359ebcd5b9>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in <dca3b561b8ad4f9fb10141d81b39ff45>:0
at Python.Runtime.Dispatcher.Dispatch (System.Collections.ArrayList args) [0x00018] in <387056c9810b431d9b668f2df5d6c027>:0
at __System_ActionDispatcher.Invoke () [0x00006] in <166e461c70884c28abc297d007aa4fef>:0
at QuantConnect.Scheduling.ScheduleManager+<>c__DisplayClass16_0.<On>b__0 (System.String name, System.DateTime time) [0x00000] in <8f9e699bbc3f46739adc6b359ebcd5b9>:0
at QuantConnect.Scheduling.ScheduledEvent.OnEventFired (System.DateTime triggerTime) [0x00036] in <8f9e699bbc3f46739adc6b359ebcd5b9>:0

To me, it's impossible not to find the symbol 'QQQ'. What's happening here? Should I change the trading schedue? Thanks :)

I am not sure why you are having a problem. I have modified the algorithm to only trade 'QQQ' and 'XLK', and set the schedule to 30 minutes after market open on the same month start days. Maybe you can see what is different between this version and yours to help you.

Hi Jingw, Karen Chaltikian, As I said above ("...trading everyday after 15 min. after market open..."), I changed the trading interval from once a month to a daily basis. I tried different schedule but in November 2016, the same error message showed up. I tried with Jingw's version and Karen's version, but both had same error messages.                                           

I don't know what happened then. I guess there's a bug we missed. I will look into this more in detail. Any suggestions and comments are welcome. Thanks :)

Jingw, I have more questions. Please see the attached backtest for the period from 2016,01,01 to 2017,9,20 and its trades logs as below. As you can see below, the quantity for both legs decreased dramatically from 2016-02-19. 

And I thought that it trades just once a month on the first day of the month but it does not. Is this supposed to do this? What am I missing? Thanks :)

Jingw, I attached a backtest for the year 2017. It looks very disappointing and has a lot different outlook from the one above. Do we need some sort of warm-up period? Can you explain about this? Thank you :)

Hi HanByul, yes this algorithm needs model selection and training period. For the two parameters here,

history data to find the best fit copula from Archimedean copulas class, and use 250 days to train the chosen copula function to find the best parameter estimation. For pairs trading, when to trade and how many trades each month really depend on if the criteria is satisfied and can't control manually. But for this reason, when there is a trading opportunity for buying the pair(buy first sell second), it could happen that the last trading opportunity is still buying signal, then since you already hold the long position, you have limited money to buy new ones. That's why the quantity decreased dramatically sometimes. But the decreasing trading quantity can be modified by changing the leverage or you can set your own trading rules when there is a pairs trading opportunity.

Jingw, Thank you for your explanation. Regarding the quantity of each leg, I think it's understandable. But I still don't understand why trading from year 2017 (see backtest right above.) looks so disappointing. Compared to the backtest starting 2016, two results are not giving me any level of confidence. If I deployed this algo in Jan. 2017, I would've gotten so bad result. However, if I did in Jan. 2016, it would've been totally a different story. What do you think? Is there any other way to improve regarding this aspect? How will this algo go in live trading? Thanks :) 

HanByul, It is reasonable. As in algorithm, we just choose copula once and use this copula in the following days. The only thing that changes is the parameter of that copula. Your backtest starting from 2016 and starting from 2017 use the different history data to choose the copula, they might choose the different copula and the algorithm will use this copula for the following backtesting. You can control the copula you use by ignoring the copula selection process to reach the same result.

Jingw, Thanks for all the answers and explanation. I will dig into more about 'copula' method and see if we can implement for multi-pairs instead of picking one pair (If you've already done for multi-pairs, please let us know. I appreciate it.). Again, thank you for your great work and the opportunity to explore the 'copula' method.  :)

your universe is: tick_syl =  [["SPY","AGG","XME","TNA","FAS","XLF","EWC","QLD"],
                                         ["DIA","JNK","EWG","TLT","FAZ","XLU","EWA","QID"]]

as a non-programmer, I'm wondering if I can replace names here and run backtest with my names?  I've tried and gotten errors

If for a Gumbel copula we get a combination of (u,v) of (1,1) -> the pdf of your algorithm returns an infinity value.

Hi Dennis,

When (u, v) -> (1, 1), log(u) and log(v) are zero so the Gumbel pdf will be "inf". u and v are  constructed with the empirical distribution function from the log return series, there must be a 1 in u or v. Therefore, in the original algorithm I try to convert the infinity value to the finite number with

# Replace nan with zero and inf with finite numbers in lpdf list 
lpdf = np.nan_to_num(lpdf)

The finite numbers are still huge so the sum would be inf. A better solution would be converting the 1 in u or v to a number less than 1 for example 0.9999.

Hi,

I cloned the file and tried to run a backtest but its giving me an error

During the algorithm initialization, the following exception has occurred: TypeError : 'numpy.float64' object is not callable
at Initialize in main.py:line 61
at <listcomp> in main.py:line 61
at _lpdf_copula in main.py:line 180
TypeError : 'numpy.float64' object is not callable

Please let me know how to fix this error.

Here's the log

I recloned main.py and now its giving me divide by zero

Tweaked it to check for at least 3 days. The -2 position wasn't set yet.

Thanks Jared! worked like a charm

One more question, I am trying to use similar strategy for currencies, is it possible to change the ticker to use currency?

2 currency pairs: EUR/USD and NZD/USD

test for cointegration of prices between both assets

Two Questions:

1. How is the Quantity calculated? (i.e. where can I find the function CalculateOrderQuantity() )

2. 1200 days to determine which copula and 250 days to determine the parameters? So 1200 days are used to decide Frank/Gumbel/etc. and the parameter theta is optimized using the 250 days before each trade? Or how many times and when is this parameter updated?

Thanks in advance,

Dennis

Hi Jing Wu,

I have tried the pair selection as you did by selecting the highest Kendall’s tau of the pair log return. I ran though the data of entire NYSE stock list and pick the highest 30 stock pair of Kendall's tau each year from 2001 to 2017. Then I backtested the pair on the next year historical data. I find that the high Kendall's tau doesn't has a direct relationship with good performance. P.S. the backtesting is using the stratigy that you posted out.

Is there a better way to pick a good pair?

Hello,

How do I implement this strategy for forex pairs? Thank you

Hi All!

Sincere apologies for the above for some reason the page did not load for me.

I am really fasciantated with the above and would like to build on it.

I have a couple of quesions -

If i want to try another type of copula in a python package, how can i import the package

My first impression is that i could only do it locally.

So i would like to set up a local backetest enviroment preferably in jypyter notbeook.

What would be the best way to acheive this?

Many thanks all,

Best,

Andrew

For the request to add more python libraries, please submit an issue on Github 

We'll review the request and try to add new libraries to LEAN.

To use LEAN locally. the easiest is with a plugin we've made for Visual Studio. It automatically takes the code you've written and sends it to the cloud for backtesting.

You can see the plugin documentation here; we would love to have support for other IDE's but currently just support Visual Studio.

I am new to the platform, but can someone explain this to me and why is it 0.4?

Hi Ted Yap, you can look for the QCAlgorithm classes, method, and properties in the Algorithm Lab: 

what is the exit point?

I checked the copula code from the tutorial (one with 427 trades, 353% Net Profit). If you look at the trade log, it seems that it opens a long position in XLK on 2010-03-03 and the short QQQ gets rejected because of insufficient capital. Another pair trade is opened on 2010-03-26 in opposite direction and half the size. Then the two ~2k shares long positions in both instruments are held for 8+ years, producing most of the total net profit. Am I reading this right?

Hi Petras,

Your reading of the trading log is correct. Please let me know if you have any further questions.

I do have a question. Do you plan fixing this bug? Because as of now the tutorial page shows a completely bogus and unrealistic strategy and results. When I fixed the bug by reducing trade sizes the resulting equity curve was basically a random walk.

Hi Petras, 

First of all, thank you for your feedback, but we didn't find any problem with our system. 

Regarding the strategy itself, the trading logic is correct. The invalid trade order happened on 2010-03-03 you observed is due to insufficient remaining capital. I've added an order size constraint in the code to avoid this. Please check out the attached backtest and especially the part of placing orders under OnData() method. 

Additionally, I've refactored the code to convert it into a universe selection model. If you want to discuss further, feel free to contact me here. 

Best regards,

Xin

Could you give me even a few tip how can I implement this strategy for only one forex pair? Cheers.

Hi Jonas,

To apply this algorithm to forex pairs, we need to redefine the `symbols` variable in PairSelection. Since `Symbol.Create` doesn't work with forex, we can use AddForex in Initialize. For example,

In PairSelection, we assign this list to the `symbols` variable.

The last thing to do is add a condition to check the `history` DataFrame before accessing it with a symbol.

See the attached backtest for reference.

Best,
Derek Melchin

Thank you so much.

I'm really interested in this strategy. So I'd like to check the backtest result myself.
By the way, could you let me know if it is possible to run on lean engine? Actually I tried to run this code base on my local, but it is failed.
Cheers.

I'd like to run this strategy on my local development with forex tick price data, not daily or minute data.

It would be appreciate if you let me know is it possible to run on local with forex tick price data, and one pair of two symbol. Even though the backtest result is not good.

Best,
Jonas

Hi Jonas,

It's possible to backtest this locally with tick data, but we only provide local forex data at the minute resolution or lower.

Best,
Derek Melchin

@Jing Wu: By stating that we short, do you mean short-selling?

@Jing Wu: I see you had to liquidate the first and second assets before selling and buying them, respectively. I don't understand why you have to do this. Please, could you explain to me?

if self.Portfolio[self.syl[0]].Quantity < 0 and self.Portfolio[self.syl[1]].Quantity > 0:

                self.Liquidate(self.syl[0])

                self.Liquidate(self.syl[1])

                quantity = float(self.CalculateOrderQuantity(self.syl[1],0.4))

                self.Sell(self.syl[1], 1 * quantity )

                self.Buy(self.syl[0], self.coef * quantity)

            else:

                self.Sell(self.syl[1], 1 * quantity )

                self.Buy(self.syl[0], self.coef * quantity) 

Hi Michael

The original opening positions of the 2 assets are opposite to the desired directions. You can liquidate them first before ordering the desired quantity from trading signals for the ease of coding/calculating, but that would be subjected to extra trading fee.

The suggested way of doing that will be:

quantity = self.CalculateOrderQuantity(self.syl[1],0.4)

if self.Portfolio[self.syl[0]].Quantity < 0 and self.Portfolio[self.syl[1]].Quantity > 0:

   self.Sell(self.syl[1], 1 * quantity + self.Portfolio[self.syl[1]].Quantity)

   self.Buy(self.syl[0], self.coef * quantity + abs(self.Portfolio[self.syl[0]].Quantity))

else:

   self.Buy(self.syl[1], 1 * quantity + abs(self.Portfolio[self.syl[1]].Quantity))

   self.Sell(self.syl[0], self.coef * quantity + self.Portfolio[self.syl[0]].Quantity)

Best,
Louis Szeto