QuantConnect Community

Split/Dividend Factor Files.

Back

The YahooDataDownloader appears to work nicely.  Is there a similar tool for creating the factor files?  It appears the necessary data is available via yahoo. For example. http://ichart.finance.yahoo.com/x?s=AAPL&a=00&b=2&c=1962&d=04&e=25&f=2016&g=v&y=0&z=30000

If said tool does not exist, does anyone understand the spec of the factor files? Or more importantly the math required to do the conversion? https://github.com/QuantConnect/Lean/blob/master/Data/equity/usa/factor_files/aapl.csv

When comparing the two samples of data from above I also noticed the 7:1 Split occurs on Jun 9 for the yahoo data but appears to occur on Jun 6 in the factor file.  Not sure it matters, but maybe someone has an explanation.

Thanks!
 

Update Backtest



10 Comments
 
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

I worked through the calculations on a spreadsheet.  The results aren’t exact for the dividend adjustment, but probably close enough.
Looks like the Factor file columns are [Date of Previous Close, Dividend Adjustment, Split Adjustment].
I believe the calculations are as follows
Dividend Adjustment  = Prev Adj - Dividend / ( Prev Close * Split Adj)
Split Adjustment  =((Prev Close / Split ) / Prev Close)

Am I missing anything? 
 

0

Factor files have three colums: date, price factor (e.g.: dividend) and split factor.
The last value is always unity, because if we multiply the stock price for it, it will not change (be adjusted).

For instance, AAPL had three splits:

2000-06-21 (Wed) -> 2:1
2005-02-28 (Mon) -> 2:1
2014-06-09 (Mon) -> 7:1

Prices are adjuted for split on the last trading day before the event and previous splits are "refactored":

2000-06-20 (Tue) -> 0.0357143 (1/2)*(1/2)*(1/7)
2005-02-25 (Fri) -> 0.0714286 (1/2)*(1/7)
2014-06-06 (Fri) -> 0.142857 (1/7)

 

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Dividends factors are a little harder to reverse engineer; but from the LEAN Project the Dividend.cs file, and the FactorFile.cs

Distribution = close - (close * priceFactorRatio);priceFactorRatio = 0;
var index = _data.IndexOfKey(date);
if (index > -1 && index < _data.Count - 1)
{
// grab the next key to ensure it's a dividend event
var thisRow = _data.Values[index];
var nextRow = _data.Values[index + 1];

// if the price factors have changed then it's a dividend event
if (thisRow.PriceFactor != nextRow.PriceFactor)
{
priceFactorRatio = thisRow.PriceFactor/nextRow.PriceFactor;
return true;
}
}
return false;

You take the current factor / previous factor to get the factor ration and then apply that to the closes to get the price.

We absolutely need this in LEAN :) If you can make a toolbox Yahoo Split and Factor file generator it would be a huge help :)

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

>We absolutely need this in LEAN :) If you can make a toolbox Yahoo Split and Factor file generator it would be a huge help :)

I agree.  Being able to write algos in a real IDE with a debugger puts Lean at a huge advantage over the competition.  The first step was open sourcing the engine, which was done. The next hurdle remains getting local data. Once that hurdle is satisfied I suspect an increased rate of contributions from novices.  As these people aren’t typically going to drop thousands of dollars on purchasing data just to be able to play around.

Back to the topic.

My approach will be to assume daily data was already downloaded. In that case I’ll need to go lookup the previous close price. I’m unfamiliar with the Lean framework, so what’s the optimal way to retrieve this daily data from within our new split/div downloader?  I see LeanDataWriter.LoadHourlyOrDailyFile is kinda what I need, but I’m guessing there is a cleaner way that will return a nice data structure rather than dealing with file parsing.

 

0
Id assume it was independent from local data and use Yahoo. This way you're

getting splits, dividends and price from one place. You could even update

the existing yahoo downloader to get it as well.
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

> Id assume it was independent from local data and use Yahoo.
This is what I ended up doing.  However, I see a potential problem if someone is using the incremental merge capability of the Yahoo downloader. For example. Downloading a single day at the end of the trading day.  The easiest way around this for now is just to download the entire stock history every time.  Maybe a slightly more efficient way that keeps the merge capability is to look for any new split/dividend events within in window that’s specified. If any exists then force a full history download, otherwise do a merge.

Here are my generated results for AAPL. I can't figure out why the values are off slightly when compared to the values in git. The first dividend calculation on 20151104 results in 0.9957377 vs. 0.9957381

My equation looks like this
0.9957377 = PrevDividendAdjustment - (Dividend / (PeviousClose * PrevSplitAdjustment ));
0.9957377 = 1 - (0.52 / (122 * 1));

My Results
19980102,0.9288142,0.0357143
20000620,0.9288142,0.0357143
20050225,0.9288142,0.0714286
20120808,0.9288142,0.1428571
20121106,0.9330894,0.1428571
20130206,0.9376360,0.1428571
20130508,0.9434302,0.1428571
20130807,0.9500057,0.1428571
20131105,0.9565650,0.1428571
20140205,0.9623695,0.1428571
20140507,0.9683196,0.1428571
20140606,0.9738740,0.1428571
20140806,0.9738740,1
20141105,0.9788234,1
20150204,0.9831409,1
20150506,0.9870720,1
20150805,0.9912316,1
20151104,0.9957377,1
20501231,1,1

From GIT
19980102,0.9311149,0.0357143
20000620,0.9311149,0.0357143
20050225,0.9311149,0.0714286
20120808,0.9311149,0.142857
20121106,0.9351125,0.142857
20130206,0.9393839,0.142857
20130508,0.9448594,0.142857
20130807,0.9511153,0.142857
20131105,0.9573959,0.142857
20140205,0.9629841,0.142857
20140507,0.9687481,0.142857
20140606,0.9741588,0.142857
20140806,0.9741588,1
20141105,0.9790044,1
20150204,0.9832496,1
20150506,0.9871304,1
20150805,0.9912512,1
20151104,0.9957381,1
20501231,1.0000000,1
 

0

It looks like the closing price used to generate the value of 0.9957381 we find in the github file is $122.01:
0.9957381 = 1 - (0.52 / (122.01 * 1))

From my experiece, different platforms display different adjusted prices when we go back enough in time, so this slight discrepancy is expected. 

This reinforces the wisdom of Jared's proposed assumption, since it is better to know from where all your variables in your equiations come from and their value.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Jeff, I think your calculations are correct.  I also think Alexandre is probably right too, although I'm not sure where the data came from if that's the case.  Everywhere I have looked, including the data that comes from QuantConnect, the closing price on 11/4/2015 is $122.00.

I have created a pull request on github that expands the ability of the YahooDataDownloader to download split and dividend data. I also created a utility that can create a FactorFile object.  Using the Yahoo Split/Dividend data and the AAPL daily equity data, my utility (FactorFileGenerator) was able to confirm Jeff's calculations (up to a small rounding error).

https://github.com/QuantConnect/Lean/pull/592

0

Nice! I’m glad to have you take this one over. :)
I have a decent implementation working, but my code is incredibly stove piped.  
The only thing I didn’t notice in your pull request is how you are handling a split and dividend events that occur on the same day.  Dividends should be applied pre-split in most cases.

Here is my code if you would like to compare the output results.  
Program.cs : https://drive.google.com/open?id=0B7uLVigTKyqrRVB1Sm9uOW9oQmM
YahooDownloader.cs : https://drive.google.com/open?id=0B7uLVigTKyqrbmNfUXh3emhzVG8

Program.cs has a bunch a symbols hacked in for testing.  I also adding an option to cache the yahoo data locally on the first pass.  After the first run just set downloader.useLocalCopy = true  to regenerate the factor files from the cached copy.
 

0

Thanks Jeff! Your code will definitely be helpful.  I do have some questions about your feedback.  I think a better place to discuss these issues would be on the github.  I'm going continue the discussion there, if that's alright?

https://github.com/QuantConnect/Lean/pull/592

0
Update Backtest



0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Loading...

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!

Discussion Tags