Back

What's the best way to download large amounts of FXCM tick data?

Hi!  I know you have the FxcmDownloader in the QuantConnect.ToolBox project and it appears to be working, but it's glacially slow.  For instance, downloading FXCM ticks from 1/1/2012, I'm at 0.62% after 90 minutes.  Is there any better method?

Update Backtest








@Gustavo is working on this as we speak! :) In the next 7-10 days we'll allow bulk downloading of FXCM data files in LEAN format. We're planning to add it to the QC-API and automatically download data required for your backtests when its available. We've spent weeks downloading all the FXCM data, checking it and then converting it to LEAN format.

I'll ping this thread when its done with the link.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Perfect!  And thanks for the quick response.

BTW, if you need any help testing just say the word.  I am a very experienced developer (and at various times the CTO of six different companies--three of them trading firms) so I have a fair amount of experience with such matters.

In a slightly related topic (for another project not related to QuantConnect) I've written an FXCM Tick Downloader based upon the .NET libraries; not the IKVM'd Java code you're using.  The fetch 300 ticks at a time then iterate method is less than pleasant, although I've addressed a part of the pseed issues with lots of threads. 

Away, I'm loving QuantConnect.  Keep up the good work...

 

1

Thank you Louis we'd love the help -- please join us on the slack group and message me when you have some time free! 

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Just following up on this thread. We've pushed out the data library - its still very beta but you can download FXCM data in bulk here: https://www.quantconnect.com/data

Select the symbols you'd like and it'll be added to your private data collection in QuantConnect :) You can download this data and use it locally. Soon we'll make this automatically download if the data is in your web library.

FXCM is free; we're working with other data providers to setup pricing now.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Related: I've added a bunch of Oanda data to my library. How do I download the giant list of files resulting in the smoothest manner possible?

I tried wget but it seems to download access tokens rather than the files themselves.

0

Ended up using the Oanda downloader in toolbox for now, would still prefer to be able to do this without messing around with command line options/startup project in MSVC tho :)

1

Understood. We plan to allow automatic downloading free data from QuantConnect when you're running a backtest. 

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed