Avoiding Vendor Lock-In: Running LEAN On Your Server

Back

Hello Everyone! Given events lately I wanted to directly address fears of vendor lock-in from choosing to use QuantConnect/LEAN as your trading platform by giving you a step-by-step guide for how you could run LEAN on your own server. 

- Why would we do this? We believe you are using QuantConnect to save you time and effort setting up the environment on your own. With QuantConnect you can get better features, data and UX than possible alone. This motivates us to provide an outstanding service so that even if you can run it yourself; you'll likely choose to use us. 

 - Why did we make LEAN powerfully open? We designed LEAN to be runnable locally; and spent weeks working on datafeed connections only relevant to the open-source project. Why would we do this? Like Linux we see LEAN as the foundation for the future of algorithmic trading. There are millions of features to build and there's only so much we can do ourselves so by making open-source a core part of our project we encourage contributions to help make it a truly community driven company. Want to see the project thrive? Help contribute

So to get started! Its pretty simple:

1) Core Technology - LEAN:

  • You will need LEAN - our cross platform open-source project kernel of QuantConnect. It builds and runs locally with almost no effort. We ship small data samples with your project so you can see it working instantly. Windows, Linux or Max are supported. By default it is a CLI program and we provide a rudimentary GUI but community members have built beautiful GUI's for Windows.

2) Infrastructure:

  • For live trading you'll need a virtual machine of at least 512MB RAM. You can pick one reasonably stable micro-machines from AWS ($7/mo) or Google Cloud Compute ($5/mo). I wouldn't recommend super discount providers as the reliability of internet connection is important in live trading. For backtesting you can backtest locally in Visual Studio.
  • OS Setup: We provide a docker file so you can setup the OS instantly by following along with the docker setup - or running just running LEAN in the docker container we provide.

3) Data:

  • Backtesting:

    We use Zip-CSV format for the data as it is an open format easily reverse engineered.

    Equity / Futures / Options: This is the only tricky part. Due to exchange licenses we can't give this to you but we can make it as easy as possible to get started. We work with data vendors who offer very reasonable rates and we have created and open-sourced data converters to convert their data into LEAN format. These converters all live in the ToolBox project of LEAN. We also provide ways to download US Equity data from Google's API but this is limited to the last 2 weeks.

    Forex / CFD: We have negotiated on your behalf and FXCM / OANDA have agreed to make their data available for free download. We provide this for free through the Data Library.

    Coarse Universe: For backtesting coarse universe is available via the Coarse Universe converter in the toolbox. This is reasonably easy to use but still requires you to gather the entire universe of equity daily prices. Fine Universe (fundamentals) is harder as its provided by Morning Star. We chose Morning Star as they had the best coverage but its not a cheap provider. We haven't open-sourced this as we think given price of data its an institutional offering not available for an individual (hedge funds - contact us). Given Morning Star costs its probably a good idea to create a Quandl fundamental converter in the long run (WIP).
     
    For more information on downloading data check out: GoogleYahoo, FXCM, Dukascopy Data Downloaders.
    You can also see the QuantQuote, NSE, AlgoSeek Options and Futures Data Converters and the FactorFile Generator.
     
  • Live Trading: 

    Streaming Data: We've provided many mechanisms for live data streaming in the open-source project. These classes are called the "IDataQueueHandler" implementations. They broadly come from three sources:
       1) Brokerage implementations - stream live prices directly from your brokerage with your data subscriptions. We've implemented IB, OANDA and FXCM brokerage IDataQueueHandler's so you can stream directly from them with almost no effort. Each brokerage will be limited in the data it can stream (i.e. FXCM can't send you options pricing). 
       2) Data vendors - we've implemented and open-sourced an IQFeed data feed. as IQFeed supports all our security types. Other data vendors weren't reliable enough.
       3) IEX Exchange - an implementation to the free IEX exchange datafeed for US Equity prices.

    History Requests: History requests are tricky as its actually pretty complex engineering to store streaming prices and serve them up quickly. We have provided open-source implementations to the brokerages we support. Data vendors can't support this at the moment.

    Universe Data: Live trading with universe data is also a challenging task to open-source as we're using lots of proprietary data to generate the full listing of symbols and prices for the universe. The format is not proprietary though and you can provide your own universe in live trading by serving a collection of CoarseFundamental objects inside of a BaseDataCollection delivered by the same IDataQueueHandler used above. It is processed here. We'll ponder on this one to see how to open-source it for personal use with publicly available data sources.

We're grateful for all of the community and will continue working to give you the best algorithmic trading platform in the world so you chose to spend your time and energy with us! 

 

PS: If you're like to contribute - there is lots to do =)

  • #1 Helping community members and sharing algorithms. Helping the community thrive is a huge help to the core team.
  • #2 Grab a bug or feature from the issues list and submit a PR. Finding and submitting descriptive bugs is a big help too.
  • #3 Contributing to our new open-sourced tutorials which are deployed here.
  • #4 Submitting new example algorithms for C#, F# and Python.
  • #5 Lastly if you're too busy you can subscribe to support the project financially.
Update Backtest






The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.



 
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hi KY,

Yes running Lean on your own is possible even if SLS opt-out is not prohibited in your country/region. Let us know if you have any problem in the setting up and deploying process of your server for Lean. Thanks for your support!

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hi, 

I am trying to run Lean on my local server for live trading FX with Oanda. So what I have did so far are 

Changed certain fields in `config.json`

"enviroment": "live-oanda",  "oanda-environment": "Practice",  "live-oanda": { "live-mode": false, ....}

and filled fields "job-user-id", "api-access-token", "oanda-access-token",   "oanda-account-id" .

Then rebuild the solution. And ran a simple algorithm. However, I got an error 

 

ERROR:: Engine.Run(): Error running algorithm System.InvalidCastException: Specified cast is not valid.
at QuantConnect.Lean.Engine.Results.LiveTradingResultHandler.Initialize (QuantConnect.Packets.AlgorithmNodePacket job, QuantConnect.Interfaces.IMessagingHandler messagingHandler, QuantConnect.Interfaces.IApi api, QuantConnect.Lean.Engine.TransactionHandlers.ITransactionHandler transactionHandler) [0x00016] in <bc7b4fac3a6b4b2586612532d9acbdee>:0
at QuantConnect.Lean.Engine.Engine.Run (QuantConnect.Packets.AlgorithmNodePacket job, QuantConnect.Lean.Engine.AlgorithmManager manager, System.String assemblyPath) [0x000b1] in <bc7b4fac3a6b4b2586612532d9acbdee>:0
20191216 09:26:36.682 ERROR:: LiveTradingResultHandler.StoreLog(): System.NullReferenceException: Object reference not set to an instance of an object
at QuantConnect.Lean.Engine.Results.LiveTradingResultHandler.StoreLog (System.Collections.Generic.IEnumerable`1[T] logs) [0x00000] in <bc7b4fac3a6b4b2586612532d9acbdee>:0

 

Would anyone please help me with this? Or point out where can I find any documentations for setting up live trading locally. 

Thanks in advance.

Xiao

 

0

I figured its my enviroment setting not correct, now everything works with the following settings

"live-mode-brokerage": "OandaBrokerage",
"data-queue-handler": "OandaBrokerage",
"setup-handler": "QuantConnect.Lean.Engine.Setup.BrokerageSetupHandler",
"result-handler": "QuantConnect.Lean.Engine.Results.LiveTradingResultHandler",
"data-feed-handler": "QuantConnect.Lean.Engine.DataFeeds.LiveTradingDataFeed",
"real-time-handler": "QuantConnect.Lean.Engine.RealTime.LiveTradingRealTimeHandler",
"transaction-handler": "QuantConnect.Lean.Engine.TransactionHandlers.BrokerageTransactionHandler",
"history-provider": "BrokerageHistoryProvider"

 

0

Hello Jared
In the very first post of this thread you wrote

>>> You will need LEAN - our cross platform open-source project kernel of QuantConnect. It builds and runs locally with almost no effort. We ship small data samples with your project so you can see it working instantly. Windows, Linux or Max are supported. By default it is a CLI program and we provide a rudimentary GUI but community members have built beautiful GUI's for Windows.

Can you suggest what are those Windows GUIs, I have found only AlgoLoop on github.


Thanks
Libor

0

https://github.com/mirthestam/lean-monitor
https://github.com/Capnode/Algoloop

I didn't know about AlgoLoop! Very cool =)

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hi,

Today, I could successfully install QC-Lean on CentOS-7  VPS and run C# default algo.  When I tried to run python algo, I faced some error.  From error message, it looks like QC is looking for puthon 3.6 dll.executable where as I have installed python 3.8.1. on VPS.

20200118 15:49:13.936 ERROR:: Loader.TryCreatePythonAlgorithm():  System.DllNotFoundException: python3.6m assembly:<unknown assembly> type:<unknown type> member:(null)
  at (wrapper managed-to-native) Python.Runtime.Runtime.Py_IsInitialized()
  at Python.Runtime.Runtime.Initialize () [0x00000] in <c66998abee2f4df39ea2647af887c7b2>:0
  at Python.Runtime.PythonEngine.Initialize (System.Collections.Generic.IEnumerable`1[T] args, System.Boolean setSysArgv) [0x0001e] in <c66998abee2f4df39ea2647af887c7b2>:0

 

Where should I change python3.6  to python3.8 ?  or should I copy python3.8 as python3.6?

 

Thanks

Shailesh

0

Hi Shailesh Raval 

Please check out the docker file we use in QuantConnect Cloud to create the images we use to run Lean.

At the moment, due to a restriction in pythonnet (the interface between C# and Python), we can only have support one version of Python and there is 3.6. So you will need to uninstall and/or downgrade your system's Python version.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hi Alexandre,

I can not use docker file because my host server (where my VPS is hosted) is using some old 2.* verson kernel and Docker needs newer 3+ verson kernel.  I can not uprade my VPS kernel..

I reinstalled python 3.6 after uninstalling/removing python 3.8.  Still it is same issue.  I have requested to migrate my vps to another server host which runs newer 3+ version kernel.  Will try docker solution then.

Thank you very much..

Shailesh.

0

I just wanted to add something really fast. I'm almost done finishing my setup on my Linux server. There's a few changes to the current directions on the github README file I'd like to suggest. Is this the best way to provide those?

0

Thanks, Clyde -- best would be to create a Github issue with the proposed solution. Suggest running it by the team at support@quantconnect.com if you'd like feedback before opening the issue.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Ok cool, I figured! Just wanted to double check!

0

If I understand correctly, looking at the launch script, it looks like we need to have built Lean locally already? That completely defeats the purpose of running it in Docker... for example, I develop mostly on a Mac... except, I have a minimal VS project with the dependencies loaded from nuget. I should just be able to create my own Dockerfile, sourcing from `quantconnect/lean:lean`, and then `COPY` in my algorithm and config files... building completely locally and having everything inside of Lean is way overkill.., This Docker setup makes no sense

0

Interesting feedback thanks Nabeel

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


A question about Local lean. For the Coarse and Fine Universe, is it means that user need to buy the data from Morning star, receive and handle those data by themselves? Since that part is not open source ?

Thank you very much.

0

Correct Sulfred, the Morning Star dataset for the fine universe selection is not within the reach of individuals to purchase so we did not open source the processing technology for that dataset. We're happy to share it but just ask the institutions interested in using the code reach out to us at support@quantconnect.com. 

Coarse universe selection you can assemble with almost any daily data source. 

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed