Back

Avoiding Vendor Lock-In: Running LEAN On Your Server

Hello Everyone! Given events lately I wanted to directly address fears of vendor lock-in from choosing to use QuantConnect/LEAN as your trading platform by giving you a step-by-step guide for how you could run LEAN on your own server. 

- Why would we do this? We believe you are using QuantConnect to save you time and effort setting up the environment on your own. With QuantConnect you can get better features, data and UX than possible alone. This motivates us to provide an outstanding service so that even if you can run it yourself; you'll likely choose to use us. 

 - Why did we make LEAN powerfully open? We designed LEAN to be runnable locally; and spent weeks working on datafeed connections only relevant to the open-source project. Why would we do this? Like Linux we see LEAN as the foundation for the future of algorithmic trading. There are millions of features to build and there's only so much we can do ourselves so by making open-source a core part of our project we encourage contributions to help make it a truly community driven company. Want to see the project thrive? Help contribute

So to get started! Its pretty simple:

1) Core Technology - LEAN:

  • You will need LEAN - our cross platform open-source project kernel of QuantConnect. It builds and runs locally with almost no effort. We ship small data samples with your project so you can see it working instantly. Windows, Linux or Max are supported. By default it is a CLI program and we provide a rudimentary GUI but community members have built beautiful GUI's for Windows.

2) Infrastructure:

  • For live trading you'll need a virtual machine of at least 512MB RAM. You can pick one reasonably stable micro-machines from AWS ($7/mo) or Google Cloud Compute ($5/mo). I wouldn't recommend super discount providers as the reliability of internet connection is important in live trading. For backtesting you can backtest locally in Visual Studio.
  • OS Setup: We provide a docker file so you can setup the OS instantly by following along with the docker setup - or running just running LEAN in the docker container we provide.

3) Data:

  • Backtesting:

    We use Zip-CSV format for the data as it is an open format easily reverse engineered.

    Equity / Futures / Options: This is the only tricky part. Due to exchange licenses we can't give this to you but we can make it as easy as possible to get started. We work with data vendors who offer very reasonable rates and we have created and open-sourced data converters to convert their data into LEAN format. These converters all live in the ToolBox project of LEAN. We also provide ways to download US Equity data from Google's API but this is limited to the last 2 weeks.

    Forex / CFD: We have negotiated on your behalf and FXCM / OANDA have agreed to make their data available for free download. We provide this for free through the Data Library.

    Coarse Universe: For backtesting coarse universe is available via the Coarse Universe converter in the toolbox. This is reasonably easy to use but still requires you to gather the entire universe of equity daily prices. Fine Universe (fundamentals) is harder as its provided by Morning Star. We chose Morning Star as they had the best coverage but its not a cheap provider. We haven't open-sourced this as we think given price of data its an institutional offering not available for an individual (hedge funds - contact us). Given Morning Star costs its probably a good idea to create a Quandl fundamental converter in the long run (WIP).
     
    For more information on downloading data check out: GoogleYahoo, FXCM, Dukascopy Data Downloaders.
    You can also see the QuantQuote, NSE, AlgoSeek Options and Futures Data Converters and the FactorFile Generator.
     
  • Live Trading: 

    Streaming Data: We've provided many mechanisms for live data streaming in the open-source project. These classes are called the "IDataQueueHandler" implementations. They broadly come from three sources:
       1) Brokerage implementations - stream live prices directly from your brokerage with your data subscriptions. We've implemented IB, OANDA and FXCM brokerage IDataQueueHandler's so you can stream directly from them with almost no effort. Each brokerage will be limited in the data it can stream (i.e. FXCM can't send you options pricing). 
       2) Data vendors - we've implemented and open-sourced an IQFeed data feed. as IQFeed supports all our security types. Other data vendors weren't reliable enough.
       3) IEX Exchange - an implementation to the free IEX exchange datafeed for US Equity prices.

    History Requests: History requests are tricky as its actually pretty complex engineering to store streaming prices and serve them up quickly. We have provided open-source implementations to the brokerages we support. Data vendors can't support this at the moment.

    Universe Data: Live trading with universe data is also a challenging task to open-source as we're using lots of proprietary data to generate the full listing of symbols and prices for the universe. The format is not proprietary though and you can provide your own universe in live trading by serving a collection of CoarseFundamental objects inside of a BaseDataCollection delivered by the same IDataQueueHandler used above. It is processed here. We'll ponder on this one to see how to open-source it for personal use with publicly available data sources.

We're grateful for all of the community and will continue working to give you the best algorithmic trading platform in the world so you chose to spend your time and energy with us! 

 

PS: If you're like to contribute - there is lots to do =)

  • #1 Helping community members and sharing algorithms. Helping the community thrive is a huge help to the core team.
  • #2 Grab a bug or feature from the issues list and submit a PR. Finding and submitting descriptive bugs is a big help too.
  • #3 Contributing to our new open-sourced tutorials which are deployed here.
  • #4 Submitting new example algorithms for C#, F# and Python.
  • #5 Lastly if you're too busy you can subscribe to support the project financially.
Update Backtest






The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.



Jared,

Like probably many other once-burned twice-shy Quantopian users, I would like to have a backup server running the open source version of QC "just in case". In all likelikhood I will rarely use it, since - as you correctly point out - it will be very laborious to keep it up to date with all the public data sources, etc.

I would describe myself at at level 4 on a scale of 10 in managing a linux server. What I don't have in expertise I make up for in sheer persistence and grit. I would like to host your Quantconnect Lean solution on Google Cloud Services, and have just signed up for an account. I was previously hosting  with a virtual server at ixwebhosting. But at $75 a month I realized it would be cheaper to go with Google, and probably more scalable. I quickly was able to set up a LAMP configuration and get the basics going on a Wordpress file, so I'm comfortable I'll be able to port my entire web hosting to Google. So now on to Quantconnect...

If my questions seem a bit ignorant, bear with me.  I'm sure among the Quantopian users there are a few thousand with similar concerns/ questions. I'm not familiar with Docker, and spent a good hour reading up on it and hoped to be able to  "launch" QC install with one command. This may be possible but I failed miserably. The best I achieved was to create a cluster with my project_id, create a directory to hold the QC files, and upload these to that directory using the command:

git clone https://github.com/QuantConnect/Lean

From there I played around a bit, but did not get far.

Does any documentation exist on setting up QC with AWS or Google Cloud? I searched through all the documentation on github and quantconnect but could find no other descriptionis of this process.

I would like to run QC using the python libraries.

Thanks

0

Hi Serge d'Adesky - the documentation for installing and running lean is ReadMe.md -- on the front page of the github repo.  

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Get it while it's hot? :)

ReadME.md

1

Thanks Jared. I had already looked in there. So far, this is what I have found:

Google Cloud does not seem to support Mono.  At least in googling this I cannot find any suggestion that they do.  I am mentioning that in case others are thinking Google Cloud offers a reliable solution. 

I then went to my normal isp (ixwebhosting) where I have a virtual dedicated server running Centos 6.7.  There the instructions given on the Mono site for CentOS also fail. I think it has to do with the proxy settings in my /etc/yum.conf files,preventing EPEL from working properly. I am taking that up with hosting support.

Is it simpler in Amazon's AWS?

0

All cloud hosts support Mono, but you need to install it: http://www.mono-project.com/docs/getting-started/install

This is done for you in the docker file. I would recommend you learn how to use docker instead of installing each component piecemeal - 

https://docs.docker.com/get-started/
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Jared, 

Thanks for your response. Unfortunately it is not quite so simple with Docker if you are using CentOs 

At my centos provider : I was not able to load Nuget. At first I had problems installing Mono itself,, which does not work "out of the box" with the instructions provided on your link. But I did get a successful install after removing duplicates. .At least it gave me a "success" message with no warnings or fatal errors about missing dependencies. However, when I tried to test it on the hello.cs file, it bombed. 

root@cloud [~]# mcs hello.cs
hello.cs(7,9): error CS0103: The name `Debug' does not exist in the current context
Compilation failed: 1 error(s), 0 warnings
root@cloud [~]# cat hello.cs


public class HelloWorld
{
    static public void Main ()
    {
        Debug.WriteLine ("Hello Mono World");
    }
}
root@cloud [~]#
 

 

I spent a lot of time learnings Docker . But once I did I understood that it would install a file repository that was guaranteed to have all necessary file dependencies, and automate the process of installing these: However, when I look at the Docker code syntax itself, I see there is no provision for some of the Linux flavors, namely CentOS

These commands will not work:

# From https://github.com/mono/docker/blob/master/4.6.1.5/

RUN apt-get update && rm -rf /var/lib/apt/lists/*

RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF

RUN echo "deb http://download.mono-project.com/repo/debian wheezy/snapshots/4.6.1.5 main" > /etc/apt/sources.list.d/mono-xamarin.list \

&& apt-get update \

&& apt-get install -y binutils mono-complete ca-certificates-mono mono-vbnc nuget referenceassemblies-pcl \

&& apt-get install -y fsharp && rm -rf /var/lib/apt/lists/* /tmp/*

for the simple reason that the apt-get methods do not work in Centos. 

Consequently, one is forced to go back to installing the various components piece by piece. 

Next I will try to run in from Docker on the Google Cloud. Maybe I will have more luck.

If I succeed in installing it on Google Cloud I will publish the steps I took in a file so others can benefit.

Thanks

Serge

0

Probably a stupid question but, is there a guide on how to backtest/debug MY algorithm (python) on the desktop environment?  

I ended up putting my files in the lean folder and changing the config.json to look at my algorithm but is that the right way to do it? Seems kinda messy. I also got a LOT of depency errors and figured out that I had to import some extra stuff that I don't know why but needs to be imported at the start of the algorithm.

from clr import AddReference
AddReference("System")
AddReference("QuantConnect.Algorithm")
AddReference("QuantConnect.Common")

from System import *
from QuantConnect import *
from QuantConnect.Algorithm import *

 

Haven't seen that in any tutorial, just something I picked up from other people's algorithms and figured I might need.

 

Also, Am I correct in the assumption that we can't backtest any algorithms with the full data that quantopian provides on it's web platform but only on data that we can download ourselves? I guess what I'm really asking is if I can use Visual Basic to better debug my code as I import it from Quantopian (LOTS OF ERRORS) and then use Quantconnect to backtest it. I don't care about backtesting on the desktop environment, I just don't know any other way to execute the code in a way that I can debug it.

1

Serge d'Adeskytry this one - Installing docker into Centos. With docker you shouldn't need to install anything (other than docker). Docker pulls down prebuilt images we've made & setup and runs the code in a virtual OS.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hi Jared,



Thanks. I finally decided to multiply my efforts on Google Cloud. Got

everything working up to the point of actually launching the container

using docker.



mono/python/pandas/docker/nuget all successfully installed and tested.



Now how do I get docker to run code that launches the virtual machine. I

suspect it has to somehow

access the Dockerfile, but that's still not clear to me.



Thanks

Serge
0

Hi Jarod,



Made more headway, and I'm almost at the finish line. Man - what a crash

course in Linux development! I figured out how to launch your dockerfiles

in docker:



Fitst attempt runs following : (sudo needed otherwise read/write problem

encountered)

*******************

sudo docker build github.com/QuantConnect/Lean

*******************



goes all the way to end successfully then at end produces error:

*******************

Step 1 : FROM quantconnect/lean:foundation

foundation: Pulling from quantconnect/lean

22ecafbbcc4a: Pull complete

580435e0a086: Pull complete

8321ffd10031: Pull complete

08b8f28a13c2: Pull complete

2b401702069a: Pull complete

4f4fb700ef54: Pull complete

eae027dcdc0e: Pull complete

93bc98227159: Pull complete

270e8fe04b55: Pull complete

4f443d1f4cf2: Pull complete

a0468abb69ce: Pull complete

a2426ffde66b: Pull complete

acf812c9476c: Pull complete

4e5d78ed336f: Pull complete

fcd185e9b8c5: Pull complete

8d5e8c8dee77: Pull complete

a42c9eab6989: Pull complete

1cd1430f2011: Pull complete

43b2c3f94d8e: Pull complete

Digest:

sha256:ef1cfe3b5dc080488129a56b674f7d676a857dfddf4b7956ea88748bcc73d22e

Status: Downloaded newer image for quantconnect/lean:foundation
0

Hi Jared,



I don't know if you saw my last email. I'm kind of of stuck trying to

implement your solution. The docker file gives me errors, which I

documented on the file. I understand the kind of time pressure you may be

under and don't want to contribute to that But at the same time I think

there may be thousands of other former Quantopian users who are in the same

boat as I am and have even less knowledge and exprerience with Linux

servers, Python and C++ coding, etc.



If you are overwhelmed by this task, perhaps I can be of assistance? Let's

talk. I think there is a tremendous opportunity for you to capture a large

part of the 90000 plus users of Quantopian, but they will need a lot more

hand holding than you are currently providing. I have some ideas about that

that you may like that I would like to bounce off of you.



Can you give me a quick call. Shouldn't be more than 10-15 minutes of your

time.

Thanks

Serge d'Adesky

0

I'm far from a Linux expert, but I've found it's possible to manually provision, configure, build and deploy a Lean algorithm and a remote host in about 15 minutes. You could try following the install instructions on the mono homepage:

http://www.mono-project.com/download/#download-lin

I then also have a few tools to synchronize binaries and logs etc.

Docker is very well respected for microservices and continuous deploy but it's overkill for what I'm doing. I'm afraid my Linux knowledge doesn't stretch very far, but if you want to try a manual route I will try to help.

I should also say that host pricing means you won't make a significant cost saving (if any): I have to self-host Lean only because I run several modified builds.

1

Hi Jared, I appreciate your efforts to avoid vendor lock-in, however in this thread, https://groups.google.com/forum/#!topic/lean-engine/xeG-ZJMvx-4 , it was noted that Lean requires internet access and needs to connect to QuantConnect servers.  You advised that "We use the same charting HTML for the backend and frontend to save double-work. The charting packages are streamed to a web socket for display.".  

While I understand this might save you some double-work, does this not create a reliance on QuantConnect and creates a bit of a vendor lock-in?  I personally would like to be able to run Lean without having an internet connection or relying on QuantConnect's servers.  I ask that you consider adding this flexibility.

0

Thanks for the feedback Richard. Using LEAN does eliminate lock in but doesn't eliminate the need for you to do work =), if you'd like a custom display please feel free to use the open source one supplied or build your own. We don't charge for information; and break even on infrastructure; but we currently do charge for convience and saved effort/time. 

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hi,

I had a Raspberry sitting around without use, thus I decided to try Lean with it and I can say it works great. My algo has been running for a few hours right now and it uses between 1 and 1.7 CPUs and about 360MB of RAM, and a naked algo uses .5 cpus and 250MB of RAM. 

I wont be running it with a live account, since In my little experience with Raspberry, I dont find it to be reliable. Nevertheless, it seems a nice environment to run new versions of my algo with practice accounts, or scanning the market for opportunities.

1

Erik Bengtson awesome!! Please attach a photo =D

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Richard, there are lots of options for drawing graphs from rollling data so you shouldn't have to use the QC client if you prefer not to. The QC engine already outputs data with all your trades and you can add extra log files with supplementary data. You will just need to set the environment to "backtesting" and you should be able to run offline.

Of course you can always go the caveman route like me and just dump your data into log.txt.

0

Here is the beast. I've taped it to avoid the power cable to disconnect

configuration: raspberry pi 3, samsung sd 16gb, Lean head version

quick setup:

  • download the raspian lite image from raspberry site, and follow instructions (unmount sd card and copy image to sd card,...)
  • now mount the sd card and create an empty file called ssh on the root folder
  • now start raspberry with the sd card
  • ssh to raspberry, and run
    • sudo apt-get update
    • sudo apt-get upgrade
    • sudo apt-get install mono-complete
    • copy compiled algorithm files to raspberry and run it with nohup
    • nohup mono algo.exe > nohup.out &
1

Hi.

Just browsed this forum.

Perhaps an aws image with everything already installed and with an update from repository script on build would make things easier for a lot of people.  Or even a vagrant image for local testing.  First day on Quantconnect - perhaps I'll try to contribute if this sort of thing does not already exist.

1

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed