Back

Not possible to build Lean:algorithm docker container

Hi All,

Can you please help.

I tried to build Lean from docker without success.

I managed to build Lean:foundation but when I try to build Lean:Algorithm I'm getting error below:

 

(base) d@dpc:~/Lean$ docker build -t quantconnect/lean:algorithm -f Dockerfile .
Sending build context to Docker daemon    564MB
Step 1/5 : FROM quantconnect/lean:foundation
 ---> 7f631430dc6a
Step 2/5 : MAINTAINER QuantConnect <contact@quantconnect.com>
 ---> Using cache
 ---> a511adb4a824
Step 3/5 : COPY ./Launcher/bin/Release /root/Lean/Launcher/bin/Release
COPY failed: stat /var/lib/docker/tmp/docker-builder415840919/Launcher/bin/Release: no such file or directory

 

Update Backtest







0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hey Don Q,

Have tried running Lean for both C# and Python? It may be possible that some files are missing because Lean was not compiled. (refer to this thread)

Please also change

COPY ./Launcher/bin/Release /root/Lean/Launcher/bin/Release

to

COPY ./Launcher/bin/Debug /root/Lean/Launcher/bin/Debug

Let us know if that works or if there are any further issues.

Best
Rahul

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hi Rahul,

What do you mean by "running Lean for both C# and Python?"

I couldn't find any tutorlial or just simple file with steps to follow.

I followed this steps on ubuntu 18.04

1) Installed docker-ce

2) Cloned Lean and build lean:foundation package which os OK. When I cloned lean there is no folder 

./Launcher/bin

git clone https://github.com/QuantConnect/Lean.git
cd Lean
docker build -t quantconnect/lean:foundation -f DockerfileLeanFoundation .
docker build -t quantconnect/lean:algorithm -f Dockerfile .

Error Message:
(base) d@dpc:~/Lean$ docker build -t quantconnect/lean:algorithm -f Dockerfile .
Sending build context to Docker daemon 564MB
Step 1/5 : FROM quantconnect/lean:foundation
---> 7f631430dc6a
Step 2/5 : MAINTAINER QuantConnect <contact@quantconnect.com>
---> Using cache
---> a511adb4a824
Step 3/5 : COPY ./Launcher/bin/Release /root/Lean/Launcher/bin/Release
COPY failed: stat /var/lib/docker/tmp/docker-builder415840919/Launcher/bin/Release: no such file or directory
0

https://www.quantconnect.com/forum/discussion/2400/avoiding-vendor-lock-in-running-lean-on-your-server

 

When I red this artickel I understood you can run lean directly from container so no compilation is needed "OS Setup: We provide a docker file so you can setup the OS instantly by following along with the docker setup - or running just running LEAN in the docker container we provide."

0

Hey Don Q,

Lean still needs to be compiled to be able to run. The dockerfile Lean foundation sets up the enviroment on which Lean runs. We still need copy a compiled version of Lean into that enviroment. This is the Dockerfile and this is the DockerfileLeanFoundation.

Best
Rahul

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


The link in the Dockerfile above doesn't work. I have my own rudimentary container running. I'm using C#. Compiling locally and then mounting/copying running inside of Docker defeats the purpose of using Docker in the first place... I need to look at the command line arguments to be able to pass in the config file instead of it getting build in, but for now... if you create a directory with your algorithm file, a copy of the `QuantConnect.Algorithm.CSharp.csproj`, renamed to "proj.csproj", and your file with your algorithm file in the "includes" section, and then the config.json file, and then use this for the Dockerfile.

For example, my algo test file is called "NabeelTestAlgo.cs", open the proj.csproj file, and add this where the other `<Compile Include="..." />` files are:

<Compile Include="NabeelTestAlgo.cs" />

 

Then the Dockerfile:

FROM mono:6.8.0.96

WORKDIR /opt/Lean

RUN apt-get update && apt-get -y install git nuget
RUN git clone https://github.com/QuantConnect/Lean.git /opt/Lean
RUN nuget restore QuantConnect.Lean.sln

COPY config.json Launcher/config.json

COPY proj.csproj Algorithm.CSharp/QuantConnect.Algorithm.CSharp.csproj
COPY *.cs Algorithm.CSharp/

RUN msbuild QuantConnect.Lean.sln "/p:RunTest=false"

CMD ["mono", "Launcher/bin/Debug/QuantConnect.Lean.Launcher.exe"]

Then build and run:

docker build -t quantconnect:latest .
docker run quantconnect:latest

This takes advantage of Dockers stage caching, and the csproj editing is required of the dumb way includes are treated by the csproj files. If QuantConnect fixes the build so it can use the .NET Core format, then this hack of having to create a copy of the csproj file will work.

If you use docker-compose, you can mount the Data directories, etc, so that it won't store them in the container.

Right now, I have a solution (I use Rider) which just pulls in the QC libraries from nuget, instead of the full Lean build... which is a pain in the butt to rebase, etc when there are changes. This container is also much more lightweight than the one supplied.

0

Here, this works, runs a backtest:

 

FROM mono:6.8.0.96

WORKDIR /opt/Lean

RUN apt-get update && apt-get -y install git nuget
RUN git clone https://github.com/QuantConnect/Lean.git /opt/Lean
RUN nuget restore QuantConnect.Lean.sln

COPY config.json Launcher/

COPY proj.csproj Algorithm.CSharp/QuantConnect.Algorithm.CSharp.csproj
COPY *.cs Algorithm.CSharp/

RUN msbuild QuantConnect.Lean.sln /p:RunTest=false

WORKDIR /opt/Lean/Launcher/bin/Debug/

CMD ["mono", "QuantConnect.Lean.Launcher.exe", "--data-folder", "/opt/Lean/Data", "--config", "/opt/Lean/Launcher/config.json"]

 

When running the container, you should mount the data and results folders. I use docker-compose, in which I create a local volume and then Docker mounts it.

If there's interest, I can create a repo with a full example and README. This is really lightweight and you don't have to compile anything locally or even have the Lean repo, and should be completely repeatable. 

0

Created a repo with an example:

https://github.com/nabeelio/QuantConnectDockerExample

 

0

Great idea thanks Nabeel, we'll update the dockerfile image to include the build of LEAN. It is currently focused on providing the right runtime environment to facilitate local python debugging. 

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thanks, Jared. I posted a detailed comment on the GitHub issue. It's a little more complicated than that if you have custom source files in C# (might be OK for Python), but I have a request which (I think) should be feasible to enable Docker builds and not have to mess with the csproj file.

0

Hi Nabeel, I also posted a detailed issue there =) I'll see what we can do this week. 

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Got it, I posted to here: 

https://github.com/QuantConnect/Lean/issues/4176#issuecomment-629861382

 

0

Hi Nabeel ,

How does your containerized algorithm receive market data?  I'm assuming your container no longer gets market data feed from QuantConnect.  Do you have to have a separate market data subscription from Interactive Brokers for example?

Thanks!

0

Hi Jon Quant ,

If you select the "live-interactive" environment:

# config.json
{
"environment": "live-interactive",
// other keys
}

the containerized algorithm should receive market data from Interactive Brokers.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed