Not able to run local backtest (message "please make sure docker is installed and running" remains)

Hi Frederik,

Can you please try running docker run hello-world and see if the output looks like this?

If it does, can you please use the same terminal session to try and run a backtest with the --verbose flag (so lean backtest --verbose "My Project") and share the full output if it still gives an error?

Hi Jasper,

hello-world gives me the following error message:

docker: Error response from daemon: Get https://registry-1.docker.io/v2/: EOF.

Thanks for your help!

After disconnecting from VPN (my bad...) and restarting docker the Hello-world prompt worked.

This is the result from backtest with the verbose flag:

Traceback (most recent call last):
  File "c:\users\sf4564\appdata\local\continuum\anaconda3\lib\site-packages\docker\api\client.py", line 159, in
__init__
    self._custom_adapter = NpipeHTTPAdapter(
NameError: name 'NpipeHTTPAdapter' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\sf4564\appdata\local\continuum\anaconda3\lib\site-packages\lean\components\docker\docker_manager.py",
line 203, in _get_docker_client
    docker_client = docker.from_env()
  File "c:\users\sf4564\appdata\local\continuum\anaconda3\lib\site-packages\docker\client.py", line 101, in from_env
    **kwargs_from_env(**kwargs)
  File "c:\users\sf4564\appdata\local\continuum\anaconda3\lib\site-packages\docker\client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "c:\users\sf4564\appdata\local\continuum\anaconda3\lib\site-packages\docker\api\client.py", line 165, in
__init__
    'Install pypiwin32 package to enable npipe:// support'
docker.errors.DockerException: Install pypiwin32 package to enable npipe:// support

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\sf4564\appdata\local\continuum\anaconda3\lib\site-packages\lean\main.py", line 30, in main
    lean.main(standalone_mode=False)
  File "c:\users\sf4564\appdata\local\continuum\anaconda3\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\sf4564\appdata\local\continuum\anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\sf4564\appdata\local\continuum\anaconda3\lib\site-packages\lean\click.py", line 80, in invoke
    result = super().invoke(ctx)
  File "c:\users\sf4564\appdata\local\continuum\anaconda3\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\sf4564\appdata\local\continuum\anaconda3\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\sf4564\appdata\local\continuum\anaconda3\lib\site-packages\lean\commands\backtest.py", line 197, in
backtest
    if update or not docker_manager.supports_dotnet_5(engine_image):
  File "c:\users\sf4564\appdata\local\continuum\anaconda3\lib\site-packages\lean\components\docker\docker_manager.py",
line 187, in supports_dotnet_5
    for img in self._get_docker_client().images.list():
  File "c:\users\sf4564\appdata\local\continuum\anaconda3\lib\site-packages\lean\components\docker\docker_manager.py",
line 205, in _get_docker_client
    raise error
lean.models.errors.MoreInfoError: Please make sure Docker is installed and running
Error: Please make sure Docker is installed and running
Visit https://www.quantconnect.com/docs/v2/lean-cli/user-guides/troubleshooting#02-Common-errors for more information

Hi Frederik,

Can you try running pip uninstall pywin32 && pip install pywin32==227 and then re-run the backtest? That package seems to be the issue here, even though it should be installed automatically maybe that didn't work correctly for some reason.

Hi Jasper,

thanks for your suggestion.

that package indeed seems to be the issue but re-installing version 227 did not solve the issue. I read here:

that pypiwin32==224 has the issue (I guess they are actually referring to pywin32 as it seems there is no higher version of pypiwin32 than 223). Anyway I tried with an older version of  pywin32 - namely 223. After pip install pywin32==223 I got the message "docker 4.4.4 has requirement pywin32==227; sys_platform == "win32", but you'll have pywin32 223 which is incompatible." but strangely enough 'lean backtest' worked...at least it ran and generated results - not 100% sure if correct.

After that I upgraded lean from 0.1.51 to 0.1.52. The upgrade found in the installation of pywin32 223 which it removed and again installed 227 instead. With the result that I get my "Please make sure Docker is installed and running" back.

So for now the strange workaround for me seems to be using an actually incompatible older version of pywin32 

Hi Frederik,

Good to know that you found a fix, but I agree that it's a strange workaround. I would love to find a more permanent fix that doesn't require you to re-install pywin32 223 every time, but I cannot reproduce the issue locally so I cannot debug it. Can you please share your Python version (python --version), your Windows version (Start > Settings > About > Windows specifications), and your Docker version (docker --version)?

Hi Jasper,

sure!

python: 3.7.4

windows: Windows 10 Enterpreise 1909

docker: 20.10.6 build 370c289

Hey did you fix this? 

Try this

Please make sure Docker is installed and running by Rushi Chaudhari - QuantConnect.com

Hi Rushi,

no I didn't fix it other than the workaround of installing an old version of pywin32.

I don't think a virtual enviroment is the cause for me - I did install lean from a normal Anaconda prompt.

Hi Frederik,

Although I haven't been able to reproduce this with Python 3.7, I was able to reproduce the same error using 3.8 and 3.9. I just released an update that fixes it for those two versions, can you please try updating the CLI (pip install --upgrade lean) and check whether you still get the error with pywin32 version 227 on Python 3.7?

HI Jasper,

still have the error on Python 3.7. Btw: I use anaconda - don't know if that's an issue.

In then tried to update my Python version to 3.8 to see if your fix is working with that version - unfortunately upgrading the base env from 3.7 to 3.8 (conda install -c anaconda python=3.8) did not work because of conflicts.

I proceeded to create a virtual env with python 3.8 (conda create -n python38 python=3.8) and activated it (conda activate python38). Still the Docker issue. Updated lean again - uninstalled and installed pywin32==227 again - still “make sure Docker is installed and running” - with the underlying error still being NameError: name 'NpipeHTTPAdapter' is not defined → docker.errors.DockerException: Install pypiwin32 package to enable npipe:// support.

Downgrading pywin32 below 227 ignoring the dependency error (docker 4.4.4 requires pywin32==227) strangely still works.

Thanks for your time you spent on checking this issue!

Hi Frederik,

Anaconda should not be the issue, I'm testing it with Miniconda myself (which is a lightweight version of Anaconda).

I just released a new version with an improved version of the bug fix, can you please try updating again (same command: pip install --upgrade lean) and see if it works?

Hi Jasper,

that actually did it! Now local backtesting as well as a local research enviroment using docker is working with python 3.7 and pywin32-227

Thank you for keeping at it and finally fixing it 😊