Share New Research 
I'm running lean locally in the docker container. I wanted to parallelize the indicator and alpha computation for different symbols using Python multiprocessing module. I'm getting a huge exception trace below.
Is multiprocessing module supported in pythonnet running in a docker container?
I reverted the code, but it was fairly obvious, something like this:
def updateSignals(symbol: Symbol, tradeBar:TradeBar):
"""Updates indicators."""
...
class MyAlgorithm(QCAlgorithm):
def OnData(self, data:Slice):
with multiprocessing.Pool(nodes=4) as pool:
symbols = [s for s in self.Securities.keys()]
updatedData = [data.Bars[s] for s in self.Securities.keys()]
asyncResults = pool.apply_async(updateSignals, (symbols, tradeBars,))
I'm fairly confident this code would work in cPython.
Stacktrace:
at Python.Runtime.PyObject.Invoke(PyTuple args, PyDict kw)
at Python.Runtime.PyObject.InvokeMethod(String name, PyTuple args, PyDict kw)
at Python.Runtime.PyObject.TryInvokeMember(InvokeMemberBinder binder, Object[] args, Object& result)
at CallSite.Target(Closure , CallSite , Object , QCAlgorithm , PythonSlice )
at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2)
at QuantConnect.Algorithm.Framework.Alphas.AlphaModelPythonWrapper.Update(QCAlgorithm algorithm, Slice data)+MoveNext() in
/LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Algorithm/Alphas/AlphaModelPythonWrapper.cs:line 83
at System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)
at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source)
at QuantConnect.Algorithm.QCAlgorithm.OnFrameworkData(Slice slice) in
/LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Algorithm/QCAlgorithm.Framework.cs:line 162
at QuantConnect.AlgorithmFactory.Python.Wrappers.AlgorithmPythonWrapper.OnFrameworkData(Slice slice) in
/LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/AlgorithmFactory/Python/Wrappers/AlgorithmPythonWrapper.cs:line 603
at QuantConnect.Lean.Engine.AlgorithmManager.Run(AlgorithmNodePacket job, IAlgorithm algorithm, ISynchronizer synchronizer,
ITransactionHandler transactions, IResultHandler results, IRealTimeHandler realtime, ILeanManager leanManager, IAlphaHandler
alphas, CancellationToken token) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Engine/AlgorithmManager.cs:line 617
20220722 19:58:44.085 ERROR:: Extensions.SetRuntimeError(): Extensions.SetRuntimeError(): RuntimeError at
1/1/2020 5:00:00 AM UTC. Context: OnData Python.Runtime.PythonException: RuntimeError : already started
File "/opt/miniconda3/lib/python3.6/site-packages/ptvsd/daemon.py", line 145, in start
raise RuntimeError('already started')
File "/opt/miniconda3/lib/python3.6/site-packages/ptvsd/daemon.py", line 110, in started
self.start()
File "/opt/miniconda3/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/opt/miniconda3/lib/python3.6/site-packages/ptvsd/daemon.py", line 158, in start_server
with self.started():
File "/opt/miniconda3/lib/python3.6/site-packages/ptvsd/_remote.py", line 50, in start_daemon
_, next_session = daemon.start_server(addr=(host, port))
File "/opt/miniconda3/lib/python3.6/site-packages/ptvsd/_remote.py", line 62, in <lambda>
start_client=(lambda daemon, h, port: start_daemon()),
File "/opt/miniconda3/lib/python3.6/site-packages/ptvsd/pydevd_hooks.py", line 136, in _start_client
return start_client(daemon, h, p)
File "/opt/miniconda3/lib/python3.6/site-packages/ptvsd/_vendored/pydevd/pydevd.py", line 1060, in connect
s = start_client(host, port)
File "/opt/miniconda3/lib/python3.6/site-packages/ptvsd/_vendored/pydevd/pydevd.py", line 2230, in _locked_settrace
debugger.connect(host, port) # Note: connect can raise error.
File "/opt/miniconda3/lib/python3.6/site-packages/ptvsd/_vendored/pydevd/pydevd.py", line 2179, in settrace
wait_for_ready_to_run,
File "/opt/miniconda3/lib/python3.6/site-packages/ptvsd/_vendored/pydevd/pydevd.py", line 2427, in settrace_forked
patch_multiprocessing=True,
File "/opt/miniconda3/lib/python3.6/site-packages/ptvsd/_vendored/pydevd/_pydev_bundle/pydev_monkey.py", line 50, in
_on_forked_process
pydevd.settrace_forked()
File "/opt/miniconda3/lib/python3.6/site-packages/ptvsd/_vendored/pydevd/_pydev_bundle/pydev_monkey.py", line 528, in new_fork
_on_forked_process()
File "/root/.local/lib/python3.6/site-packages/multiprocess/popen_fork.py", line 66, in _launch
self.pid = os.fork()
File "/root/.local/lib/python3.6/site-packages/multiprocess/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/root/.local/lib/python3.6/site-packages/multiprocess/context.py", line 277, in _Popen
return Popen(process_obj)
File "/root/.local/lib/python3.6/site-packages/multiprocess/process.py", line 105, in start
self._popen = self._Popen(self)
File "/root/.local/lib/python3.6/site-packages/multiprocess/pool.py", line 239, in _repopulate_pool
w.start()
File "/root/.local/lib/python3.6/site-packages/multiprocess/pool.py", line 174, in __init__
self._repopulate_pool()
File "/root/.local/lib/python3.6/site-packages/pathos/multiprocessing.py", line 123, in _serve
_pool = Pool(nodes)
File "/root/.local/lib/python3.6/site-packages/pathos/multiprocessing.py", line 111, in __init__
self._serve()
File "/Library/diyhf/lib_alpha/alpha_avalanche.py", line 87, in Update
pool = multiprocessing.ProcessPool(nodes=4)
===
at Python.Runtime.PyObject.Invoke(PyTuple args, PyDict kw)
at Python.Runtime.PyObject.InvokeMethod(String name, PyTuple args, PyDict kw)
at Python.Runtime.PyObject.TryInvokeMember(InvokeMemberBinder binder, Object[] args, Object& result)
at CallSite.Target(Closure , CallSite , Object , QCAlgorithm , PythonSlice )
at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2)
at QuantConnect.Algorithm.Framework.Alphas.AlphaModelPythonWrapper.Update(QCAlgorithm algorithm, Slice data)+MoveNext() in
/LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Algorithm/Alphas/AlphaModelPythonWrapper.cs:line 83
at System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)
at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source)
at QuantConnect.Algorithm.QCAlgorithm.OnFrameworkData(Slice slice) in
/LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Algorithm/QCAlgorithm.Framework.cs:line 162
at QuantConnect.AlgorithmFactory.Python.Wrappers.AlgorithmPythonWrapper.OnFrameworkData(Slice slice) in
/LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/AlgorithmFactory/Python/Wrappers/AlgorithmPythonWrapper.cs:line 603
at QuantConnect.Lean.Engine.AlgorithmManager.Run(AlgorithmNodePacket job, IAlgorithm algorithm, ISynchronizer synchronizer,
ITransactionHandler transactions, IResultHandler results, IRealTimeHandler realtime, ILeanManager leanManager, IAlphaHandler
alphas, CancellationToken token) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Engine/AlgorithmManager.cs:line 617 
Fred Painchaud
Hi Denis,
I've also spent a lot of time trying to make multiprocessing work in the Docker container and even in the Cloud. No success so far. I've also tried the Ray Python library without success (that one specifically complains about Docker). See https://www.quantconnect.com/forum/discussion/13663/fix-feature-python-library-ray/p1 and https://github.com/QuantConnect/Lean/issues/6338. Ray is far more superior than multiprocessing in many many many regards for algo-trading. It works very well outside of Docker.
Sorry this does not help you, except for telling you you are not the only one who tried without success. If you find a way to make it work in the LEAN docker image, please report back. What I did not do yet is change that docker image to play with the configuration, etc. There is certainly a way to change the docker image to make both multiprocessing and ray work. Except if there is a weird bug in Docker, I don't see any reason why multiprocessing and Ray should not work. I also don't see why it should not work re Python.NET. Well, of course, with all the proper synchronisation between processes where needed… Or else, I can see multiple reasons why the .NET CLR would find its Python runtime subsidiary acting very weird :).
Fred
Denis Gorodetskiy
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!