I was using live paper trading using 1 L-Micro-Node and was trying to run multiple concurrent ML training thread either using Schedule.On call or threading.
It looks like Quantconnect does not run multiple concurrent training either from Schedule.On call or from Self.Train call. It always does the trainings seqeuantially.
And python threading to run concurrent training, without a luck - in that case runtime just hangs and then times out eventually.
What is the solution here? Do I need to get multiple L-Micro-Node.
Note: Back testing does not work at all. It always times out when running concurrent ML trainings.
Mia Alissi
Based on the information provided in the extracts, it seems that QuantConnect's LEAN engine does not support concurrent training of multiple machine learning models within a single algorithm. The
self.Train
method is designed to run sequentially, not concurrently. This is likely due to the fact that running multiple concurrent training processes would require significant computational resources and could potentially lead to issues with memory management and performance.In terms of running multiple algorithms concurrently, it appears that each algorithm requires its own server or node. This is mentioned in several of the forum discussions, such as this one where a user discusses running multiple live algorithms using paper trading. Each live algorithm requires its own live server or node, which comes at a cost.
If you're experiencing timeouts during backtesting when running concurrent ML trainings, it might be due to the computational intensity of training multiple ML models at the same time. As mentioned in this discussion, backtesting in QuantConnect is limited by the single thread speed, which could be a bottleneck when running multiple concurrent trainings.
In conclusion, if you want to run multiple concurrent ML trainings, you might need to consider running each training on a separate node or server. However, this would increase the cost. Alternatively, you could consider optimizing your ML training process to reduce the computational resources required, or explore other ways to achieve your goal within the constraints of the platform.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
B Das
In that case, how to pass information and data between two or more nodes/servers?
Mia Alissi
Based on the information provided in the extracts, there isn't a built-in feature in QuantConnect's LEAN engine to directly communicate or pass data between two or more nodes/servers. As mentioned in this forum discussion, theoretically, you could create a control server that manages the communication between two algorithms, but this would likely require significant custom development and is not a feature provided out of the box by QuantConnect.
In another forum discussion, a user asked about allowing an algorithm to communicate with external software. The response from QuantConnect staff was that you can download signals/parameters from an external source to seed a universe or start your algorithm, but not upload signals/state. This suggests that while it's possible to pull data into an algorithm from an external source, pushing data out from an algorithm to an external source (or another algorithm) is not directly supported.
In conclusion, while it's theoretically possible to set up a system to pass data between different nodes/servers running different algorithms, doing so would likely require significant custom development and is not a feature directly supported by QuantConnect. You might want to consider other ways to achieve your goal within the constraints of the platform.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
B Das
Can you please give more details about this ‘control server’? Like how to build it and how does it work?
B Das
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!