How do I set up cloud computing (Azure/AWS/etc.) to run LEAN? by Andre Stevens

Andre Stevens

| Reward Discussion

Hey everyone,

I made a post a couple days ago about speeding up backtests locally in LEAN, but now I'm wondering how I can get started using cloud computing to run even faster.

How can I set up my LEAN to run on say, Microsoft Azure or AWS? A quick google suggested to use Virtual Machines, as well as some other options, but I have no experience in this area and would like help setting up.

Additionally, how can I secure my data (encryption, etc.) before running it on the cloud?

Thanks!

Update Backtest

Andre Stevens

Accepted Answer

Update Backtest

Notebook

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Andre Stevens

367 ,

For anyone else interested, this seems to be how https://blogs.msdn.microsoft.com/visualstudio/2015/01/08/azure-virtual-machine-images-for-visual-studio/

though i'm still looking into it.

Still figuring out the additional encryption and security part of it though though.

JayJayD

First of all, QC is already cloud computing. Do you know that you can have a cloud-based live algorithm for free with the QC subscription? The only reason why you should use Lean with some cloud services is because you cannot run your algorithm in the QC live environment.

Said that, you have basically two options (you know the difference between IaaS and PaaS, I guess):

Deploy a VM, login in the VM, download the code, build and run.
If you use Azure, you can publish as WebJob simply double click over the Launcher and select the option. Of course, you have to set up a storage account with the data (you have the data, I guess).

Of course, then you have to maintain the service, update Lean periodically, pump new data, etc.

Finally, unless you really know why you need to use Lean outside QC, do it; if not simply subscribe to QC and be happy. Trust me, I have a Lean instance running in an Azure VM since a couple of months.

Zach Oakes

1.3k ,

Hey guys, I had a similar question -- though not about hosting the algorithm in AWS / Azure, but more so interested in importing data from S3 (like it's imported from Dropbox). Is this an option ?

I have some models in Sagemaker that make schedules predictions that could be imported and used in QC signals if possible. S3 import functionality would be a nice feature given the growth of tools like PySpark with AWS compute clusters and AutoML in Sagemaker.

Also would offer a way to import models from AWs docker objects, and an easy way to reuse existing ML models.

I'm tempted to just fool around with trying normal methods of S3 import, but figured I would ask.

Zach

Shile Wen

STAFF ,

Hi Zach,

Connecting to AWS S3 directly is not supported at the moment, however, you can have the S3 return data through a REST API call, or have it update a remote CSV (which not limited to DropBox, here I use Github, any remote file hosting should work).

Best,
Shile Wen

Zach Oakes

1.3k ,

Shile,

Thanks for the reply -- I'll check out the link. I have used dropbox, I suppose I could schedule predictions and simply add a step where it writes the data to dropbox. Curious to see how it works where you're reading from Github? Dropbox seems to have alot of downtime is my concern there -- not sure if it's a production level storage option.

Shile Wen

STAFF ,

Hi Zach,

To be able to read the data from Github, I uploaded a .csv file to a Repository (which I named FileStore), then navigated to the .csv file, then clicked "Raw" button get the raw file (example), then copied that URL and read it in using the built-in data reader. As for updating this file programmatically, there are various methods to update git files, such as pygit2 for Python.

Best,
Shile Wen

Zach Oakes

1.3k ,

That's very clever. Do you think the logic is interchangeable with my dropbox Parse() method? Here, I'll find it. I'm using the Dropbox library in python, it's not great, but simple enough. Just the downtime is my big concern -- I've seen it down like 4x in the past week -- crazy for a company solely built on storage uptime.

def Parse(self, url): 

        # Download file from url as string

        file = self.Download(url).split("\n")

        

        # # Remove formatting characters

        data = [x.replace("\r", "").replace(" ", "") for x in file]

        

        # # Split data by date and symbol

        split_data = [x.split(",") for x in data]

        

        # Dictionary to hold list of active symbols for each date, keyed by date

        #dates = [i for i in split_data[0]]









        parse = lambda d: datetime.strptime(d, '%Y%m%d').date()    #Issues here... pd.to_datetime()?

        symbolsByDate = {}

        #longsByDate = {}

        #shortsByDate = {}



        #signalBySymbol = {}

        self.LE_T = []

        #self.SE_T = []

        

        #Data Format: Date      Symbol1     Symbol2     Symbol3 ...

        # Parse data into dictionary -- LOOPS THROUGH ROWS essentially (by vector)

        for arr in split_data:

            #Add Symbols to universe!

            date = pd.to_datetime(arr[0]).date()           #Alternative?

            #date = parse(arr[0])                                        #Replaced this w. original from ex. 





            #self.Debug(f"{arr} -- {arr[1]}")                                    #Print Debug             

            symbols = [Symbol.Create(ticker, SecurityType.Equity, Market.USA) for ticker in arr[1:]]

        #...

Shile Wen

STAFF ,

Hi Zach,

As long as the file format stays the same between moving it from Dropbox to Github, if you change the URL string to the Github link everything should work. Just be sure it's the Raw link as the normal link won't work.

Best Regards,
Shile Wen

Zach Oakes

1.3k ,

Thanks!

Andre Stevens

Update Backtest

Notebook

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!