Back

Running a local research environment?

Is it possible to run the Jupyter research environment locally and use data from QuantConnect as you can do with algorithm backtesting? I haven't been able to find anything in the forums. I cloned Lean from github and got python algorithms to work after some fiddling, but I'm a bit stumped on where to start with the QuantConnect.Jupyter project. 

I'd like to do this because a) I want to do some research on options and the research option API is not that great so I thought I'd modify it a bit, and b) my notebook keeps disconnecting randomly in the QC environment.

Update Backtest








Ok, so I made a little bit of progress with this. I installed Jupyter and pythonnet locally, then used the QuantBook template from the online environment as inspiration. I can load the .net libraries, but when I try to create a new instance of QuantBook, I get the following error:

QuantBook.Main(): System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Data\market-hours\market-hours-database.json'.

It would appear that some data files are necessary for this to run. Where can I find them?

%matplotlib inline
import sys
sys.path.append(r"E:\programmieren\LEAN\Lean\Jupyter\bin\Debug")

from clr import AddReference
AddReference("QuantConnect.Common")
AddReference("QuantConnect.Jupyter")
AddReference("QuantConnect.Indicators")
from System import *
from QuantConnect import *
from QuantConnect.Data.Market import TradeBar, QuoteBar
from QuantConnect.Jupyter import *
from QuantConnect.Indicators import *
from datetime import datetime, timedelta
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from QuantConnect.Data.Fundamental import IncomeStatement

# Create an instance
qb = QuantBook()

1

https://github.com/QuantConnect/Lean/tree/master/Data/market-hours

direct link to the file

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Ok, thanks, so it's in the repository. I actually made some more progress by pointing the config.json file in the QuantConnect.Jupyter project to the correct location like this:

"data-folder": "E:\\programmieren\\LEAN\\Lean\\Data",
"plugin-directory": "E:\\programmieren\\LEAN\\Lean\\Jupyter\\bin\\Debug",

I also modified the notebook to change the working directory to the location where the Jupyter project is builiding

%matplotlib inline
import os
os.chdir(r"E:\programmieren\LEAN\Lean\Jupyter\bin\Debug")

from clr import AddReference
AddReference("QuantConnect.Common")
AddReference("QuantConnect.Jupyter")
AddReference("QuantConnect.Indicators")
from System import *
from QuantConnect import *
from QuantConnect.Data.Market import TradeBar, QuoteBar
from QuantConnect.Jupyter import *
from QuantConnect.Indicators import *
from datetime import datetime, timedelta
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from QuantConnect.Data.Fundamental import IncomeStatement

# Create an instance
qb = QuantBook()

Now I can at least use the classes. Unfortunately, when trying to get data, I only get an empty dataframe

spy = qb.AddEquity("SPY")
h1 = qb.History(360, Nullable[Resolution](Resolution.Daily))

Can this be hooked up to the QC data through the API somehow or is this as far as I'll get? I tried to supply the same handlers as in the backtest configuration, but I couldn't get it to do anything. I'll see if I can set up a local project to debug it from C# instead of using the notebook.

By the way: how did you manage to get pythonnet to resolve the Resolution? function arguments correctly? I was only able to do it with an explicit cast.

1

As it turns out, I misunderstood how the backtesting works. It's loading data from the file system, so you actually need to download this from somewhere. I set up a debug project so I could get command line output from the QuantBook object and it complained about some missing files in the data folder, which, apart from some example data, is empty.

So for a local research environment, you'll need to get the data from somewhere else. Maybe someone from QC would be nice enough to provide assistance with how to get the file format right as the data seems to be stored as csvs in zip files.

 

As a proof of concept, I proceeded to download some forex data from FXCM via the QC data page. I can load it from my test program like this

using QuantConnect.Jupyter;

namespace JupyterDebug
{
class Program
{
static void Main(string[] args)
{
QuantBook qb = new QuantBook();
qb.AddForex("EURUSD");
var test = qb.History(180, QuantConnect.Resolution.Minute);
Console.Out.WriteLine(test.Repr());
Console.Out.WriteLine(test.Length().ToString());
Console.ReadKey();
}

}
}

Which prints out some data. Unfortunately, trying to replicate it in the notebook, I get the following error:

int() argument must be a string or a number, not 'String[]'

trying to call this:

h1 = qb.History(30, Nullable[Resolution](Resolution.Minute))

Playing around a little with error messages (I haven't been able to find out how to debug when triggering code from the notebook, attaching to python.exe didn't work) I found that the error apparently happens in the function 

public PyObject GetDataFrame(IEnumerable<Slice> data)
{
var maxLevels = 0;
var sliceDataDict = new Dictionary<Symbol, PandasData>();

foreach (var slice in data)
{
foreach (var baseData in slice.Values)
{
PandasData value;
if (!sliceDataDict.TryGetValue(baseData.Symbol, out value))
{
sliceDataDict.Add(baseData.Symbol, value = new PandasData(baseData));
maxLevels = Math.Max(maxLevels, value.Levels);
}
value.Add(baseData);
}
}
using (Py.GIL())
{
if (sliceDataDict.Count == 0)
{
return _pandas.DataFrame();
}
var dataFrames = sliceDataDict.Select(x => x.Value.ToPandasDataFrame(_pandas, maxLevels));
return _pandas.concat(dataFrames.ToArray());
}
}

more specifically on the last line in the return statement in the ToArray function.

 

Since my original goal of messing with options is probably not gonna happen in the local environment for me as I don't have enough cash to burn for an adequate data provider, I will probably leave it at that, but I'd still be interested in getting at least the proof-of-concept to work just for kicks.

1

Stefan Reutter"how did you manage to get pythonnet to resolve the Resolution?"
You need to install QuantConnect's version of pythonnet.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Alexandre Catarino, thank you. I see. Do you expect that this would also fix the conversion error? I'll give it a whirl when I have some spare time later this week.

1

Hi Stefan Reutter, thank you for putting this up! I'm getting started with QC and cloned the Basic Template to run locally. I have a working python distribution with jupyter notebook installed. But I have never used pythonnet to import dll files into python.

After installing pythonnet and running the first cell, I got this error.

%matplotlib inline
# Imports
from clr import AddReference
AddReference("System")
AddReference("QuantConnect.Common")
AddReference("QuantConnect.Jupyter")
AddReference("QuantConnect.Indicators")

---------------------------------------------------------------------------
FileNotFoundException Traceback (most recent call last)
<ipython-input-1-917d30d6139a> in <module>()
3 from clr import AddReference
4 AddReference("System")
----> 5 AddReference("QuantConnect.Common")
6 AddReference("QuantConnect.Jupyter")
7 AddReference("QuantConnect.Indicators")

FileNotFoundException: Unable to find assembly 'QuantConnect.Common'.
at Python.Runtime.CLRModule.AddReference(String name)
Could you please tell me what I have done wrong and how to fix it?
 
 
0

By the way I found this python package quantconnect that said it was developed by QC python team. Can any body tell me what it is for and how to use it?

Thanks

0

So I was too eager to setup a local research environment and missed some details in the comments #8773 and #8774 above. In which, Stefan already pointed out that this is not an efficient way because the engine loads data from file system and one needs to manually download data for backtesting locally.

I was able to import the engine by following comment #8773 anyway. The exact steps are:

1. Changing plugin path in config.json, 

2. Rebuilding the QuantConnect.Jupyter project and 

3. Inserting the path into jupyter notebook using

import os

os.chdir("PATH TO YOUR JUPYTER DEBUG FOLDER")

.

0

Hi!

I followed the document and setup everything. library on each python side or c# side loaded. Works well.

%matplotlib inline
# Imports
from clr import AddReference
AddReference("System")
AddReference("QuantConnect.Common")
AddReference("QuantConnect.Jupyter")
AddReference("QuantConnect.Indicators")
from System import *
from QuantConnect import *
from QuantConnect.Data.Custom import *
from QuantConnect.Data.Market import TradeBar, QuoteBar
from QuantConnect.Jupyter import *
from QuantConnect.Indicators import *
from datetime import datetime, timedelta
import matplotlib.pyplot as plt
import pandas as pd

from QuantConnect.Data.Fundamental import IncomeStatement

But I got an error when I try to create a QuantBook instance.

# Create an instance
qb = QuantBook()

 

---------------------------------------------------------------------------
Exception Traceback (most recent call last)
<ipython-input-3-3f08bd7e189f> in <module>()
1 # Create an instance
----> 2 qb = QuantBook()

Exception: QuantBook.Main(): System.ArgumentException: Unable to locate any exports matching the requested typeName: QuantConnect.Lean.Engine.Results.BacktestingResultHandler
Parameter name: typeName
at QuantConnect.Util.Composer.GetExportedValueByTypeName[T](String typeName) in C:\Users\m\github\Lean\Common\Util\Composer.cs:line 136
at QuantConnect.Lean.Engine.LeanEngineAlgorithmHandlers.FromConfiguration(Composer composer) in C:\Users\m\github\Lean\Engine\LeanEngineAlgorithmHandlers.cs:line 171
at QuantConnect.Jupyter.QuantBook..ctor() in C:\Users\m\github\Lean\Jupyter\QuantBook.cs:line 69
at QuantConnect.Jupyter.QuantBook..ctor() in C:\Users\m\github\Lean\Jupyter\QuantBook.cs:line 81

Any Ideas? Is there any addtional parameters I need to give at config.json? 

I am using 

Windows 10

python 3.6

Thanks!

0

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed