Hi --
I'm trying to find a working example of live trading with a custom data feed. I've found two examples using Bitcoin on Github. These seem to be the only examples I can find that employ custom data with live trading. Backtesting works as expected.
But when these examples are used in live trading, no trades are executed. No errors.
I've scoured this forum, and I haven't found any other examples using custom data with live trading. Are there any I missed?
And since neither example is working, I have to wonder: is the problem with the examples, or is custom data not working on live at all?
Thanks!
https://github.com/QuantConnect/Lean/blob/master/Algorithm.Python/CustomDataBitcoinAlgorithm.py
https://github.com/QuantConnect/Lean/blob/master/Algorithm.Python/CustomDataRegressionAlgorithm.py
# QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
# Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from clr import AddReference
AddReference("System")
AddReference("QuantConnect.Algorithm")
AddReference("QuantConnect.Common")
from System import *
from QuantConnect import *
from QuantConnect.Algorithm import *
from QuantConnect.Data import SubscriptionDataSource
from QuantConnect.Python import PythonData
from datetime import date, timedelta, datetime
import numpy as np
import json
### <summary>
### Demonstration of using an external custom datasource. LEAN Engine is incredibly flexible and allows you to define your own data source.
### This includes any data source which has a TIME and VALUE. These are the *only* requirements. To demonstrate this we're loading in "Bitcoin" data.
### </summary>
### <meta name="tag" content="using data" />
### <meta name="tag" content="custom data" />
### <meta name="tag" content="crypto" />
class CustomDataBitcoinAlgorithm(QCAlgorithm):
def Initialize(self):
self.SetStartDate(2011, 9, 13)
self.SetEndDate(datetime.now().date() - timedelta(1))
self.SetCash(100000)
# Define the symbol and "type" of our generic data:
self.AddData(Bitcoin, "BTC")
def OnData(self, data):
if not data.ContainsKey("BTC"): return
close = data["BTC"].Close
# If we don't have any weather "SHARES" -- invest"
if not self.Portfolio.Invested:
# Weather used as a tradable asset, like stocks, futures etc.
self.SetHoldings("BTC", 1)
self.Debug("Buying BTC 'Shares': BTC: {0}".format(close))
self.Debug("Time: {0} {1}".format(datetime.now(), close))
class Bitcoin(PythonData):
'''Custom Data Type: Bitcoin data from Quandl - http://www.quandl.com/help/api-for-bitcoin-data'''
def GetSource(self, config, date, isLiveMode):
if isLiveMode:
return SubscriptionDataSource("https://www.bitstamp.net/api/ticker/", SubscriptionTransportMedium.Rest);
#return "http://my-ftp-server.com/futures-data-" + date.ToString("Ymd") + ".zip";
# OR simply return a fixed small data file. Large files will slow down your backtest
return SubscriptionDataSource("https://www.quandl.com/api/v3/datasets/BCHARTS/BITSTAMPUSD.csv?order=asc", SubscriptionTransportMedium.RemoteFile);
def Reader(self, config, line, date, isLiveMode):
coin = Bitcoin()
coin.Symbol = config.Symbol
if isLiveMode:
# Example Line Format:
# {"high": "441.00", "last": "421.86", "timestamp": "1411606877", "bid": "421.96", "vwap": "428.58", "volume": "14120.40683975", "low": "418.83", "ask": "421.99"}
try:
liveBTC = json.loads(line)
# If value is zero, return None
value = liveBTC["last"]
if value == 0: return None
coin.Time = datetime.now()
coin.Value = value
coin["Open"] = float(liveBTC["open"])
coin["High"] = float(liveBTC["high"])
coin["Low"] = float(liveBTC["low"])
coin["Close"] = float(liveBTC["last"])
coin["Ask"] = float(liveBTC["ask"])
coin["Bid"] = float(liveBTC["bid"])
coin["VolumeBTC"] = float(liveBTC["volume"])
coin["WeightedPrice"] = float(liveBTC["vwap"])
return coin
except ValueError:
# Do nothing, possible error in json decoding
return None
# Example Line Format:
# Date Open High Low Close Volume (BTC) Volume (Currency) Weighted Price
# 2011-09-13 5.8 6.0 5.65 5.97 58.37138238, 346.0973893944 5.929230648356
if not (line.strip() and line[0].isdigit()): return None
try:
data = line.split(',')
# If value is zero, return None
value = data[4]
if value == 0: return None
coin.Time = datetime.strptime(data[0], "%Y-%m-%d")
coin.Value = value
coin["Open"] = float(data[1])
coin["High"] = float(data[2])
coin["Low"] = float(data[3])
coin["Close"] = float(data[4])
coin["VolumeBTC"] = float(data[5])
coin["VolumeUSD"] = float(data[6])
coin["WeightedPrice"] = float(data[7])
return coin;
except ValueError:
# Do nothing, possible error in json decoding
return None
Jared Broad
Hey Mark,
Generally speaking, live custom data is working well for many people - but these examples might be out of date. If you have a specific live source we can look into why it might not have triggered. We'll review the examples provided and see if they need updating.
The most common reason live data doesn't trigger is the data is not present on the schedule presented in the historic data. e.g. The previous days show "midnight" but the data isn't available until 3 days later in reality.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Mark L
Thanks Jared -- The source is one I'm generating myself, still a work in progress. Are there any other examples you're aware of aside from those two?
Re: schedule issues - for what it's worth, in the example CustomDataBitcoinAlgorithm.py, the timestamp from the custom data feed is ignored and instead the data is given a timestamp of 'now.' So I would think that's not the cause of the issue, in at least that example.
Mark L
Update: CustomDataBitcoinAlgorithm.py finally initiated a trade 4 hours after I launched it. I expected that algo to initiate a trade immediately, since it doesn't have a warmup set. Is there a default warmup period if one isn't set, or am I missing something else here?
Thanks!
Jared Broad
In this case its daily data; it'll generate a new bar when a new day arrives. In this custom bitcoin data, I recommend you use our live feeds which are pushed to your algorithm as part of our normal systems. Assuming you've got some other custom data you're actually working with please provide a realistic example for better assistance.
A warm-up helps you trade faster if your algorithm had indicators or other states that needed to be primed first.
Live assistance is only given via the support system with live logs attached for precisely this reason. We can spend days guessing what's wrong or we could have looked at a log and seen it was only deployed a few minutes.
https://www.youtube.com/watch?v=MIAJU6xWM0A&ab_channel=QuantConnectThe material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Mislav Sagovac
I have tried above in live trading and it doesn't work for me. There no errors, no logs. I doesn't do anything?
Derek Melchin
Hi Mislav,
The reason the data is not passed to OnData is because `datetime.now()` doesn't align with the algorithm time. To resolve the issue, we should use
We're working on updating the example in the GitHub repo to reflect this.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Mislav Sagovac
Thanks Derek Melchin I tried it in paper trading and it works with your modification (the algo bought the Bitcoin so probably it works).
It doesn't work for my custom data in live (paper) trading, so there is still a mystery for me what Time property should be equal to in live trading, but this is for other thread.
Derek Melchin
Hi Mislav,
Please refer to this related thread.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Mark L
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!