Back

IQFeed for live trading in LEAN

I'm thinking about using IQFeed for market data with IB using the " live-interactive-iqfeed " environment

Quick question. Do i have to start IQFeed prior to starting LEAN or will the LEAN engine start the IQFeed process?

Thanks

Update Backtest







When i start up LEAN with the live-interactive-iqfeed environment variable IB gateway launches and then the IQConnect process is launched. But the login fails in IQConnect. I know that the username and password is correct as i'm able to use the feed with the IQFeed sample applications. Any idea why it would fail when using LEAN?

0

I'm able to get a bit further by starting the IQConnect program before launching LEAN. If that is done, LEAN connects to the admin port but then there's another failure

20180807 14:04:19.313 Trace:: IQFeed.Connect(): Connecting to admin...
20180807 14:04:20.341 Trace:: Loading IQFeed futures symbol map file...
20180807 14:04:20.344 Trace:: Found up-to-date IQFeed symbol universe file in local cache. Loading it...
20180807 14:04:33.997 ERROR:: IQFeed.Connect(): Error Connecting to IQFeed: Object reference not set to an instance of an object.

0

Does anyone know what are the correct values for these fields are in the config for IQFeed? I believe that productName needs to be valid for an application to attach to IQConnect

    // iqfeed configuration
    "iqfeed-productName": "",
    "iqfeed-version": "1.0",

0

Does one need to be registered as a developer with IQFeed to get the product name?

0

Yes I think so, I remember in our testing we were registered. What object

was the null exception on?
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thanks Jared.

The null exception seems to be coming from the call to;

_symbolUniverse = new IQFeedDataQueueUniverseProvider()

0

Please post the full stack trace and error message

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


QuantConnect.ToolBox.IQFeed.IQFeedDataQueueUniverseProvider.LoadSymbols() in Lean\\ToolBox\\IQFeed\\IQFeedDataQueueUniverseProvider.cs:line 361
QuantConnect.ToolBox.IQFeed.IQFeedDataQueueUniverseProvider..ctor() in Lean\\ToolBox\\IQFeed\\IQFeedDataQueueUniverseProvider.cs:line 90
QuantConnect.ToolBox.IQFeed.IQFeedDataQueueHandler.Connect() Lean\\ToolBox\\IQFeed\\IQFeedDataQueueHandler.cs:line 227
0

Just as a note i tested the IQFeed independantly on LEAN and it seems to work okay, so feed it good.

0

Jared Broad I will try to debug a bit if i get a chance on my end

0

I was able to debug a bit and found the initial problem. There's a ticker that causes the method ParseFutureTicker to fail. The ticker symbol is TST$Y

If i skip that symbol i no longer get the 'Object reference not set to an instance of an object' error

0

But after that error i see another error during Engine.Run()

20180809 19:51:52.500 ERROR:: Engine.Run(): During the algorithm initialization, the following exception has occurred: System.InvalidOperationException: Sequence contains no elements
at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
at QuantConnect.ToolBox.IQFeed.IQFeedDataQueueUniverseProvider.LookupSymbols(String lookupName, SecurityType securityType, String securityCurrency, String securityExchange) in Lean\ToolBox\IQFeed\IQFeedDataQueueUniverseProvider.cs:line 148
at QuantConnect.ToolBox.IQFeed.IQFeedDataQueueHandler.LookupSymbols(String lookupName, SecurityType securityType, String securityCurrency, String securityExchange) in Lean\ToolBox\IQFeed\IQFeedDataQueueHandler.cs:line 522
at QuantConnect.Lean.Engine.DataFeeds.Enumerators.Factories.FuturesChainUniverseSubscriptionEnumeratorFactory.CreateEnumerator(SubscriptionRequest request, IDataProvider dataProvider) in Lean\Engine\DataFeeds\Enumerators\Factories\FuturesChainUniverseSubscriptionEnumeratorFactory.cs:line 75
at QuantConnect.Lean.Engine.DataFeeds.LiveTradingDataFeed.CreateUniverseSubscription(SubscriptionRequest request) in Lean\Engine\DataFeeds\LiveTradingDataFeed.cs:line 692
at QuantConnect.Lean.Engine.DataFeeds.LiveTradingDataFeed.AddSubscription(SubscriptionRequest request) in Lean\Engine\DataFeeds\LiveTradingDataFeed.cs:line 171
at QuantConnect.Lean.Engine.DataFeeds.LiveTradingDataFeed.<>c__DisplayClass23_0.<Initialize>b__2(Object sender, NotifyCollectionChangedEventArgs args) in Lean\Engine\DataFeeds\LiveTradingDataFeed.cs:line 140
at QuantConnect.Securities.UniverseManager.OnCollectionChanged(NotifyCollectionChangedEventArgs e) in Lean\Common\Securities\UniverseManager.cs:line 263
at QuantConnect.Securities.UniverseManager.Add(Symbol key, Universe universe) in Lean\Common\Securities\UniverseManager.cs:line 171
at QuantConnect.Algorithm.QCAlgorithm.OnEndOfTimeStep() in Lean\Algorithm\QCAlgorithm.Universe.cs:line 138
at QuantConnect.Algorithm.QCAlgorithm.PostInitialize() in Lean\Algorithm\QCAlgorithm.cs:line 548
at QuantConnect.Lean.Engine.Setup.BrokerageSetupHandler.<>c__DisplayClass25_0.<Setup>b__1() in Lean\Engine\Setup\BrokerageSetupHandler.cs:line 219Sequence contains no elements
0

I did a bit of debugging on this error. It seems that the LookupSymbols method in IQFeedDataQueueUniverseProvider.cs cannot find a symbol in the _symbolUniverse and the result variable is null.

Later result is accessed and this causes an exception.

The code that fails is the call var symbolData = ... below

public IEnumerable<Symbol> LookupSymbols ( string lookupName, SecurityType securityType, string securityCurrency = null,
string securityExchange = null ) {
Func<Symbol, string> lookupFunc;

// for option, futures contract we search the underlying
if ( securityType == SecurityType.Option ||
securityType == SecurityType.Future ) {
lookupFunc = symbol => symbol.HasUnderlying ? symbol.Underlying.Value : string.Empty;
} else {
lookupFunc = symbol => symbol.Value;
}

var result = _symbolUniverse.Where ( x => lookupFunc ( x.Symbol ) == lookupName &&
x.Symbol.ID.SecurityType == securityType &&
( securityCurrency == null || x.SecurityCurrency == securityCurrency ) &&
( securityExchange == null || x.SecurityExchange == securityExchange ) )
.ToList();

bool onDemandRequests = result.All ( symbolData => !symbolData.IsDataLoaded() );

if ( onDemandRequests ) {
var exchanges = securityType == SecurityType.Future ?
_futuresExchanges.Values.Reverse().ToArray() :
new string[] { };

// sorting list of available contracts by exchange priority, taking the top 1
var symbolData =
result
.OrderByDescending ( e => Array.IndexOf ( exchanges, e ) )
.First();
0

The symbol passed into this method as variable lookupName was 'CL' as i'm running a Crude oil futures algo.

0

I'm attaching a copy of my test algorithm here. It subscribes to three futures contracts and just simply prints out the ticks

using System;
using QuantConnect.Data;
using QuantConnect.Securities;

namespace QuantConnect.Algorithm.CSharp {

public class MarketSpeedAlgorithm : QCAlgorithm {

private const string RootWTI = Futures.Energies.CrudeOilWTI;
private const string RootES = Futures.Indices.SP500EMini;
private const string RootNQ = Futures.Indices.NASDAQ100EMini;

public override void Initialize() {
SetEndDate ( 2018, 08, 13 );

var futureWTI_Tick = AddFuture ( RootWTI, Resolution.Tick );
futureWTI_Tick.SetFilter ( TimeSpan.Zero, TimeSpan.FromDays ( 30 ) );

var futureES_Tick = AddFuture ( RootES, Resolution.Tick );
futureES_Tick.SetFilter ( TimeSpan.Zero, TimeSpan.FromDays ( 182 ) );

var futureNQ_Tick = AddFuture ( RootNQ, Resolution.Tick );
futureNQ_Tick.SetFilter ( TimeSpan.Zero, TimeSpan.FromDays ( 182 ) );
}

public override void OnData ( Slice slice ) {
foreach ( var chain in slice.FutureChains ) {
foreach ( var contract in chain.Value ) {

var bid = contract.BidPrice;
var ask = contract.AskPrice;
var last = contract.LastPrice;
var volume = contract.Volume;

Log ( String.Format ( "{0},Bid={1} Ask={2} Last={3} Volume={4}",
contract.Symbol.Value,
bid, ask, last, volume ) );
}
}
}
}

} // namespace
0

This algorithm works fine when i have my setting as "environment": "live-interactive" but i get the failure mentioned above when using "environment": "live-interactive-iqfeed"

I'd really like to get the IQFeed working but it's not obvious why the code fails with IQFeed. Any help appreciated.

0

I'm using LEAN master branch and the latest version of the IQFeed client ( 6.0.0.5 )

0

Thanks ScalpTrader we'll look into it and report back what we find. Just a hunch - it might not be able to find the universe of future contracts through IQFeed. 

You can prove this by putting a breakpoint: "IQFeedDataQueueUniverseProvider.cs" > Line 223 and making sure its downloading and parsing the symbols you're trying to subscribe. 

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Looks like it's going into that method and successfully pulls the symbols from dtniq. I'm attaching log upto the failure point

20180814 13:04:22.516 Trace:: InteractiveBrokersBrokerage.ResetHandler(): thread started.
20180814 13:04:22.530 Trace:: IQFeed.Connect(): Launching client...
20180814 13:04:28.605 Trace:: IQFeed.Connect(): Connecting to admin...
20180814 13:04:35.892 Trace:: Loading IQFeed futures symbol map file...
20180814 13:04:39.531 Trace:: Loading and unzipping IQFeed symbol universe file (http://www.dtniq.com/product/mktsymbols_v2.zip)...
20180814 13:05:17.953 Trace:: ParseFutureTicker failed for ticker TST$Y
20180814 13:05:19.985 Trace:: Finished loading IQFeed symbol universe file.
20180814 13:05:20.027 Trace:: IQFeed.Connect(): Connecting to L1 data...
20180814 13:05:20.032 Trace:: IQFeed.Connect(): Connecting to Historical data...
20180814 13:05:20.033 Trace:: Config.Get(): Configuration key not found. Key: data-directory - Using default value: ../../../Data
20180814 13:05:20.033 Trace:: Config.GetValue(): max-history-minutes - Using default value: 5
20180814 13:05:20.041 Trace:: BaseDataExchange(DataQueueExchange) Starting...
20180814 13:05:20.041 Trace:: BaseDataExchange(CustomDataExchange) Starting...
20180814 13:05:20.049 Trace:: BrokerageSetupHandler.Setup(): Initializing algorithm...
20180814 13:05:20.049 Trace:: LiveTradingResultHandler.SendStatusUpdate(): Initializing Initializing algorithm...
20180814 13:05:20.054 Trace:: Config.GetValue(): send-via-api - Using default value: False
20180814 13:05:20.058 Trace:: Config.Get(): Configuration key not found. Key: security-data-feeds - Using default value:
20180814 13:05:20.148 Trace:: LiveTradingDataFeed.CreateUniverseSubscription(): Creating futures chain universe: /CL
20180814 13:05:21.051 Trace:: LiveTradingResultHandler.SendStatusUpdate(): LoggingIn Logging into brokerage...
20180814 13:05:21.051 Trace:: BrokerageSetupHandler.Setup(): Connecting to brokerage...
20180814 13:05:21.056 Trace:: InteractiveBrokersBrokerage.Connect(): Attempting to connect (1/5) ...
20180814 13:05:21.123 Trace:: IB message processing thread started: #24
20180814 13:05:21.125 Trace:: InteractiveBrokersBrokerage.HandleNextValidID(): 1
20180814 13:05:21.125 Trace:: IB next valid id received.
20180814 13:05:21.135 Trace:: InteractiveBrokersBrokerage.HandleError(): RequestId: -1 ErrorCode: 2104 - Market data farm connection is OK:usfuture
20180814 13:05:21.137 Trace:: Brokerage.OnMessage(): Warning - Code: 2104 - Market data farm connection is OK:usfuture
20180814 13:05:23.643 Trace:: BrokerageSetupHandler.Setup(): Fetching cash balance from brokerage...
20180814 13:05:23.645 Trace:: InteractiveBrokersBrokerage.CheckIbGateway(): start
20180814 13:05:23.755 Trace:: InteractiveBrokersBrokerage.CheckIbGateway(): end
20180814 13:05:23.764 Trace:: BrokerageSetupHandler.Setup(): Fetching open orders from brokerage...
20180814 13:05:23.769 Trace:: InteractiveBrokersBrokerage.HandleOpenOrderEnd()
20180814 13:05:23.770 Trace:: BrokerageSetupHandler.Setup(): Fetching holdings from brokerage...
20180814 13:05:23.771 Trace:: InteractiveBrokersBrokerage.CheckIbGateway(): start
20180814 13:05:23.851 Trace:: InteractiveBrokersBrokerage.CheckIbGateway(): end
20180814 13:05:23.862 ERROR:: Engine.Run(): During the algorithm initialization, the following exception has occurred: System.InvalidOperationException: Sequence contains no elements
at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
0

Did a bit more debugging on this.

Exception happens in this method

public IEnumerable<Symbol> LookupSymbols(string lookupName, SecurityType securityType, string securityCurrency = null, string securityExchange = null)
{
return _symbolUniverse.LookupSymbols(lookupName, securityType, securityCurrency, securityExchange);
}

_symbolUniverse was null

The other variables passed looked correct ( lookupName = "CL", securityType = Future )

0

Hi Jared,

Did you find anything wrong on your side?

Thanks.

0

If _symbolUniverse is null then Connect() was never called; this isn't what the logs show. 

IQFeedDataQueueUniverseProvider.cs -> Before L360 -> Try this: 

if (futuresTicker == "TST$Y") continue;

Does L90 return a lot of symbols?

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Yes after the code'

    symbols = LoadSymbols()

returns 83800 symbols in it. Looks good.

Skipping the TST$Y ticker avoids the first error but it still fails at the 'sequence contains no elements' error mentioned previously.

0

I'm not sure if this is correct or not but when the method LoadSymbols calls the following code

var parsed = SymbolRepresentation.ParseFutureTicker ( futuresTicker );
var underlyingString = parsed.Underlying;

With the futuresTicker = "CLQF19" it retruns "CLQ" as the underlying string

But the _iqFeedNameMap maps the string "QCL" to CL

So maybe the underlying string for the symbol and the map are out of sync?

0

@Jared What do you think the chances of this getting fixed are in the near term? I understand it's open source and part of the toolbox. I just curretly cannot move forward and really need the stability of the IQFeed.

0

Actually, I had error about "unzipping" when I tried to use LEAN as a "desktop" framework for options only. For stocks it worked fine.  I've been using desktop version in testing mode on Windows 10

I didn't spend much time debugging it and switched back to live testing, but I assume that in case of options it downloads history as some Zip archive and, most probably, some path in Lean is hardcoded for Linux, or live server, so desktop version simply don't know where to unpack historical quotes. 

0

ScalpTrader please give that a go. We've pushed a fix for the futures IQFeed adaptor.

artemiusgreat Not sure what you're saying but IQFeed is only for live-trading in LEAN. If you have a specific issue you can clearly state the community may look into it with your help.

Open source is free as in freedom to modify and extend as you wish; not free as in free-beer.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Jared Broad I have tested the fix submitted by JJD and it seems to be working well. Thanks a lot for getting this fixed.

0

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed