Discussion Forum - QuantConnect.com

ScalpTrader INVESTOR

| Reward Discussion

I'm thinking about using IQFeed for market data with IB using the " live-interactive-iqfeed " environment

Quick question. Do i have to start IQFeed prior to starting LEAN or will the LEAN engine start the IQFeed process?

Thanks

Update Backtest

ScalpTrader

Accepted Answer

Update Backtest

Notebook

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

ScalpTrader

509 Pro ,

When i start up LEAN with the live-interactive-iqfeed environment variable IB gateway launches and then the IQConnect process is launched. But the login fails in IQConnect. I know that the username and password is correct as i'm able to use the feed with the IQFeed sample applications. Any idea why it would fail when using LEAN?

ScalpTrader

509 Pro ,

I'm able to get a bit further by starting the IQConnect program before launching LEAN. If that is done, LEAN connects to the admin port but then there's another failure

20180807 14:04:19.313 Trace:: IQFeed.Connect(): Connecting to admin...
20180807 14:04:20.341 Trace:: Loading IQFeed futures symbol map file...
20180807 14:04:20.344 Trace:: Found up-to-date IQFeed symbol universe file in local cache. Loading it...
20180807 14:04:33.997 ERROR:: IQFeed.Connect(): Error Connecting to IQFeed: Object reference not set to an instance of an object.

ScalpTrader

509 Pro ,

Does anyone know what are the correct values for these fields are in the config for IQFeed? I believe that productName needs to be valid for an application to attach to IQConnect

    // iqfeed configuration
    "iqfeed-productName": "",
    "iqfeed-version": "1.0",

ScalpTrader

509 Pro ,

Does one need to be registered as a developer with IQFeed to get the product name?

Jared Broad

STAFF Pro ,

Yes I think so, I remember in our testing we were registered. What object

was the null exception on?

ScalpTrader

509 Pro ,

Thanks Jared.

The null exception seems to be coming from the call to;

_symbolUniverse = new IQFeedDataQueueUniverseProvider()

Jared Broad

STAFF Pro ,

Please post the full stack trace and error message

ScalpTrader

509 Pro ,

QuantConnect.ToolBox.IQFeed.IQFeedDataQueueUniverseProvider.LoadSymbols() in Lean\\ToolBox\\IQFeed\\IQFeedDataQueueUniverseProvider.cs:line 361

QuantConnect.ToolBox.IQFeed.IQFeedDataQueueUniverseProvider..ctor() in Lean\\ToolBox\\IQFeed\\IQFeedDataQueueUniverseProvider.cs:line 90

QuantConnect.ToolBox.IQFeed.IQFeedDataQueueHandler.Connect() Lean\\ToolBox\\IQFeed\\IQFeedDataQueueHandler.cs:line 227

ScalpTrader

509 Pro ,

Just as a note i tested the IQFeed independantly on LEAN and it seems to work okay, so feed it good.

ScalpTrader

509 Pro ,

Jared Broad I will try to debug a bit if i get a chance on my end

ScalpTrader

509 Pro ,

I was able to debug a bit and found the initial problem. There's a ticker that causes the method ParseFutureTicker to fail. The ticker symbol is TST$Y

If i skip that symbol i no longer get the 'Object reference not set to an instance of an object' error

ScalpTrader

509 Pro ,

But after that error i see another error during Engine.Run()

20180809 19:51:52.500 ERROR:: Engine.Run(): During the algorithm initialization, the following exception has occurred: System.InvalidOperationException: Sequence contains no elements

   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)

   at QuantConnect.ToolBox.IQFeed.IQFeedDataQueueUniverseProvider.LookupSymbols(String lookupName, SecurityType securityType, String securityCurrency, String securityExchange) in Lean\ToolBox\IQFeed\IQFeedDataQueueUniverseProvider.cs:line 148

   at QuantConnect.ToolBox.IQFeed.IQFeedDataQueueHandler.LookupSymbols(String lookupName, SecurityType securityType, String securityCurrency, String securityExchange) in Lean\ToolBox\IQFeed\IQFeedDataQueueHandler.cs:line 522

   at QuantConnect.Lean.Engine.DataFeeds.Enumerators.Factories.FuturesChainUniverseSubscriptionEnumeratorFactory.CreateEnumerator(SubscriptionRequest request, IDataProvider dataProvider) in Lean\Engine\DataFeeds\Enumerators\Factories\FuturesChainUniverseSubscriptionEnumeratorFactory.cs:line 75

   at QuantConnect.Lean.Engine.DataFeeds.LiveTradingDataFeed.CreateUniverseSubscription(SubscriptionRequest request) in Lean\Engine\DataFeeds\LiveTradingDataFeed.cs:line 692

   at QuantConnect.Lean.Engine.DataFeeds.LiveTradingDataFeed.AddSubscription(SubscriptionRequest request) in Lean\Engine\DataFeeds\LiveTradingDataFeed.cs:line 171

   at QuantConnect.Lean.Engine.DataFeeds.LiveTradingDataFeed.<>c__DisplayClass23_0.<Initialize>b__2(Object sender, NotifyCollectionChangedEventArgs args) in Lean\Engine\DataFeeds\LiveTradingDataFeed.cs:line 140

   at QuantConnect.Securities.UniverseManager.OnCollectionChanged(NotifyCollectionChangedEventArgs e) in Lean\Common\Securities\UniverseManager.cs:line 263

   at QuantConnect.Securities.UniverseManager.Add(Symbol key, Universe universe) in Lean\Common\Securities\UniverseManager.cs:line 171

   at QuantConnect.Algorithm.QCAlgorithm.OnEndOfTimeStep() in Lean\Algorithm\QCAlgorithm.Universe.cs:line 138

   at QuantConnect.Algorithm.QCAlgorithm.PostInitialize() in Lean\Algorithm\QCAlgorithm.cs:line 548

   at QuantConnect.Lean.Engine.Setup.BrokerageSetupHandler.<>c__DisplayClass25_0.<Setup>b__1() in Lean\Engine\Setup\BrokerageSetupHandler.cs:line 219Sequence contains no elements

ScalpTrader

509 Pro ,

I did a bit of debugging on this error. It seems that the LookupSymbols method in IQFeedDataQueueUniverseProvider.cs cannot find a symbol in the _symbolUniverse and the result variable is null.

Later result is accessed and this causes an exception.

The code that fails is the call var symbolData = ... below

    public IEnumerable<Symbol> LookupSymbols ( string lookupName, SecurityType securityType, string securityCurrency = null,

            string securityExchange = null ) {

        Func<Symbol, string> lookupFunc;



        // for option, futures contract we search the underlying

        if ( securityType == SecurityType.Option ||

                securityType == SecurityType.Future ) {

            lookupFunc = symbol => symbol.HasUnderlying ? symbol.Underlying.Value : string.Empty;

        } else {

            lookupFunc = symbol => symbol.Value;

        }



        var result = _symbolUniverse.Where ( x => lookupFunc ( x.Symbol ) == lookupName &&

                                             x.Symbol.ID.SecurityType == securityType &&

                                             ( securityCurrency == null || x.SecurityCurrency == securityCurrency ) &&

                                             ( securityExchange == null || x.SecurityExchange == securityExchange ) )

                     .ToList();



        bool onDemandRequests = result.All ( symbolData => !symbolData.IsDataLoaded() );



        if ( onDemandRequests ) {

            var exchanges = securityType == SecurityType.Future ?

                            _futuresExchanges.Values.Reverse().ToArray() :

                            new string[] { };



            // sorting list of available contracts by exchange priority, taking the top 1

            var symbolData =

                result

                .OrderByDescending ( e => Array.IndexOf ( exchanges, e ) )

                .First();

ScalpTrader

509 Pro ,

The symbol passed into this method as variable lookupName was 'CL' as i'm running a Crude oil futures algo.

ScalpTrader

509 Pro ,

I'm attaching a copy of my test algorithm here. It subscribes to three futures contracts and just simply prints out the ticks

using System;

using QuantConnect.Data;

using QuantConnect.Securities;



namespace QuantConnect.Algorithm.CSharp {



public class MarketSpeedAlgorithm : QCAlgorithm {



    private const string RootWTI = Futures.Energies.CrudeOilWTI;

    private const string RootES = Futures.Indices.SP500EMini;

    private const string RootNQ = Futures.Indices.NASDAQ100EMini;



    public override void Initialize() {

        SetEndDate ( 2018, 08, 13 );



        var futureWTI_Tick = AddFuture ( RootWTI, Resolution.Tick );

        futureWTI_Tick.SetFilter ( TimeSpan.Zero, TimeSpan.FromDays ( 30 ) );



        var futureES_Tick = AddFuture ( RootES, Resolution.Tick );

        futureES_Tick.SetFilter ( TimeSpan.Zero, TimeSpan.FromDays ( 182 ) );



        var futureNQ_Tick = AddFuture ( RootNQ, Resolution.Tick );

        futureNQ_Tick.SetFilter ( TimeSpan.Zero, TimeSpan.FromDays ( 182 ) );

    }



    public override void OnData ( Slice slice ) {

        foreach ( var chain in slice.FutureChains ) {

            foreach ( var contract in chain.Value ) {



                var bid = contract.BidPrice;

                var ask = contract.AskPrice;

                var last = contract.LastPrice;

                var volume = contract.Volume;



                Log ( String.Format ( "{0},Bid={1} Ask={2} Last={3} Volume={4}",

                                      contract.Symbol.Value,

                                      bid, ask, last, volume ) );

            }

        }

    }

}



} // namespace

ScalpTrader

509 Pro ,

This algorithm works fine when i have my setting as "environment": "live-interactive" but i get the failure mentioned above when using "environment": "live-interactive-iqfeed"

I'd really like to get the IQFeed working but it's not obvious why the code fails with IQFeed. Any help appreciated.

ScalpTrader

509 Pro ,

I'm using LEAN master branch and the latest version of the IQFeed client ( 6.0.0.5 )

Jared Broad

STAFF Pro ,

Thanks ScalpTrader we'll look into it and report back what we find. Just a hunch - it might not be able to find the universe of future contracts through IQFeed.

You can prove this by putting a breakpoint: "IQFeedDataQueueUniverseProvider.cs" > Line 223 and making sure its downloading and parsing the symbols you're trying to subscribe.

ScalpTrader

509 Pro ,

Looks like it's going into that method and successfully pulls the symbols from dtniq. I'm attaching log upto the failure point

20180814 13:04:22.516 Trace:: InteractiveBrokersBrokerage.ResetHandler(): thread started.

20180814 13:04:22.530 Trace:: IQFeed.Connect(): Launching client...

20180814 13:04:28.605 Trace:: IQFeed.Connect(): Connecting to admin...

20180814 13:04:35.892 Trace:: Loading IQFeed futures symbol map file...

20180814 13:04:39.531 Trace:: Loading and unzipping IQFeed symbol universe file (http://www.dtniq.com/product/mktsymbols_v2.zip)...

20180814 13:05:17.953 Trace:: ParseFutureTicker failed for ticker TST$Y

20180814 13:05:19.985 Trace:: Finished loading IQFeed symbol universe file.

20180814 13:05:20.027 Trace:: IQFeed.Connect(): Connecting to L1 data...

20180814 13:05:20.032 Trace:: IQFeed.Connect(): Connecting to Historical data...

20180814 13:05:20.033 Trace:: Config.Get(): Configuration key not found. Key: data-directory - Using default value: ../../../Data

20180814 13:05:20.033 Trace:: Config.GetValue(): max-history-minutes - Using default value: 5

20180814 13:05:20.041 Trace:: BaseDataExchange(DataQueueExchange) Starting...

20180814 13:05:20.041 Trace:: BaseDataExchange(CustomDataExchange) Starting...

20180814 13:05:20.049 Trace:: BrokerageSetupHandler.Setup(): Initializing algorithm...

20180814 13:05:20.049 Trace:: LiveTradingResultHandler.SendStatusUpdate(): Initializing Initializing algorithm...

20180814 13:05:20.054 Trace:: Config.GetValue(): send-via-api - Using default value: False

20180814 13:05:20.058 Trace:: Config.Get(): Configuration key not found. Key: security-data-feeds - Using default value:

20180814 13:05:20.148 Trace:: LiveTradingDataFeed.CreateUniverseSubscription(): Creating futures chain universe: /CL

20180814 13:05:21.051 Trace:: LiveTradingResultHandler.SendStatusUpdate(): LoggingIn Logging into brokerage...

20180814 13:05:21.051 Trace:: BrokerageSetupHandler.Setup(): Connecting to brokerage...

20180814 13:05:21.056 Trace:: InteractiveBrokersBrokerage.Connect(): Attempting to connect (1/5) ...

20180814 13:05:21.123 Trace:: IB message processing thread started: #24

20180814 13:05:21.125 Trace:: InteractiveBrokersBrokerage.HandleNextValidID(): 1

20180814 13:05:21.125 Trace:: IB next valid id received.

20180814 13:05:21.135 Trace:: InteractiveBrokersBrokerage.HandleError(): RequestId: -1 ErrorCode: 2104 - Market data farm connection is OK:usfuture

20180814 13:05:21.137 Trace:: Brokerage.OnMessage(): Warning - Code: 2104 - Market data farm connection is OK:usfuture

20180814 13:05:23.643 Trace:: BrokerageSetupHandler.Setup(): Fetching cash balance from brokerage...

20180814 13:05:23.645 Trace:: InteractiveBrokersBrokerage.CheckIbGateway(): start

20180814 13:05:23.755 Trace:: InteractiveBrokersBrokerage.CheckIbGateway(): end

20180814 13:05:23.764 Trace:: BrokerageSetupHandler.Setup(): Fetching open orders from brokerage...

20180814 13:05:23.769 Trace:: InteractiveBrokersBrokerage.HandleOpenOrderEnd()

20180814 13:05:23.770 Trace:: BrokerageSetupHandler.Setup(): Fetching holdings from brokerage...

20180814 13:05:23.771 Trace:: InteractiveBrokersBrokerage.CheckIbGateway(): start

20180814 13:05:23.851 Trace:: InteractiveBrokersBrokerage.CheckIbGateway(): end

20180814 13:05:23.862 ERROR:: Engine.Run(): During the algorithm initialization, the following exception has occurred: System.InvalidOperationException: Sequence contains no elements

   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)

ScalpTrader

509 Pro ,

Did a bit more debugging on this.

Exception happens in this method

public IEnumerable<Symbol> LookupSymbols(string lookupName, SecurityType securityType, string securityCurrency = null, string securityExchange = null)

{

 return _symbolUniverse.LookupSymbols(lookupName, securityType, securityCurrency, securityExchange);

}

_symbolUniverse was null

The other variables passed looked correct ( lookupName = "CL", securityType = Future )

ScalpTrader

509 Pro ,

Hi Jared,

Did you find anything wrong on your side?

Thanks.

Jared Broad

STAFF Pro ,

If _symbolUniverse is null then Connect() was never called; this isn't what the logs show.

IQFeedDataQueueUniverseProvider.cs -> Before L360 -> Try this:

if (futuresTicker == "TST$Y") continue;

Does L90 return a lot of symbols?

ScalpTrader

509 Pro ,

Yes after the code'

symbols = LoadSymbols()

returns 83800 symbols in it. Looks good.

Skipping the TST$Y ticker avoids the first error but it still fails at the 'sequence contains no elements' error mentioned previously.

ScalpTrader

509 Pro ,

I'm not sure if this is correct or not but when the method LoadSymbols calls the following code

var parsed = SymbolRepresentation.ParseFutureTicker ( futuresTicker );

var underlyingString = parsed.Underlying;

With the futuresTicker = "CLQF19" it retruns "CLQ" as the underlying string

But the _iqFeedNameMap maps the string "QCL" to CL

So maybe the underlying string for the symbol and the map are out of sync?

ScalpTrader

509 Pro ,

@Jared What do you think the chances of this getting fixed are in the near term? I understand it's open source and part of the toolbox. I just curretly cannot move forward and really need the stability of the IQFeed.

Artemiusgreat

2k Pro ,

Actually, I had error about "unzipping" when I tried to use LEAN as a "desktop" framework for options only. For stocks it worked fine. I've been using desktop version in testing mode on Windows 10.

I didn't spend much time debugging it and switched back to live testing, but I assume that in case of options it downloads history as some Zip archive and, most probably, some path in Lean is hardcoded for Linux, or live server, so desktop version simply don't know where to unpack historical quotes.

Jared Broad

STAFF Pro ,

ScalpTrader please give that a go. We've pushed a fix for the futures IQFeed adaptor.

artemiusgreat Not sure what you're saying but IQFeed is only for live-trading in LEAN. If you have a specific issue you can clearly state the community may look into it with your help.

Open source is free as in freedom to modify and extend as you wish; not free as in free-beer.

ScalpTrader

509 Pro ,

Jared Broad I have tested the fix submitted by JJD and it seems to be working well. Thanks a lot for getting this fixed.

ScalpTrader INVESTOR

Update Backtest

Notebook

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!

Platform

QuantConnect Community

Alpha League Competition: $1,000 Weekly Prize Pool

Filter Discussion by Tags

240100 Quants.

Online Now

Stay Connected

IQFeed for live trading in LEAN

Started By:

Join QuantConnect for Free Sign Up

Platform

SIGN IN

QuantConnect Community

Alpha League Competition: $1,000 Weekly Prize Pool

Filter Discussion by Tags

240100 Quants.

Online Now

Stay Connected

IQFeed for live trading in LEAN

Started By:

Join QuantConnect for Free Sign Up