Issue with Data Download Oanda by Loudk

Loudk

, edit

Hi,

I have followed steps to setup the needful, set up the "job-user-id" and "api-access-token" keys, data-provider: "QuantConnect.Lean.Engine.DataFeeds.ApiDataProvider" and when run it shows following red message :

ERROR:: ApiDataProvider.Fetch(): Unable to remotely retrieve data for path ../../../Data/forex\fxcm\minute\eurusd

I purchased Oanda historical data and not Fxcm.

Kindly assist.

Thanks.

Update Backtest

15 Comments

Loudk

, ,

Accepted Answer

Notebook

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Jared Broad

Your algorithm is probably not specifying OANDA data. If so; please share the algorithm here for assistance.

Loudk

109 ,

Hi, this is my config.json

{

  // this configuration file works by first loading all top-level

  // configuration items and then will load the specified environment

  // on top, this provides a layering affect. environment names can be

  // anything, and just require definition in this file. There's

  // two predefined environments, 'backtesting' and 'live', feel free

  // to add more!



  "environment": "backtesting", // "live-paper", "backtesting", "live-interactive", "live-interactive-iqfeed"



  // algorithm class selector

  "algorithm-type-name": "BasicTemplateForexAlgorithm",



  // Algorithm language selector - options CSharp, FSharp, VisualBasic, Python, Java

  "algorithm-language": "CSharp",



  //Physical DLL location

  "algorithm-location": "QuantConnect.Algorithm.CSharp.dll",

  //"algorithm-location": "QuantConnect.Algorithm.Python.dll",

  //"algorithm-location": "QuantConnect.Algorithm.FSharp.dll",

  //"algorithm-location": "QuantConnect.Algorithm.VisualBasic.dll",

  //"algorithm-location": "QuantConnect.Algorithm.Java.dll",



  // engine

  "data-folder": "../../../Data/",



  // handlers

  "log-handler": "QuantConnect.Logging.CompositeLogHandler",

  "messaging-handler": "QuantConnect.Messaging.Messaging",

  "job-queue-handler": "QuantConnect.Queues.JobQueue",

  "api-handler": "QuantConnect.Api.Api",

  "command-queue-handler": "QuantConnect.Queues.EmptyCommandQueueHandler",

  // this command queue will check a file to read json serialized commands (use TypeNameHandling.All)

  //"command-queue-handler": "QuantConnect.Queues.FileCommandQueueHandler",

  // used by the FileComandQueueHandler, serialize a ICommand and it will be run against algorithm

  "command-json-file": "command.json",

  "map-file-provider": "QuantConnect.Data.Auxiliary.LocalDiskMapFileProvider",

  "factor-file-provider": "QuantConnect.Data.Auxiliary.LocalDiskFactorFileProvider",

  "data-provider": "QuantConnect.Lean.Engine.DataFeeds.ApiDataProvider",



  // limits on number of symbols to allow

  "symbol-minute-limit": 10000,

  "symbol-second-limit": 10000,

  "symbol-tick-limit": 10000,



  // if one uses true in following token, market hours will remain open all hours and all days.

  // if one uses false will make lean operate only during regular market hours.

  "force-exchange-always-open": false,



  // save list of transactions to the specified csv file

  "transaction-log": "",



  // To get your api access token go to quantconnect.com/account

  "job-user-id": "",

  "api-access-token": "",



  // live data configuration

  "live-data-url": "ws://www.quantconnect.com/api/v2/live/data/",

  "live-data-port": 8020,



  // interactive brokers configuration

  "ib-account": "",

  "ib-user-name": "",

  "ib-password": "",

  "ib-host": "127.0.0.1",

  "ib-port": "4002",

  "ib-agent-description": "Individual",

  "ib-use-tws": false,

  "ib-tws-dir": "C:\\Jts",

  "ib-trading-mode": "paper",

  "ib-controller-dir": "C:\\IBController",



  // tradier configuration

  "tradier-account-id": "",

  "tradier-access-token": "",

  "tradier-refresh-token": "",

  "tradier-issued-at": "",

  "tradier-lifespan": "",

  "tradier-refresh-session": true,



  // oanda configuration

  "oanda-environment": "Practice",

  "oanda-access-token": "",

  "oanda-account-id": "",



  // fxcm configuration

  "fxcm-server": "http://www.fxcorporate.com/Hosts.jsp",

  "fxcm-terminal": "Demo", //Real or Demo

  "fxcm-user-name": "",

  "fxcm-password": "",

  "fxcm-account-id": "",



  // iqfeed configuration

  "iqfeed-username": "",

  "iqfeed-password": "",

  "iqfeed-productName": "",

  "iqfeed-version": "1.0",



  // Required to access data from Quandl

  // To get your access token go to https://www.quandl.com/account/api

  "quandl-auth-token": "",



  // parameters to set in the algorithm (the below are just samples)

  "parameters": {

    "ema-fast": 10,

    "ema-slow": 20

  },



  "environments": {



    // defines the 'backtesting' environment

    "backtesting": {

      "live-mode": false,



      "setup-handler": "QuantConnect.Lean.Engine.Setup.ConsoleSetupHandler",

      "result-handler": "QuantConnect.Lean.Engine.Results.BacktestingResultHandler",

      "data-feed-handler": "QuantConnect.Lean.Engine.DataFeeds.FileSystemDataFeed",

      "real-time-handler": "QuantConnect.Lean.Engine.RealTime.BacktestingRealTimeHandler",

      "transaction-handler": "QuantConnect.Lean.Engine.TransactionHandlers.BacktestingTransactionHandler"

    },



    // defines the 'live-paper' environment

    "live-paper": {

      "live-mode": true,



      // the paper brokerage requires the BacktestingTransactionHandler

      "live-mode-brokerage": "PaperBrokerage",



      "setup-handler": "QuantConnect.Lean.Engine.Setup.BrokerageSetupHandler",

      "result-handler": "QuantConnect.Lean.Engine.Results.LiveTradingResultHandler",

      "data-feed-handler": "QuantConnect.Lean.Engine.DataFeeds.LiveTradingDataFeed",

      "data-queue-handler": "QuantConnect.Lean.Engine.DataFeeds.Queues.LiveDataQueue",

      "real-time-handler": "QuantConnect.Lean.Engine.RealTime.LiveTradingRealTimeHandler",

      "transaction-handler": "QuantConnect.Lean.Engine.TransactionHandlers.BacktestingTransactionHandler"

    },



    // defines the 'live-tradier' environment

    "live-tradier": {

      "live-mode": true,



      // this setting will save tradier access/refresh tokens to a tradier-tokens.txt file

      // that can be read in next time, this makes it easier to start/stop a tradier algorithm

      "tradier-save-tokens": true,



      // real brokerage implementations require the BrokerageTransactionHandler

      "live-mode-brokerage": "TradierBrokerage",

      "data-queue-handler": "TradierBrokerage",



      "setup-handler": "QuantConnect.Lean.Engine.Setup.BrokerageSetupHandler",

      "result-handler": "QuantConnect.Lean.Engine.Results.LiveTradingResultHandler",

      "data-feed-handler": "QuantConnect.Lean.Engine.DataFeeds.LiveTradingDataFeed",

      "real-time-handler": "QuantConnect.Lean.Engine.RealTime.LiveTradingRealTimeHandler",

      "transaction-handler": "QuantConnect.Lean.Engine.TransactionHandlers.BrokerageTransactionHandler"

    },



    // defines the 'live-interactive' environment

    "live-interactive": {

      "live-mode": true,



      // real brokerage implementations require the BrokerageTransactionHandler

      "live-mode-brokerage": "InteractiveBrokersBrokerage",

      "setup-handler": "QuantConnect.Lean.Engine.Setup.BrokerageSetupHandler",

      "result-handler": "QuantConnect.Lean.Engine.Results.LiveTradingResultHandler",

      "data-feed-handler": "QuantConnect.Lean.Engine.DataFeeds.LiveTradingDataFeed",

      "data-queue-handler": "QuantConnect.Brokerages.InteractiveBrokers.InteractiveBrokersBrokerage",

      "real-time-handler": "QuantConnect.Lean.Engine.RealTime.LiveTradingRealTimeHandler",

      "transaction-handler": "QuantConnect.Lean.Engine.TransactionHandlers.BrokerageTransactionHandler",

      "history-provider": "BrokerageHistoryProvider"

    },



    // defines the 'live-interactive-iqfeed' environment

    "live-interactive-iqfeed": {

      "live-mode": true,



      // real brokerage implementations require the BrokerageTransactionHandler

      "live-mode-brokerage": "InteractiveBrokersBrokerage",

      "setup-handler": "QuantConnect.Lean.Engine.Setup.BrokerageSetupHandler",

      "result-handler": "QuantConnect.Lean.Engine.Results.LiveTradingResultHandler",

      "data-feed-handler": "QuantConnect.Lean.Engine.DataFeeds.LiveTradingDataFeed",

      "data-queue-handler": "QuantConnect.ToolBox.IQFeed.IQFeedDataQueueHandler",

      "real-time-handler": "QuantConnect.Lean.Engine.RealTime.LiveTradingRealTimeHandler",

      "transaction-handler": "QuantConnect.Lean.Engine.TransactionHandlers.BrokerageTransactionHandler",

      "history-provider": "QuantConnect.ToolBox.IQFeed.IQFeedDataQueueHandler"

    },



    // defines the 'live-fxcm' environment

    "live-fxcm": {

      "live-mode": true,



      // real brokerage implementations require the BrokerageTransactionHandler

      "live-mode-brokerage": "FxcmBrokerage",

      "data-queue-handler": "FxcmBrokerage",

      "setup-handler": "QuantConnect.Lean.Engine.Setup.BrokerageSetupHandler",

      "result-handler": "QuantConnect.Lean.Engine.Results.LiveTradingResultHandler",

      "data-feed-handler": "QuantConnect.Lean.Engine.DataFeeds.LiveTradingDataFeed",

      "real-time-handler": "QuantConnect.Lean.Engine.RealTime.LiveTradingRealTimeHandler",

      "transaction-handler": "QuantConnect.Lean.Engine.TransactionHandlers.BrokerageTransactionHandler",

      "history-provider": "BrokerageHistoryProvider"

    },



    // defines the 'live-oanda' environment

    "live-oanda": {

      "live-mode": true,



      // real brokerage implementations require the BrokerageTransactionHandler

      "live-mode-brokerage": "OandaBrokerage",

      "data-queue-handler": "OandaBrokerage",

      "setup-handler": "QuantConnect.Lean.Engine.Setup.BrokerageSetupHandler",

      "result-handler": "QuantConnect.Lean.Engine.Results.LiveTradingResultHandler",

      "data-feed-handler": "QuantConnect.Lean.Engine.DataFeeds.LiveTradingDataFeed",

      "real-time-handler": "QuantConnect.Lean.Engine.RealTime.LiveTradingRealTimeHandler",

      "transaction-handler": "QuantConnect.Lean.Engine.TransactionHandlers.BrokerageTransactionHandler",

      "history-provider": "BrokerageHistoryProvider"

    },



    // defines the 'backtesting-desktop' environment

    "backtesting-desktop": {

      "live-mode": false,

      "send-via-api": true,



      "setup-handler": "QuantConnect.Lean.Engine.Setup.ConsoleSetupHandler",

      "result-handler": "QuantConnect.Lean.Engine.Results.BacktestingResultHandler",

      "data-feed-handler": "QuantConnect.Lean.Engine.DataFeeds.FileSystemDataFeed",

      "real-time-handler": "QuantConnect.Lean.Engine.RealTime.BacktestingRealTimeHandler",

      "transaction-handler": "QuantConnect.Lean.Engine.TransactionHandlers.BacktestingTransactionHandler",

      "messaging-handler": "QuantConnect.Messaging.StreamingMessageHandler",

      "log-handler": "QuantConnect.Logging.QueueLogHandler",

      "desktop-http-port": "1234",

      "desktop-exe": "../../../UserInterface/bin/Debug/QuantConnect.Views.exe"

    },



    // defines the 'live-desktop' environment

    "live-desktop": {

      "live-mode": true,

      "send-via-api": true,



      // Set your own brokerage and data queue handlers here.

      // Live desktop charting isn't as cool as on quantconnect.com but its pretty neat!

      "live-mode-brokerage": "FxcmBrokerage",

      "data-queue-handler": "FxcmBrokerage",

      "setup-handler": "QuantConnect.Lean.Engine.Setup.BrokerageSetupHandler",

      "result-handler": "QuantConnect.Lean.Engine.Results.LiveTradingResultHandler",

      "data-feed-handler": "QuantConnect.Lean.Engine.DataFeeds.LiveTradingDataFeed",

      "real-time-handler": "QuantConnect.Lean.Engine.RealTime.LiveTradingRealTimeHandler",

      "transaction-handler": "QuantConnect.Lean.Engine.TransactionHandlers.BrokerageTransactionHandler",

      "messaging-handler": "QuantConnect.Messaging.StreamingMessageHandler",

      "log-handler": "QuantConnect.Logging.QueueLogHandler",

      "desktop-http-port": "1234",

      "desktop-exe": "../../../UserInterface/bin/Release/QuantConnect.Views.exe"

    }

  }

}

Loudk

109 ,

oppps i forgot to remove my id ... can please help delete the above ?

Jared Broad

Your algorithm needs to specify Market.Oanda; e.g.

AddForex("EURUSD", Resolution.Minute, Market.Oanda);

Loudk

109 ,

Thank you, Jared. I'll re-try. about my token in the code snippet, can please help delete it ?

Jared Broad

Already done

Loudk

109 ,

Sorry please ignore previous comment as issue's resolved. Thanks !

Loudk

109 ,

Encounter with this issue , may i know how to proceed ? Thanks.

20170412 23:42:44 ERROR:: ZipDataCacheProvider.Fetch(): Inner try/catch Ionic.Zip.ZipException: Cannot read that as a ZipFile ---> Ionic.Zip.BadReadException: Bad signature (0x6D783F3C) at position 0x00000000
at Ionic.Zip.ZipEntry.ReadHeader(ZipEntry ze, Encoding defaultEncoding)
at Ionic.Zip.ZipEntry.ReadEntry(ZipContainer zc, Boolean first)
at Ionic.Zip.ZipFile.ReadIntoInstance_Orig(ZipFile zf)
at Ionic.Zip.ZipFile.ReadIntoInstance(ZipFile zf)
--- End of inner exception stack trace ---
at Ionic.Zip.ZipFile.ReadIntoInstance(ZipFile zf)
at Ionic.Zip.ZipFile.Read(Stream zipStream, TextWriter statusMessageWriter, Encoding encoding, EventHandler`1 readProgress)
at QuantConnect.Lean.Engine.DataFeeds.ZipDataCacheProvider.<>c__DisplayClass5_0.<Fetch>b__0(String x) in D:\Lean-master\Engine\DataFeeds\ZipDataCacheProvider.cs:line 80
at QuantConnect.Extensions.<>c__DisplayClass7_0`2.<AddOrUpdate>b__0() in D:\Lean-master\Common\Extensions.cs:line 150
at System.Lazy`1.CreateValue()
at System.Lazy`1.LazyInitValue()
at QuantConnect.Extensions.AddOrUpdate[TKey,TValue](ConcurrentDictionary`2 dictionary, TKey key, Func`2 addValueFactory, Func`3 updateValueFactory) in D:\Lean-master\Common\Extensions.cs:line 151
at QuantConnect.Lean.Engine.DataFeeds.ZipDataCacheProvider.Fetch(String key) in D:\Lean-master\Engine\DataFeeds\ZipDataCacheProvider.cs:line 77
20170412 23:42:44 ERROR:: ZipDataCacheProvider.Fetch(): Inner try/catch Ionic.Zip.ZipException: Cannot read that as a ZipFile ---> Ionic.Zip.BadReadException: Bad signature (0x6D783F3C) at position 0x00000000
at Ionic.Zip.ZipEntry.ReadHeader(ZipEntry ze, Encoding defaultEncoding)
at Ionic.Zip.ZipEntry.ReadEntry(ZipContainer zc, Boolean first)
at Ionic.Zip.ZipFile.ReadIntoInstance_Orig(ZipFile zf)
at Ionic.Zip.ZipFile.ReadIntoInstance(ZipFile zf)
--- End of inner exception stack trace ---
at Ionic.Zip.ZipFile.ReadIntoInstance(ZipFile zf)
at Ionic.Zip.ZipFile.Read(Stream zipStream, TextWriter statusMessageWriter, Encoding encoding, EventHandler`1 readProgress)
at QuantConnect.Lean.Engine.DataFeeds.ZipDataCacheProvider.<>c__DisplayClass5_0.<Fetch>b__0(String x) in D:\Lean-master\Engine\DataFeeds\ZipDataCacheProvider.cs:line 80
at QuantConnect.Extensions.<>c__DisplayClass7_0`2.<AddOrUpdate>b__0() in D:\Lean-master\Common\Extensions.cs:line 150
at System.Lazy`1.CreateValue()
at System.Lazy`1.LazyInitValue()
at QuantConnect.Extensions.AddOrUpdate[TKey,TValue](ConcurrentDictionary`2 dictionary, TKey key, Func`2 addValueFactory, Func`3 updateValueFactory) in D:\Lean-master\Common\Extensions.cs:line 151
at QuantConnect.Lean.Engine.DataFeeds.ZipDataCacheProvider.Fetch(String key) in D:\Lean-master\Engine\DataFeeds\ZipDataCacheProvider.cs:line 77

Alexandre Catarino

It looks like the engine is trying to decompress a bad file.
Could please checkout the files that were downloaded and see whether they are OK?
You will probably see one or more files with zero size and when you try to open (click on) it, the default software for openning zip files will show you an error. If so, delete this file and rerun Lean. It will download the file again and the issue will be fixed.
If the problem persists, please tell us which the problematic file is.

Loudk

109 ,

Thank you, Alexandre. It was this 1 file with size 1KB that couldn't be opened.

This XML file does not appear to have any style information associated with it. The document tree is shown below.

<Error>

<Code>NoSuchKey</Code>

<Message>The specified key does not exist.</Message>

<Key>forex/oanda/minute/eurusd/20151225_quote.zip</Key>

<RequestId>D268303B85EA454C</RequestId>

<HostId>

WsO2T9VYO8kHz4oTj8kXR3uPohmM9r33ZkrX8MyNE09YdRIV0HfiCWQZgaBGwiZH9ayKxDpprXA=

</HostId>

</Error>

Loudk

109 ,

20160101_quote.zip also with 1KB size but damaged.

This XML file does not appear to have any style information associated with it. The document tree is shown below.

<Error>

<Code>NoSuchKey</Code>

<Message>The specified key does not exist.</Message>

<Key>forex/oanda/minute/eurusd/20160101_quote.zip</Key>

<RequestId>E2100115483F6C63</RequestId>

<HostId>

MeyX9iEyAGFUb9y46KMhm9CaNSuqudCTBB9bXmkYIZidyeT2Z/djE6Al+eXp6KlTFQGv7EIsh+w=

</HostId>

</Error>

Loudk

109 ,

eurusd

20170101_quote.zip

Jared Broad

A few of those dates might be corrupted & we'll fix them shortly. The others are new years day and christmas eve -- meaning you're probably using an old version of the market hours database. If you update it those dates will be registered as holidays and it won't ask for the data.

Loudk

109 ,

Hi, i installed Lean Master on 13-Apr hence should have the latest market hours database ? I have crossed check with the file in Github... May i know how to go about updating the version ?

i resides in Singapore, not sure if there's other settings that i need to do ?

Thanks.

Jared Broad

Hi loudk - check out this guide about keeping your local LEAN up to date:

https://github.com/QuantConnect/Lean/blob/master/CONTRIBUTING.md#keeping-your-master-up-to-date