Data Issues

Create Data Issue

Introduction

Report an issue with a dataset on the Data Issues page.

Description

Every report must include a reproducible case, so our Data Team can reproduce the issue. Attach the project of a backtest or notebook that shows the issue with the select-project property and the backtest with the select-backtest property, or include a code snippet in the content property. Follow the best practices when you write the report. Before you report an issue, use the /community/data-library/issues/list endpoint to check if an open issue already covers the same ticker and resolution.

Request

The dataset, time window, and description of the issue. The /community/data-library/issues/create API accepts requests in the following format:

CreateDataIssueRequest Model - Request body to report a data issue.
ticker string
required
example: SPY
Ticker of the security with the issue.
security string Enum
required
Asset class of the dataset. Options : ['Equity', 'Equity Option', 'Index', 'Index Option', 'Future', 'Future Option', 'Forex', 'CFD', 'Crypto', 'Crypto Future']
market string
required
example: USA
Market of the dataset. The valid markets depend on the asset class, as shown in the market dropdown on the Data Issues page.
resolution string Enum
required
Resolution of the dataset. The valid resolutions depend on the asset class and market. Options : ['Tick', 'Second', 'Minute', 'Hour', 'Daily', 'Coarse Fundamental', 'Fine Fundamental', 'Margin']
from-date string
required
example: 2024-08-01
First date of the issue, in YYYY-MM-DD format.
to-date string
required
example: 2024-08-02
Last date of the issue, in YYYY-MM-DD format. The end of the time window must be after the start.
from-time string
example: 09:30
Time of day the issue starts, in HH:MM format. Required for Second and Minute resolution when the time window is shorter than 7 days.
to-time string
example: 16:00
Time of day the issue ends, in HH:MM format. Required for Second and Minute resolution when the time window is shorter than 7 days.
corrupted-files string Enum
Include to report files that can't be read. Include at least one issue type. Options : ['Corrupted Files']
missing-points string Enum
Include to report missing data points. Options : ['Missing Points']
missing-files string Enum
Include to report missing files. Options : ['Missing Files']
wrong-values string Enum
Include to report wrong values. Options : ['Wrong Values']
listing-error string Enum
Include to report a listing error. Options : ['Listing Error']
split-error string Enum
Include to report a split error. Options : ['Split Error']
dividend-error string Enum
Include to report a dividend error. Options : ['Dividend Error']
name-mapping-error string Enum
Include to report a ticker mapping error. Options : ['Name Mapping Error']
content string
required
example: SPY data is missing on 2024-08-01.
HTML description of the issue.
select-project integer
Id of one of your projects to attach to the issue, so the Data team can see the code that shows the issue. To get your project Ids, use the /projects/read endpoint.
select-backtest string
Id of a backtest of the attached project to attach to the issue. Requires select-project. To get the backtest Ids of a project, use the /backtests/list endpoint.
Example 
{
  "ticker": "SPY",
  "security": "Equity",
  "market": "USA",
  "resolution": "Tick",
  "from-date": "2024-08-01",
  "to-date": "2024-08-02",
  "from-time": "09:30",
  "to-time": "16:00",
  "corrupted-files": "Corrupted Files",
  "missing-points": "Missing Points",
  "missing-files": "Missing Files",
  "wrong-values": "Wrong Values",
  "listing-error": "Listing Error",
  "split-error": "Split Error",
  "dividend-error": "Dividend Error",
  "name-mapping-error": "Name Mapping Error",
  "content": "SPY data is missing on 2024-08-01.",
  "select-project": 0,
  "select-backtest": "string"
}

Responses

The /community/data-library/issues/create API provides a response in the following format:

200 Success

CreateDataIssueResponse Model - Response from reporting a data issue.
discussionId integer
Id of the new data issue. The issue page is https://www.quantconnect.com/datasets/issue/{discussionId}.
success boolean
Indicate if the API request was successful.
errors string Array
List of errors with the API call.
Example 
{
  "discussionId": 0,
  "success": true,
  "errors": [
    "string"
  ]
}

401 Authentication Error

UnauthorizedError Model - Unauthorized response from the API. Key is missing, invalid, or timestamp is too old for hash.
www_authenticate string
Header

Examples

The following example demonstrates listing the open data issues for a ticker and reporting a new issue with an attached project and backtest when none exists through the cloud API.

from base64 import b64encode
from hashlib import sha256
from time import time
from requests import get, post
BASE_URL = 'https://www.quantconnect.com/api/v2/'

# You need to replace these with your actual credentials.
# You can request your credentials at https://www.quantconnect.com/settings/
# You can find our organization ID at https://www.quantconnect.com/organization/ 
USER_ID = 0
API_TOKEN = '____'
ORGANIZATION_ID = '____'

def get_headers():
    # Get timestamp
    timestamp = f'{int(time())}'
    time_stamped_token = f'{API_TOKEN}:{timestamp}'.encode('utf-8')

    # Get hased API token
    hashed_token = sha256(time_stamped_token).hexdigest()
    authentication = f'{USER_ID}:{hashed_token}'.encode('utf-8')
    authentication = b64encode(authentication).decode('ascii')

    # Create headers dictionary.
    return {
        'Authorization': f'Basic {authentication}',
        'Timestamp': timestamp
    }

# Authenticate to verify credentials
response = post(f'{BASE_URL}/authenticate', headers = get_headers())
print(response.json())

# --------------------


### List Data Issues
# Send a POST request to the /community/data-library/issues/list endpoint to find open issues for SPY
response = post(f'{BASE_URL}/community/data-library/issues/list', headers=get_headers(), json={
    "status": "open",  # open or closed
    "start": 0,        # Index of the first issue to return
    "end": 20,         # Index after the last issue to return
    "search": "SPY"    # Exact ticker match
})
# Parse the JSON response into python managable dict
result = response.json()
# Keep the open issues that already cover SPY minute data
existing = [i for i in result.get('results', []) if i['resolution'] == 'Minute']
# Check if the request was successful and print the issues
if result['success']:
    print("Open Data Issues:")
    print(result)

### List Backtests
# Send a POST request to the /backtests/list endpoint to find a backtest that shows the issue
project_id = 12345678  # Replace with the Id of your project
response = post(f'{BASE_URL}/backtests/list', headers=get_headers(), json={
    "projectId": project_id,
    "includeStatistics": False
})
# Parse the JSON response into python managable dict
result = response.json()
# Pick the first backtest of the project to attach to the issue
backtest_id = result['backtests'][0]['backtestId'] if result.get('backtests') else None

### Create Data Issue (only when no open issue covers it)
if not existing:
    # Send a POST request to the /community/data-library/issues/create endpoint to report the issue
    response = post(f'{BASE_URL}/community/data-library/issues/create', headers=get_headers(), json={
        "ticker": "SPY",
        "security": "Equity",
        "market": "USA",
        "resolution": "Minute",
        "from-date": "2024-08-01",
        "to-date": "2024-08-01",
        "from-time": "09:30",                 # Minute data over less than 7 days needs times
        "to-time": "16:00",
        "missing-points": "Missing Points",  # At least one issue type
        "content": "SPY minute bars are missing on 2024-08-01 between 09:30 and 16:00.",
        "select-project": project_id,         # Optional. Project that shows the issue
        "select-backtest": backtest_id        # Optional. Backtest of that project
    })
    # Parse the JSON response into python managable dict
    result = response.json()
    # Check if the request was successful and print the link to the issue
    if result['success']:
        print(f"https://www.quantconnect.com/datasets/issue/{result['discussionId']}")

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: