Flat period in GBP data

Here is a complete diagnosis and fix for Chris's issue with the flat GBP futures plot, including explanations and enhanced diagnostics.

from AlgorithmImports import *
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.dates as mdates

qb = QuantBook()

# 1) Add the same continuous future contract as Chris, keeping the same configuration
gbp_future = qb.add_future(
    Futures.Currencies.GBP,
    resolution=Resolution.DAILY,
    data_normalization_mode=DataNormalizationMode.BACKWARDS_PANAMA_CANAL,
    data_mapping_mode=DataMappingMode.OPEN_INTEREST,
    contract_depth_offset=0,
    extended_market_hours=True
)

# Define the start and end dates for the historical data
start_date = datetime(2023, 9, 1)
end_date = datetime(2024, 5, 31)

# 2) Get the historical data as a TradeBar dataframe
history = qb.history(TradeBar, gbp_future.symbol, start_date, end_date, Resolution.DAILY)

# 3) Print out structure and index info for transparency
print("================ DATAFRAME STRUCTURE ================")
print(history.head())
print("\nIndex Type:", type(history.index))
print("Index Names:", history.index.names)
print("Index Sample (first 5):", history.index[:5])
print("DataFrame Shape:", history.shape)

# We'll check if it's a MultiIndex or single-level index
if isinstance(history.index, pd.MultiIndex):
    # Typically, a MultiIndex with levels [Time, Symbol]
    continuous_df = history.xs(gbp_future.symbol, level="Symbol")
    print("\nDetected MultiIndex. Extracted continuous contract slice using xs(...).")
else:
    continuous_df = history
    print("\nDetected single-level index. Proceeding with direct usage.")

# 4) Show basic stats for the close column
close_series = continuous_df["close"]
print("\n================ CLOSE COLUMN STATISTICS ================")
print("Min Close:", close_series.min())
print("Max Close:", close_series.max())
print("Number of unique Close values:", close_series.nunique())
print("NaN count:", close_series.isna().sum())

# 5) Identify any periods with unchanged close prices (>5 consecutive days) or with gaps/NaNs
nan_locations = close_series[close_series.isna()]
if not nan_locations.empty:
    print("\nNaN Locations:")
    print(nan_locations)
else:
    print("\nNo NaN values found in close prices.")

# Find flatlines >5 consecutive days
changes = (close_series.diff() != 0).cumsum()
run_lengths = changes.value_counts()
flatline_groups = run_lengths[run_lengths > 5].index

if len(flatline_groups) > 0:
    print("\nFlatline runs (>5 consecutive days) detected:")
    for grp in flatline_groups:
        group_indices = changes[changes == grp].index
        print(f"Group {grp} - Size {run_lengths[grp]} - From {group_indices[0]} to {group_indices[-1]}")
else:
    print("\nNo flatline runs of more than 5 consecutive days detected.")

# 6) Plot the properly selected close price series
# Ensure index is a proper datetime type
if not pd.api.types.is_datetime_64_any_dtype(continuous_df.index):
    continuous_df.index = pd.to_datetime(continuous_df.index)

plt.figure(figsize=(10, 6))
plt.plot(continuous_df.index, continuous_df["close"], label="GBP Continuous")
plt.title("GBP Continuous Futures Close")
plt.xlabel("Date")
plt.ylabel("Close Price")
plt.legend()
plt.tight_layout()
plt.show()

print("\n================ NOTES =================")
print(
    "If you see any persistent flatline areas, it may be due to "
    "DataNormalizationMode or DataMappingMode used for continuous futures. "
    "Consider trying DataNormalizationMode.RAW or DataNormalizationMode.BACKWARDS_RATIO "
    "to see if that resolves it."
)

How to use it:

• This script will help Chris diagnose why GBP continuous futures might appear flat by revealing if the issue is due to the way the DataFrame is being accessed or where the flatline/NaN values really are.
• It avoids potentially hazardous hard-coded slices (like ('1899-12-30', '/6B')) and instead inspects the full structure.
• Should you see persistent flatlines, after this diagnostic, try changing the data_normalization_mode parameter to BACKWARDS_RATIO or RAW as a next step.

If you have questions on interpreting the diagnostics, or want an automatic test of alternate normalization modes, let me know!

Sources: Pandas Data Frame History Algorithm, Universes - QuantConnect.com, Futures - QuantConnect.com, Residual Momentum, Individual Contracts - QuantConnect.com

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File /opt/miniconda3/lib/python3.11/site-packages/pandas/core/indexes/multi.py:1542, in MultiIndex._get_level_number(self, level)
  1541 try:
-> 1542     level = self.names.index(level)
  1543 except ValueError as err:

ValueError: 'Symbol' is not in list

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Cell In[1], line 30
    27 # We'll check if it's a MultiIndex or single-level index
    28 if isinstance(history.index, pd.MultiIndex):
    29     # Typically, a MultiIndex with levels [Time, Symbol]
---> 30     continuous_df = history.xs(gbp_future.symbol, level="Symbol")
    31     print("\nDetected MultiIndex. Extracted continuous contract slice using xs(...).")
    32 else:

File /opt/miniconda3/lib/python3.11/site-packages/pandas/core/generic.py:4209, in NDFrame.xs(self, key, axis, level, drop_level)
  4207 if not isinstance(labels, MultiIndex):
  4208     raise TypeError("Index must be a MultiIndex")
-> 4209 loc, new_ax = labels.get_loc_level(key, level=level, drop_level=drop_level)
  4211 # create the tuple of the indexer
  4212 _indexer = [slice(None)] * self.ndim

File /opt/miniconda3/lib/python3.11/site-packages/pandas/core/indexes/multi.py:3031, in MultiIndex.get_loc_level(self, key, level, drop_level)
  2990 """
  2991 Get location and sliced index for requested label(s)/level(s).
  2992 
  (...)
  3028 (1, None)
  3029 """
  3030 if not isinstance(level, (list, tuple)):
-> 3031     level = self._get_level_number(level)
  3032 else:
  3033     level = [self._get_level_number(lev) for lev in level]

File /opt/miniconda3/lib/python3.11/site-packages/pandas/core/indexes/multi.py:1545, in MultiIndex._get_level_number(self, level)
  1543 except ValueError as err:
  1544     if not is_integer(level):
-> 1545         raise KeyError(f"Level {level} not found") from err
  1546     if level < 0:
  1547         level += self.nlevels

KeyError: 'Level Symbol not found'

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[2], line 62
    59     print("\nNo flatline runs of more than 5 consecutive days detected.")
    60 # 6) Plot the properly selected close price series
    61 # Ensure index is a proper datetime type
---> 62 if not pd.api.types.is_datetime_64_any_dtype(continuous_df.index):
    63     continuous_df.index = pd.to_datetime(continuous_df.index)
    64 plt.figure(figsize=(10, 6))

AttributeError: module 'pandas.api.types' has no attribute 'is_datetime_64_any_dtype'

================ DATAFRAME STRUCTURE ================
                                       close    high     low    open   volume
expiry     symbol time                                                        
1899-12-30 /6B    2023-09-01 16:00:00  1.2606  1.2721  1.2595  1.2716  50302.0
                 2023-09-04 16:00:00  1.2646  1.2654  1.2633  1.2652   6453.0
                 2023-09-05 16:00:00  1.2581  1.2604  1.2565  1.2597  30369.0
                 2023-09-06 16:00:00  1.2525  1.2563  1.2500  1.2557  48173.0
                 2023-09-07 16:00:00  1.2489  1.2508  1.2464  1.2485  31470.0

Index Type: <class 'pandas.core.indexes.multi.MultiIndex'>
Index Names: ['expiry', 'symbol', 'time']
Index Sample (first 5): MultiIndex([('1899-12-30', /6B, '2023-09-01 16:00:00'),
           ('1899-12-30', /6B, '2023-09-04 16:00:00'),
           ('1899-12-30', /6B, '2023-09-05 16:00:00'),
           ('1899-12-30', /6B, '2023-09-06 16:00:00'),
           ('1899-12-30', /6B, '2023-09-07 16:00:00')],
          names=['expiry', 'symbol', 'time'])
DataFrame Shape: (192, 5)

Detected MultiIndex. Extracted continuous contract slice using xs(...).

================ CLOSE COLUMN STATISTICS ================
Min Close: 1.2092
Max Close: 1.2889
Number of unique Close values: 147
NaN count: 0

No NaN values found in close prices.

Flatline runs (>5 consecutive days) detected:
Group 72 - Size 21 - From (Timestamp('1899-12-30 00:00:00'), Timestamp('2023-12-15 16:00:00')) to (Timestamp('1899-12-30 00:00:00'), Timestamp('2024-01-16 16:00:00'))

Hi Chris

Please file a data issue if you think it is a data missing issue, and we will investigate it. Thank you!

Best
Louis