Securities

Filtering Data

Introduction

Unfiltered raw data can be faulty for a number of reasons, including invalid data entry. Moreover, high-frequency traders can deploy bait-and-switch strategies by submitting bait orders to deceive other market participants, making raw data noisy and untradeable. To avoid messing up with our trading logic and model training, you can filter out suspicious raw data with a data filter.

Set Models

To set a data filter for a security, call the SetDataFilterset_data_filter property on the Security object.

// Use the SetDataFilter method to use the SecurityDataFilter on the SPY ETF data.
var spy = AddEquity("SPY");
spy.SetDataFilter(new SecurityDataFilter());
# Use the set_data_filter method to use the SecurityDataFilter on the SPY ETF data.
spy = self.add_equity("SPY")
spy.set_data_filter(SecurityDataFilter())

You can also set the data filter model in a security initializer. If your algorithm has a universe, use the security initializer technique. In order to initialize single security subscriptions with the security initializer, call AddSecurityInitializeradd_security_initializer before you create the subscriptions.

public class AddSecurityInitializerExampleAlgorithm : QCAlgorithm
{
    public override void Initialize()
    {
        // In the Initialize method, set the security initializer to set models of assets.
        AddSecurityInitializer(CustomSecurityInitializer);
    }

    private void CustomSecurityInitializer(Security security)
    {
        // Overwrite some of the reality models
        security.SetDataFilter(new SecurityDataFilter());
    }
}
class AddSecurityInitializerExampleAlgorithm(QCAlgorithm):
    def initialize(self) -> None:
        # In the Initialize method, set the security initializer to set models of assets.
        self.add_security_initializer(self._custom_security_initializer)

    def _custom_security_initializer(self, security: Security) -> None:
        # Overwrite some of the reality models        
        security.set_data_filter(SecurityDataFilter())

Default Behavior

The following table shows the default data filter for each security type:

Security TypeDefault Filter
EquityEquityDataFilter
OptionOptionDataFilter
ForexForexDataFilter
IndexIndexDataFilter
CfdCfdDataFilter
OthersSecurityDataFilter

None of the preceding filters filter out any data.

Model Structure

Data filtering models should implement the ISecurityDataFilter interface. Extensions of the ISecurityDataFilter interface must implement the Filter method, which receives Security and BaseData objects and then returns a boolean object that represents if the data point should be filtered out.

Data filtering models must implement a filter method, which receives Security and BaseData objects and then returns a boolean object that represents if the data point should be filtered out.

// Include or exclude a data point from the algorithm.
public class MyDataFilter : ISecurityDataFilter
{
    public override bool Filter(Security vehicle, BaseData data)
    {
        return true;
    }
}
# Include or exclude a data point from the algorithm.
class MyDataFilter(SecurityDataFilter):
    def filter(self, vehicle: Security, data: BaseData) -> bool:
        return True

Examples

The following examples demonstrate some common practices for filtering data.

Example 1: Filter Out Outliers

When analyzing high-frequency price data, it's important to filter out potential outliers and anomalies that may skew the analysis. One effective method is to use simple moving average (SMA) and standard deviation indicators to identify ticks that significantly deviate from the short-term trend. By comparing each tick to the indicator values, you can flag any data points that fall outside a threshold (for example, three standard deviations). This filtration process removes suspicious or erroneous price information from entering your algorithm, ensuring a cleaner dataset for trading. 

public class CustomDataFilterAlgorithm : QCAlgorithm
{
    public override void Initialize()
    {
        SetStartDate(2024, 9, 1);
        SetEndDate(2024, 9, 3);
        dynamic equity = AddEquity("AAPL", Resolution.Tick);
        // Create the indicators.
        equity.Sma = SMA(equity.Symbol, 100);
        equity.Std = IndicatorExtensions.Of(new StandardDeviation(100), equity.Sma, true);
        // Set the data filter.
        equity.SetDataFilter(new CustomDataFilter());
    }
}

class CustomDataFilter : SecurityDataFilter
{
    public CustomDataFilter() : base() { }
    public override bool Filter(Security vehicle, BaseData data)
    {
        // Wait until the indicators are ready.
        var security = vehicle as dynamic;
        if (!(security.Sma.IsReady && security.Std.Window.IsReady))
        {
            return true; // Keep the data point.
        }
        // Check if the current value is within 3 standard deviations of the mean.
        // Return true (keep) or false (discard).
        var sma = security.Sma.Current.Value;
        var std = security.Std.Current.Value;
        return sma - 3m*std <= data.Value && data.Value <= sma + 3m*std;
    }
}
class CustomDataFilterAlgorithm(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2024, 9, 1)
        self.set_end_date(2024, 9, 3)
        equity = self.add_equity("AAPL", Resolution.TICK)
        # Create the indicators.
        equity.sma = self.sma(equity.symbol, 100)
        equity.std = IndicatorExtensions.of(StandardDeviation(100), equity.sma, True)
        # Set the data filter.
        equity.set_data_filter(CustomDataFilter())


class CustomDataFilter(SecurityDataFilter):

    def filter(self, vehicle: Security, data: BaseData) -> bool:
        # Wait until the indicators are ready.
        security = vehicle
        if not (security.sma.is_ready and security.std.is_ready):
            return True # Keep the data point.
        # Check if the current value is within 3 standard deviations of the mean.
        # Return True (keep) or False (discard).
        sma = security.sma.current.value
        std = security.std.current.value
        return sma - 3*std <= data.value <= sma + 3*std

Example 2: Filter Out Major Exchanges

When you trade illiquid financial instruments, it can be advantageous to focus on the BATS exchange since its quote data may not fully reflect the fair market value. Due to the lower trading volume and visibility of BATS, the quotes there may lag behind the true value of illiquid assets. A carefully designed algorithm can analyze the BATS feed to identify situations where the quotes appears to be undervalued compared to the asset's intrinsic worth. By executing trades to capture this disconnect, rather than arbitraging between exchanges, you may be able to profit from the market inefficiencies present in the less liquid instrument. The following example demonstrates how to only consume data from the BATS exchange: 

public class BatsDataFilterAlgorithm : QCAlgorithm
{
    public override void Initialize()
    {
        SetStartDate(2024, 9, 1);
        SetEndDate(2024, 9, 3);
        dynamic equity = AddEquity("AAPL", Resolution.Tick);
        // Set the data filter.
        equity.SetDataFilter(new BatsDataFilter());
    }
}

class BatsDataFilter : SecurityDataFilter
{
    public BatsDataFilter() : base() { }
    public override bool Filter(Security vehicle, BaseData data)
    {
        // Get the tick object.
        var tick = data as Tick;
        // Return true (keep) or false (discard).
        return tick != null && tick.Exchange == Exchange.BATS;
    }
}
class BatsDataFilterAlgorithm(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2024, 9, 1)
        self.set_end_date(2024, 9, 3)
        equity = self.add_equity("AAPL", Resolution.TICK)
        # Set the data filter.
        equity.set_data_filter(BatsDataFilter())


class BatsDataFilter(SecurityDataFilter):

    def filter(self, vehicle: Security, data: BaseData) -> bool:
        # Get the tick object.
        tick = Tick(data)
        # Return True (keep) or False (discard).
        return tick and tick.exchange == Exchange.BATS.name

Example 3: Perturb Prices for Robustness Tests

A data filter can edit a data point instead of just accepting or rejecting it. LEAN applies the filter before it assembles the Slice object and then passes the same data point to the Security objects, the consolidators, and your OnDataon_data method, so an edit inside the filter reaches every part of your algorithm, including the indicators that you create with the automatic indicator helper methods. The following example multiplies each bar by a small random shock so that you can re-run a backtest across many synthetic price paths and measure how much of your performance depends on the exact price history. It derives the shock from the bar end time so that the trade and quote subscriptions of the security apply the same multiplier to the same bar, and it applies that single positive multiplier to all the price fields, which preserves the relationship between the open, high, low, and close prices and between the bid and ask prices. To generate a different price path, change the seed parameter and run the backtest again. History requests don't pass through the data filter, so they return the original prices. 

public class PriceNoiseFilterAlgorithm : QCAlgorithm
{
    private ExponentialMovingAverage _ema;

    public override void Initialize()
    {
        SetStartDate(2024, 9, 3);
        SetEndDate(2024, 9, 5);
        var equity = AddEquity("AAPL", Resolution.Minute);
        // Set the data filter. Change the seed to get a different price path.
        var seed = GetParameter("seed", 1);
        var sigma = GetParameter("sigma", 0.0005);
        equity.SetDataFilter(new PriceNoiseFilter(seed, sigma));
        // The filter perturbs the data before the consolidators update it, so this
        // indicator is built from the perturbed prices.
        _ema = EMA(equity.Symbol, 20, Resolution.Minute);
    }
}

class PriceNoiseFilter : SecurityDataFilter
{
    private readonly int _seed;
    private readonly double _sigma;

    public PriceNoiseFilter(int seed, double sigma) : base()
    {
        _seed = seed;
        _sigma = sigma;
    }

    public override bool Filter(Security vehicle, BaseData data)
    {
        // Seed on the bar time so that the trade and quote subscriptions
        // apply the same shock to the same bar.
        var rng = new Random(_seed ^ data.EndTime.GetHashCode());
        // Draw a standard normal value with the Box-Muller transform.
        var z = Math.Sqrt(-2 * Math.Log(1 - rng.NextDouble())) * Math.Sin(2 * Math.PI * rng.NextDouble());
        var shock = (decimal)(1 + z * _sigma);
        if (data is TradeBar tradeBar)
        {
            // The Close setter also updates the Value property.
            tradeBar.Open *= shock;
            tradeBar.High *= shock;
            tradeBar.Low *= shock;
            tradeBar.Close *= shock;
        }
        else if (data is QuoteBar quoteBar)
        {
            foreach (var side in new[] { quoteBar.Bid, quoteBar.Ask })
            {
                if (side == null)
                {
                    continue;
                }
                side.Open *= shock;
                side.High *= shock;
                side.Low *= shock;
                side.Close *= shock;
            }
            // The Value property of a QuoteBar isn't derived from the Bid and Ask
            // properties, so scale it explicitly.
            quoteBar.Value *= shock;
        }
        // Return true (keep) or false (discard).
        return true;
    }
}
import random


class PriceNoiseFilterAlgorithm(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2024, 9, 3)
        self.set_end_date(2024, 9, 5)
        equity = self.add_equity("AAPL", Resolution.MINUTE)
        # Set the data filter. Change the seed to get a different price path.
        seed = self.get_parameter("seed", 1)
        sigma = self.get_parameter("sigma", 0.0005)
        equity.set_data_filter(PriceNoiseFilter(seed, sigma))
        # The filter perturbs the data before the consolidators update it, so this
        # indicator is built from the perturbed prices.
        self._ema = self.ema(equity.symbol, 20, Resolution.MINUTE)


class PriceNoiseFilter(SecurityDataFilter):

    def __init__(self, seed: int, sigma: float) -> None:
        super().__init__()
        self._seed = seed
        self._sigma = sigma

    def filter(self, vehicle: Security, data: BaseData) -> bool:
        # Seed on the bar time so that the trade and quote subscriptions
        # apply the same shock to the same bar.
        rng = random.Random(f"{self._seed}:{data.end_time}")
        shock = 1 + rng.gauss(0, self._sigma)
        if isinstance(data, TradeBar):
            # The close setter also updates the value property.
            data.open *= shock
            data.high *= shock
            data.low *= shock
            data.close *= shock
        elif isinstance(data, QuoteBar):
            for side in [data.bid, data.ask]:
                if side:
                    side.open *= shock
                    side.high *= shock
                    side.low *= shock
                    side.close *= shock
            # The value property of a QuoteBar isn't derived from the bid and ask
            # properties, so scale it explicitly.
            data.value *= shock
        # Return True (keep) or False (discard).
        return True

Other Examples

For more examples, see the following algorithms:

Demonstration Algorithms
CustomSecurityDataFilterRegressionAlgorithm.py Python TickDataFilteringAlgorithm.py Python CustomSecurityDataFilterRegressionAlgorithm.cs C# TickDataFilteringAlgorithm.cs C#

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: