---
title: "Filtering Data"
description: "Unfiltered data may contain errors from faulty entries & deceptive trades. Filter suspicious data to preserve trading logic and model integrity."
canonical: https://www.quantconnect.com/docs/v2/writing-algorithms/securities/filtering-data
markdown: https://www.quantconnect.com/docs/v2/writing-algorithms/securities/filtering-data.md
language: python
source: https://github.com/QuantConnect/Documentation/tree/master/03%20Writing%20Algorithms/03%20Securities/05%20Filtering%20Data
modified: 2026-11-01
index: https://www.quantconnect.com/llms.txt
---

# Filtering Data

[Documentation](https://www.quantconnect.com/docs/v2.md) > [Writing Algorithms](https://www.quantconnect.com/docs/v2/writing-algorithms.md) > Securities > Filtering Data

> Code examples are shown in Python. Append `?lang=csharp` to this URL for the C# version of this page.

## Introduction

Unfiltered raw data can be faulty for a number of reasons, including invalid data entry. Moreover, high-frequency traders can deploy bait-and-switch strategies by submitting bait orders to deceive other market participants, making raw data noisy and untradeable. To avoid messing up with our trading logic and model training, you can filter out suspicious raw data with a data filter.

## Set Models

To set a data filter for a security, call the `set_data_filter` property on the `Security` object.

```python
# Use the set_data_filter method to use the SecurityDataFilter on the SPY ETF data.
spy = self.add_equity("SPY")
spy.set_data_filter(SecurityDataFilter())
```

You can also set the data filter model in a [security initializer](https://www.quantconnect.com/docs/v2/writing-algorithms/initialization.md#07-Set-Security-Initializer). If your algorithm has a universe, use the security initializer technique. In order to initialize single security subscriptions with the security initializer, call `add_security_initializer` before you create the subscriptions.

```python
class AddSecurityInitializerExampleAlgorithm(QCAlgorithm):
    def initialize(self) -> None:
        # In the Initialize method, set the security initializer to set models of assets.
        self.add_security_initializer(self._custom_security_initializer)

    def _custom_security_initializer(self, security: Security) -> None:
        # Overwrite some of the reality models
        security.set_data_filter(SecurityDataFilter())
```

## Default Behavior

The following table shows the default data filter for each security type:

| Security Type | Default Filter |
| --- | --- |
| `Equity` | [EquityDataFilter](https://github.com/QuantConnect/Lean/blob/master/Common/Securities/Equity/EquityDataFilter.cs) |
| `Option` | [OptionDataFilter](https://github.com/QuantConnect/Lean/blob/master/Common/Securities/Option/OptionDataFilter.cs) |
| `Forex` | [ForexDataFilter](https://github.com/QuantConnect/Lean/blob/master/Common/Securities/Forex/ForexDataFilter.cs) |
| `Index` | [IndexDataFilter](https://github.com/QuantConnect/Lean/blob/master/Common/Securities/Index/IndexDataFilter.cs) |
| `Cfd` | [CfdDataFilter](https://github.com/QuantConnect/Lean/blob/master/Common/Securities/Cfd/CfdDataFilter.cs) |
| Others | [SecurityDataFilter](https://github.com/QuantConnect/Lean/blob/master/Common/Securities/SecurityDataFilter.cs) |

None of the preceding filters filter out any data.

## Model Structure

Data filtering models must implement a `filter` method, which receives `Security` and `BaseData` objects and then returns a `boolean` object that represents if the data point should be filtered out.

```python
# Include or exclude a data point from the algorithm.
class MyDataFilter(SecurityDataFilter):
    def filter(self, vehicle: Security, data: BaseData) -> bool:
        return True
```

## Examples

The following examples demonstrate some common practices for filtering data.

### Example 1: Filter Out Outliers

When analyzing high-frequency price data, it's important to filter out potential outliers and anomalies that may skew the analysis. One effective method is to use [simple moving average](https://www.quantconnect.com/docs/v2/writing-algorithms/indicators/supported-indicators/simple-moving-average.md) (SMA) and [standard deviation](https://www.quantconnect.com/docs/v2/writing-algorithms/indicators/supported-indicators/standard-deviation.md) indicators to identify ticks that significantly deviate from the short-term trend. By comparing each tick to the indicator values, you can flag any data points that fall outside a threshold (for example, three standard deviations). This filtration process removes suspicious or erroneous price information from entering your algorithm, ensuring a cleaner dataset for trading.

```python
class CustomDataFilterAlgorithm(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2024, 9, 1)
        self.set_end_date(2024, 9, 3)
        equity = self.add_equity("AAPL", Resolution.TICK)
        # Create the indicators.
        equity.sma = self.sma(equity.symbol, 100)
        equity.std = IndicatorExtensions.of(StandardDeviation(100), equity.sma, True)
        # Set the data filter.
        equity.set_data_filter(CustomDataFilter())

class CustomDataFilter(SecurityDataFilter):

    def filter(self, vehicle: Security, data: BaseData) -> bool:
        # Wait until the indicators are ready.
        security = vehicle
        if not (security.sma.is_ready and security.std.is_ready):
            return True # Keep the data point.
        # Check if the current value is within 3 standard deviations of the mean.
        # Return True (keep) or False (discard).
        sma = security.sma.current.value
        std = security.std.current.value
        return sma - 3*std <= data.value <= sma + 3*std
```

### Example 2: Filter Out Major Exchanges

When you trade illiquid financial instruments, it can be advantageous to focus on the BATS exchange since its quote data may not fully reflect the fair market value. Due to the lower trading volume and visibility of BATS, the quotes there may lag behind the true value of illiquid assets. A carefully designed algorithm can analyze the BATS feed to identify situations where the quotes appears to be undervalued compared to the asset's intrinsic worth. By executing trades to capture this disconnect, rather than arbitraging between exchanges, you may be able to profit from the market inefficiencies present in the less liquid instrument. The following example demonstrates how to only consume data from the BATS exchange:

```python
class BatsDataFilterAlgorithm(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2024, 9, 1)
        self.set_end_date(2024, 9, 3)
        equity = self.add_equity("AAPL", Resolution.TICK)
        # Set the data filter.
        equity.set_data_filter(BatsDataFilter())

class BatsDataFilter(SecurityDataFilter):

    def filter(self, vehicle: Security, data: BaseData) -> bool:
        # Get the tick object.
        tick = Tick(data)
        # Return True (keep) or False (discard).
        return tick and tick.exchange == Exchange.BATS.name
```

### Example 3: Perturb Prices for Robustness Tests

A data filter can edit a data point instead of just accepting or rejecting it. LEAN applies the filter before it assembles the `Slice` object and then passes the same data point to the `Security` objects, the [consolidators](https://www.quantconnect.com/docs/v2/writing-algorithms/consolidating-data/getting-started.md), and your `on_data` method, so an edit inside the filter reaches every part of your algorithm, including the [indicators](https://www.quantconnect.com/docs/v2/writing-algorithms/indicators/key-concepts.md) that you create with the [automatic indicator](https://www.quantconnect.com/docs/v2/writing-algorithms/indicators/automatic-indicators.md) helper methods. The following example multiplies each bar by a small random shock so that you can re-run a backtest across many synthetic price paths and measure how much of your performance depends on the exact price history. It derives the shock from the bar end time so that the trade and quote subscriptions of the security apply the same multiplier to the same bar, and it applies that single positive multiplier to all the price fields, which preserves the relationship between the open, high, low, and close prices and between the bid and ask prices. To generate a different price path, change the `seed` parameter and run the backtest again. [History requests](https://www.quantconnect.com/docs/v2/writing-algorithms/historical-data/history-requests.md) don't pass through the data filter, so they return the original prices.

```python
import random

class PriceNoiseFilterAlgorithm(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2024, 9, 3)
        self.set_end_date(2024, 9, 5)
        equity = self.add_equity("AAPL", Resolution.MINUTE)
        # Set the data filter. Change the seed to get a different price path.
        seed = self.get_parameter("seed", 1)
        sigma = self.get_parameter("sigma", 0.0005)
        equity.set_data_filter(PriceNoiseFilter(seed, sigma))
        # The filter perturbs the data before the consolidators update it, so this
        # indicator is built from the perturbed prices.
        self._ema = self.ema(equity.symbol, 20, Resolution.MINUTE)

class PriceNoiseFilter(SecurityDataFilter):

    def __init__(self, seed: int, sigma: float) -> None:
        super().__init__()
        self._seed = seed
        self._sigma = sigma

    def filter(self, vehicle: Security, data: BaseData) -> bool:
        # Seed on the bar time so that the trade and quote subscriptions
        # apply the same shock to the same bar.
        rng = random.Random(f"{self._seed}:{data.end_time}")
        shock = 1 + rng.gauss(0, self._sigma)
        if isinstance(data, TradeBar):
            # The close setter also updates the value property.
            data.open *= shock
            data.high *= shock
            data.low *= shock
            data.close *= shock
        elif isinstance(data, QuoteBar):
            for side in [data.bid, data.ask]:
                if side:
                    side.open *= shock
                    side.high *= shock
                    side.low *= shock
                    side.close *= shock
            # The value property of a QuoteBar isn't derived from the bid and ask
            # properties, so scale it explicitly.
            data.value *= shock
        # Return True (keep) or False (discard).
        return True
```

### Other Examples

For more examples, see the following algorithms:

**Demonstration Algorithms**

- [CustomSecurityDataFilterRegressionAlgorithm.py](https://github.com/QuantConnect/Lean/blob/master/Algorithm.Python/CustomSecurityDataFilterRegressionAlgorithm.py) (Python)
- [TickDataFilteringAlgorithm.py](https://github.com/QuantConnect/Lean/blob/master/Algorithm.Python/TickDataFilteringAlgorithm.py) (Python)