Securities

Filtering Data

Introduction

Unfiltered raw data can be faulty for a number of reasons, including invalid data entry. Moreover, high-frequency traders can deploy bait-and-switch strategies by submitting bait orders to deceive other market participants, making raw data noisy and untradeable. To avoid messing up with our trading logic and model training, you can filter out suspicious raw data with a data filter.

Set Models

To set a data filter for a security, call the SetDataFilterset_data_filter property on the Security object.

// In Initialize
var spy = AddEquity("SPY");
spy.SetDataFilter(new SecurityDataFilter());
# In Initialize
spy = self.add_equity("SPY")
spy.set_data_filter(SecurityDataFilter())

You can also set the data filter model in a security initializer. If your algorithm has a universe, use the security initializer technique. In order to initialize single security subscriptions with the security initializer, call SetSecurityInitializerset_security_initializer before you create the subscriptions.

// In Initialize
SetSecurityInitializer(new MySecurityInitializer(BrokerageModel, new FuncSecuritySeeder(GetLastKnownPrices)));

// Outside of the algorithm class
class MySecurityInitializer : BrokerageModelSecurityInitializer
{
    public MySecurityInitializer(IBrokerageModel brokerageModel, ISecuritySeeder securitySeeder)
        : base(brokerageModel, securitySeeder) {}    
    
    public override void Initialize(Security security)
    {
        // First, call the superclass definition
        // This method sets the reality models of each security using the default reality models of the brokerage model
        base.Initialize(security);

        // Next, overwrite some of the reality models        
        security.SetDataFilter(new SecurityDataFilter());    
    }
}
# In Initialize
self.set_security_initializer(MySecurityInitializer(self.brokerage_model, FuncSecuritySeeder(self.get_last_known_prices)))

# Outside of the algorithm class
class MySecurityInitializer(BrokerageModelSecurityInitializer):

    def __init__(self, brokerage_model: IBrokerageModel, security_seeder: ISecuritySeeder) -> None:
        super().__init__(brokerage_model, security_seeder)
    def initialize(self, security: Security) -> None:
        # First, call the superclass definition
        # This method sets the reality models of each security using the default reality models of the brokerage model
        super().initialize(security)

        # Next, overwrite some of the reality models        
        security.set_data_filter(SecurityDataFilter())

Default Behavior

The following table shows the default data filter for each security type:

Security TypeDefault Filter
EquityEquityDataFilter
OptionOptionDataFilter
ForexForexDataFilter
IndexIndexDataFilter
CfdCfdDataFilter
OthersSecurityDataFilter

None of the preceding filters filter out any data.

Model Structure

Data filtering models should implement the ISecurityDataFilter interface. Extensions of the ISecurityDataFilter interface must implement the Filter method, which receives Security and BaseData objects and then returns a boolean object that represents if the data point should be filtered out.

Data filtering models must implement a filter method, which receives Security and BaseData objects and then returns a boolean object that represents if the data point should be filtered out.

public class MyDataFilter : ISecurityDataFilter
{
    public override bool Filter(Security vehicle, BaseData data)
    {
        return true;
    }
}
class MyDataFilter(SecurityDataFilter):
    def filter(self, vehicle: Security, data: BaseData) -> bool:
        return True

Examples

Demonstration Algorithms
CustomSecurityDataFilterRegressionAlgorithm.py Python TickDataFilteringAlgorithm.py Python CustomSecurityDataFilterRegressionAlgorithm.cs C# TickDataFilteringAlgorithm.cs C#

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: