Open Source

Brokerage Development Guide

Brokerage Development Guide

Updated July 14th, 2018: The Brokerage Development Guide is an active work in progress.

Introduction

Creating a fully supported brokerage is a challenging endeavor. LEAN requires a number of individual pieces which together form a complete brokerage implementation. This guide aims to describe in as much detail as possible what you need to do for each module. There are nine key components to implement - these are described individually in the sections below.

Each submitted brokerage pull-request must pass all tests before being merged. Partially completed brokerage implementations are acceptable if merged to a branch. It's easy to fall behind master so be sure to keep your branch updated with the master branch. Please make sure to read LEAN's coding style guidelines and comply with the code commenting and design standards.

Guide Contents

Implementation Steps
1.Laying a Foundation (IBrokerageFactory)
Stub out the implementation and initialize a brokerage instance.
2.Creating The Brokerage (IBrokerage)
Installing key brokerage application logic, where possible using a brokerage SDK.
3.Translating Symbol Conventions (ISymbolMapper)
Translate brokerage specific tickers to LEAN format for a uniform algorithm design experience.
4.Describe Broker Limitations (IBrokerageModel)
Describe brokerage support of orders and set transaction models.
5.Enable Live Data Streaming (IDataQueueHandler)
Live streaming data service from brokerage supplied source.
6.Enable Serving Historical Data (IHistoryProvider)
Tap into the brokerage historical data API to serve history for live algorithms.
7.Download Data (IDataDownloader)
Save data from the brokerage to disk in LEAN format.
8.Describe Brokerage Fee Structures (IFeeModel)
Enable accurate backtesting with specific fee structures of the brokerage.
9.Update Algorithm API for Easy Setup of Brokerage Models (ISecurityTransactionModel)
Combine the various models together to form a brokerage set.

The root of the brokerage system is the algorithm job packets. These hold configuration information about how to run LEAN. The program logic is a little convoluted; it moves from config.json > create job packet > create brokerage factory matching name > set job packet brokerage data > factory creates brokerage instance. Because of this we'll start creating a brokerage at the root -- the configuration and brokerage factory...

Step 1: Laying a Foundation

IBrokerageFactory
Primary RoleCreate and initialize a brokerage instance.
InterfaceIBrokerageFactory.cs
ExampleGDAXBrokerageFactory.cs
Target LocationIn Brokerages Folder in Brokerages Solution

The IBrokerageFactory creates brokerage instances with a Job Packet which configures LEAN. It contains the name of the selected brokerage to create, which is used to create the right BrokerageFactory type. The configuration live-mode-brokerage key is used to set the brokerage name.

1.1 Create Authentication Configuration

In the configuration file add a few key-values with your brokerage configuration information. This will be used for most local debugging and testing as the default. E.g. oanda-access-token and oanda-account-id. These will be copied to the job packet which contains a matching field BrokerageData. This is a dictionary of <string,string> pairs.

1.2 Create Brokerage Data

By default the IBrokerageFactory.BrokerageData implementation should load all required configuration from the config file using the Config class. E.g. Config.Get("oanda-access-token"). This can be a simple pass through to the config for most brokerages.

1.3 Create Brokerage Model

Brokerage Models tell LEAN what order types a brokerage supports, whether we're allowed to update an order, and what transaction models to use for fills. It is important to do but something we can come back to later. For now, we should just create a stub implementation which we'll extend and improve later. This file MyBrokerageBrokerageModel.cs lives in the /Common/Brokerages folder. For now, you can make it an empty implementation inheriting from the DefaultBrokerageModel. See this example of a partially implemented model here. Set your empty placeholder model to the BrokerageModel property.

1.4 Create Brokerage Instance in Factory

The Brokerage Factory uses a job packet to create an initialized brokerage instance. This happens in the CreateBrokerage() method. You should assume the job has the best source of data not the class BrokerageData property. The BrokerageData property on the factory are the starting default values sourced from config which can be overridden by a runtime job.

Given our IBrokerage implementation hasn't been started yet let's make a placeholder file for our brokerage with the methods stubbed out: MyBrokerage.cs in the MyBrokerage folder, in the Brokerages solution. All the methods should throw a new NotImplementedException() for now except for the constructor which should save any required authentication data to private variables. The CreateBrokerage() method should create a brokerage object but not connect to the brokerage. The connection is done later in the LEAN start-up process.

1.5 Create Configuration Environment

In the config file LEAN has helper environments which group configuration flags together and override the root configuration values. You should make a mybrokerage-live environment for your brokerage which specifies the brokerage type name for live-mode-brokerage. You should copy the paper-trading brokerage setup to start. You should set the environment value to your new brokerage environment for testing.

Tip:

In the IBrokerageFactory examples, you'll see code like this: Composer.Instance.AddPart<IDataQueueHandler>(dataQueueHandler), which is adding parts to the "Composer". The Composer is a system in LEAN for loading types dynamically. In this case, it is adding an instance of the DataQueueHandler for the brokerage to the composer. You can think of the composer as a library, and adding parts is like adding books to its collection. But we'll come back to this later...

Stage 1: Checklist
Configuration keys and placeholder values for brokerage authentication requirements.
Created folder for brokerage in Brokerages solution; with MyBrokerageFactory.cs.
Implemented all interfaces of the BrokerageFactory (some with stub implementations).
Create a stub MyBrokerage.cs with Not Implemented exceptions.
Create a stub MyBrokerageBrokerageModel.cs inheriting from DefaultBrokerageModel.
Created a mybrokerage live configuration environment specifying your class.
Set the environment configuration to your new brokerage environment.
Action:

Build the solution. Although running won't work the stub implementations should still build.

Step 2: Creating the Brokerage

IBrokerage
Primary RoleBrokerage connection, orders and fill events.
InterfaceIBrokerage.cs
ExampleGDAXBrokerage.cs
Target LocationQuantConnect.Brokerages.sln

The IBrokerage holds the bulk of the core logic responsible for running the brokerage implementation. Many smaller models described later use the Brokerage implementation internally so its best to start implementation of the IBrokerage now. Brokerage classes can get quite large so you should use a partial class modifier to break up the files in appropriate categories. It has many important roles vital for the stability of a running algorithm. These include:

  1. Maintain Connection - Connect and maintain connection while algorithm running.
  2. Setup State - Initialize the algorithm portfolio, open orders and cashbook.
  3. Order Operations - Create, update and cancel orders.
  4. Order Events - Receive order fills and apply them to portfolio.
  5. Account Events - Track non-order events (cash deposits/removals).
  6. Brokerage Events - Interpret brokerage messages and act when required.
  7. Serve History Requests - Provide historical data on request.

Often brokerages will have their own ticker styles, order class names, event names. Many of the methods in the brokerage implementation may simply be converting from the brokerage object format into LEAN format. You should plan accordingly to write neat code.

The brokerage will eventually implement 3 interfaces:

class MyBrokerage : IBrokerage, IDataDownloader, IHistoryProvider { ... }

Implementation Style. This guide will focus mostly on implementing the brokerage step by step in LEAN; as its a more natural workflow for most people. You can also follow a more test-driven-development process by following the test harness. To do this create a new test class which extends from the base class in /Tests/Brokerages/BrokerageTests.cs. This test-framework tests all the methods for an IBrokerage implementation.

2.1 Brokerage Requirements

QuantConnect is best used with streaming or socket based brokerage connections. Streaming brokerage implementations allow for the easiest translation of broker events into LEAN events. Without streaming order-events you will need to poll for to check for fills. In our experience this is fraught with additional risks and challenges.

2.2 Installing a Library

Most brokerages will provide a wrapper for their API. You should use this where possible as long as it has a permissive license. Although it is technically possible to embed an external github repository we've elected to not do this to make LEAN easier to install (submodules can be tricky for beginners). You should copy the library into its own subfolder of the brokerage implementation: /Brokerages/MyBrokerage/BrokerLib/*.

LEAN Open-Source. If you copy and paste code from an external source leave the comments and headers intact, and if they do not have a comment header be sure to add one to each file referencing the source. Let's keep the attributions in place.

Tip:

Libraries will need to be .NET Framework 4.6.2 compatible as LEAN is fully cross-platform via Mono.

Action:

Build the project again to make sure the library is compiling successfully. Its good to make sure your library is integrated successfully before continuing.

2.4 IBrokerage Class Implementation

  • Base Class

    Using a base class is optional but allows you to reuse event methods we have provided. The Brokerage object implements these event handlers and marks the remaining items as abstract.

    LEAN provides an optional base class BaseWebsocketsBrokerage which seeks to connect and maintain a socket connection and pass messages to an event handler. As each socket connection is different consider carefully before using this class; it might be easier and more maintainable to simply maintain your own socket connection.

    Tip:

    Brush up on the partial class keyword. This will help you break-up your class later.

  • Class Constructor

    Once the scaffolding brokerage methods are in place (overrides of the abstract base classes); you can focus on the class constructor. Here if you are using a brokerage SDK you can create a new instance of their library, and store it to a class variable for later use. You should make your constructor take all the arguments it needs. This is passed to you from code you implemented in the previous section, the IBrokerageFactory.

    Class Constructor Examples
    Interactive BrokersLaunching external process to create brokerage.
    FXCMSimple example; saving class variables and setting defaults.
    OANDACreating SDK instance and assigning internal event handlers.
    GDAXOffload constructor work to BrokerageFactory and use BaseWebsocketBrokerage base class.
  • string Name

    The Name property is a human readable brokerage name for debugging and logging. For US Equity regulated brokerages convention states this name generally ends in the word "Brokerage".

  • void Connect()

    The Connect method triggers logic for establishing a link to your brokerage. Normally we do not do this in the constructor as it would make algorithms and brokerages die in the BrokerageFactory process. For most brokerages this will be simply calling the connect method on your SDK library.

    Connection Method Examples
    Interactive BrokersConnecting to external process with brokerage SDK.
    FXCMUsing SDK to create event handlers and connection thread.
    OANDASimple example, calling SDK.
    GDAXEstablish websocket connection and monitoring in a thread.

    Exceptions

    In the event of a soft failure like internet connection lost, or server 502 errors you should create a new BrokerageMessageEvent. By doing this you allow the algorithm to handle the brokerage messages. For example: Interactive Brokers resets socket connections at different times globally, so users in other parts of the world can get disconnected at strange times of the day. Knowing this, they may elect to have their algorithm ignore specific disconnection attempts.

    In the event of a hard-failure such as the password being incorrect or an unsupported API method you should throw a real exception with details of the error. You can see examples of using both of these techniques in the FXCM Connect method.

  • void Disconnect()

    Disconnect is called at the end of the algorithm before shutting down LEAN.

  • bool IsConnected

    A boolean property to indicate the state of the brokerage connection. Depending on your connection style this may be automatically handled for you and simply require passing back the value from your SDK. Alternatively you may need to maintain your own connection state flag in your brokerage.

  • bool PlaceOrder(Order order)

    Send a new LEAN order to the brokerage and report back the success or failure. The PlaceOrder method accepts a generic Order object which is the base class for all order types. The first step of placing an order is often to convert it from LEAN format into the Brokerage SDK required format. Brokerage implementations should aim to support as many LEAN order types as possible. There may be other order types in the brokerage but implementing them is considered out of scope of a rev-0 brokerage implementation.

    Once the order type is converted you should send it to your brokerage submit API. Often you will receive an immediate reply indicating the order was successfully placed.

    Part of the order conversion might be converting the brokerage ticker (e.g. LEAN name "EURUSD", OANDA name is "EUR/USD"). This is done with a BrokerageSymbolMapper class. We'll get to that in Step 3, for now you can pass a request for the brokerage ticker to the stub implementation.

    The PlaceOrder() method returns true when the order is accepted to the brokerage. If it is invalid, immediately rejected or there is an internet outage you can return false.

    Tip:

    Create an internal method: BrokerOrder ConvertOrder(Order order) for conversion to brokerage orders. You will use this over and over again. Converting order types is an error prone process and its recommended to carefully review each order after you've completed a port. Remember to check each required brokerage order property, brokers like InteractiveBrokers have hundreds of properties.

    Tip:

    Use the IsConnected boolean to check if you're connected before placing the order. If not throw an exception to halt the algorithm.

  • bool UpdateOrder(Order order)

    Transmit an update request to the API and return true if it was successfully processed. There are no simple examples here! Updating an order is one of the most tricky part of the brokerage implementations. You can easily run into synchronization issues.

    Update Order Examples
    Interactive BrokersExternal application, updating multiple asset classes.
    FXCMRelatively simple example via async brokerage callback.
    OANDA v1, v2Complex example with 2 async APIs, multiple parameters.
    GDAXUpdating isn't supported; throw an exception.
  • bool CancelOrder(Order order)

  • bool UpdateOrder(Order order)

  • List<Order> GetOpenOrders()

  • List<Holding> GetAccountHoldings()

  • List<Cash> GetCashBalance()

  • bool AccountInstantlyUpdated

  • IEnumerable<BaseData> GetHistory(HistoryRequest request)

  • bool AccountInstantlyUpdated

Step 3: Translating Symbol Conventions

Step 4: Describe Brokerage Limitations

Step 5: Enable Live Data Streaming

Step 6: Enable Serving Historical Data

Step 7: Downloading Data

Step 8: Brokerage Fee Structures

Step 9: Updating Algorithm API

You can also see our Documentation and Videos. You can also get in touch with us via Chat.

Did you find this page Helpful ?