Contents
Open Source
Brokerage Development Guide
Introduction
Creating a fully supported brokerage is a challenging endeavor. LEAN requires a number of individual pieces which together form a complete brokerage implementation. This guide aims to describe in as much detail as possible what you need to do for each module. There are nine key components to implement - these are described individually in the sections below.
Each submitted brokerage pull-request must pass all tests before being merged. Partially completed brokerage implementations are acceptable if merged to a branch. It's easy to fall behind master so be sure to keep your branch updated with the master branch. Please make sure to read LEAN's coding style guidelines and comply with the code commenting and design standards.
Guide Contents
Implementation Steps | |
---|---|
1. | Laying a Foundation (IBrokerageFactory) Stub out the implementation and initialize a brokerage instance. |
2. | Creating The Brokerage (IBrokerage) Installing key brokerage application logic, where possible using a brokerage SDK. |
3. | Translating Symbol Conventions (ISymbolMapper) Translate brokerage specific tickers to LEAN format for a uniform algorithm design experience. |
4. | Describe Broker Limitations (IBrokerageModel) Describe brokerage support of orders and set transaction models. |
5. | Enable Live Data Streaming (IDataQueueHandler) Live streaming data service from brokerage supplied source. |
6. | Enable Serving Historical Data (IHistoryProvider) Tap into the brokerage historical data API to serve history for live algorithms. |
7. | Download Data (IDataDownloader) Save data from the brokerage to disk in LEAN format. |
8. | Describe Brokerage Fee Structures (IFeeModel) Enable accurate backtesting with specific fee structures of the brokerage. |
9. | Update Algorithm API for Easy Setup of Brokerage Models (ISecurityTransactionModel) Combine the various models together to form a brokerage set. |
The root of the brokerage system is the algorithm job packets. These hold configuration information about how to run LEAN. The program logic is a little convoluted; it moves from config.json > create job packet > create brokerage factory matching name > set job packet brokerage data > factory creates brokerage instance. Because of this we'll start creating a brokerage at the root -- the configuration and brokerage factory...
Step 1: Laying a Foundation
IBrokerageFactory | |
---|---|
Primary Role | Create and initialize a brokerage instance. |
Interface | IBrokerageFactory.cs |
Example | GDAXBrokerageFactory.cs |
Target Location | In Brokerages Folder in Brokerages Solution |
The IBrokerageFactory creates brokerage instances with a Job Packet which configures LEAN. It contains the name of the selected brokerage to create, which is used to create the right BrokerageFactory type. The configuration live-mode-brokerage
key is used to set the brokerage name.
1.1 Create Authentication Configuration
In the configuration file add a few key-values with your brokerage configuration information. This will be used for most local debugging and testing as the default. E.g. oanda-access-token
and oanda-account-id
. These will be copied to the job packet which contains a matching field BrokerageData. This is a dictionary of <string,string> pairs.
1.2 Create Brokerage Data
By default the IBrokerageFactory.BrokerageData
implementation should load all required configuration from the config file using the Config
class. E.g. Config.Get("oanda-access-token")
. This can be a simple pass through to the config for most brokerages.
1.3 Create Brokerage Model
Brokerage Models tell LEAN what order types a brokerage supports, whether we're allowed to update an order, and what transaction models to use for fills. It is important to do but something we can come back to later. For now, we should just create a stub implementation which we'll extend and improve later. This file MyBrokerageBrokerageModel.cs
lives in the /Common/Brokerages
folder. For now, you can make it an empty implementation inheriting from the DefaultBrokerageModel
. See this example of a partially implemented model here. Set your empty placeholder model to the BrokerageModel property.
1.4 Create Brokerage Instance in Factory
The Brokerage Factory uses a job packet to create an initialized brokerage instance. This happens in the CreateBrokerage()
method. You should assume the job
has the best source of data not the class BrokerageData property. The BrokerageData
property on the factory are the starting default values sourced from config which can be overridden by a runtime job.
Given our IBrokerage implementation hasn't been started yet let's make a placeholder file for our brokerage with the methods stubbed out: MyBrokerage.cs
in the MyBrokerage
folder, in the Brokerages
solution. All the methods should throw a new NotImplementedException()
for now except for the constructor which should save any required authentication data to private variables. The CreateBrokerage()
method should create a brokerage object but not connect to the brokerage. The connection is done later in the LEAN start-up process.
1.5 Create Configuration Environment
In the config file LEAN has helper environments which group configuration flags together and override the root configuration values. You should make a mybrokerage-live
environment for your brokerage which specifies the brokerage type name for live-mode-brokerage
. You should copy the paper-trading brokerage setup to start. You should set the environment
value to your new brokerage environment for testing.
In the IBrokerageFactory
examples, you'll see code like this: Composer.Instance.AddPart<IDataQueueHandler>(dataQueueHandler)
, which is adding parts to the "Composer". The Composer is a system in LEAN for loading types dynamically. In this case, it is adding an instance of the DataQueueHandler for the brokerage to the composer. You can think of the composer as a library, and adding parts is like adding books to its collection. But we'll come back to this later...
Stage 1: Checklist | |
---|---|
Configuration keys and placeholder values for brokerage authentication requirements. | |
Created folder for brokerage in Brokerages solution; with MyBrokerageFactory.cs . | |
Implemented all interfaces of the BrokerageFactory (some with stub implementations). | |
Create a stub MyBrokerage.cs with Not Implemented exceptions. | |
Create a stub MyBrokerageBrokerageModel.cs inheriting from DefaultBrokerageModel. | |
Created a mybrokerage live configuration environment specifying your class. | |
Set the environment configuration to your new brokerage environment. |
Build the solution. Although running won't work the stub implementations should still build.
Step 2: Creating the Brokerage
IBrokerage | |
---|---|
Primary Role | Brokerage connection, orders and fill events. |
Interface | IBrokerage.cs |
Example | GDAXBrokerage.cs |
Target Location | QuantConnect.Brokerages.sln |
The IBrokerage holds the bulk of the core logic responsible for running the brokerage implementation. Many smaller models described later use the Brokerage implementation internally so its best to start implementation of the IBrokerage
now. Brokerage classes can get quite large so you should use a partial
class modifier to break up the files in appropriate categories. It has many important roles vital for the stability of a running algorithm. These include:
- Maintain Connection - Connect and maintain connection while algorithm running.
- Setup State - Initialize the algorithm portfolio, open orders and cashbook.
- Order Operations - Create, update and cancel orders.
- Order Events - Receive order fills and apply them to portfolio.
- Account Events - Track non-order events (cash deposits/removals).
- Brokerage Events - Interpret brokerage messages and act when required.
- Serve History Requests - Provide historical data on request.
Often brokerages will have their own ticker styles, order class names, event names. Many of the methods in the brokerage implementation may simply be converting from the brokerage object format into LEAN format. You should plan accordingly to write neat code.
The brokerage will eventually implement 3 interfaces:
class MyBrokerage : IBrokerage, IDataDownloader, IHistoryProvider { ... }
Implementation Style. This guide will focus mostly on implementing the brokerage step by step in LEAN; as its a more natural workflow for most people. You can also follow a more test-driven-development process by following the test harness. To do this create a new test class which extends from the base class in /Tests/Brokerages/BrokerageTests.cs
. This test-framework tests all the methods for an IBrokerage implementation.
2.1 Brokerage Requirements
QuantConnect is best used with streaming or socket based brokerage connections. Streaming brokerage implementations allow for the easiest translation of broker events into LEAN events. Without streaming order-events you will need to poll for to check for fills. In our experience this is fraught with additional risks and challenges.
2.2 Installing a Library
Most brokerages will provide a wrapper for their API. You should use this where possible as long as it has a permissive license. Although it is technically possible to embed an external github repository we've elected to not do this to make LEAN easier to install (submodules can be tricky for beginners). You should copy the library into its own subfolder of the brokerage implementation: /Brokerages/MyBrokerage/BrokerLib/*
.
LEAN Open-Source. If you copy and paste code from an external source leave the comments and headers intact, and if they do not have a comment header be sure to add one to each file referencing the source. Let's keep the attributions in place.
Libraries will need to be .NET Framework 4.6.2 compatible as LEAN is fully cross-platform via Mono.
Build the project again to make sure the library is compiling successfully. Its good to make sure your library is integrated successfully before continuing.
2.4 IBrokerage Class Implementation
-
Base Class
Using a base class is optional but allows you to reuse event methods we have provided. The
Brokerage
object implements these event handlers and marks the remaining items asabstract
.LEAN provides an optional base class
BaseWebsocketsBrokerage
which seeks to connect and maintain a socket connection and pass messages to an event handler. As each socket connection is different consider carefully before using this class; it might be easier and more maintainable to simply maintain your own socket connection.Tip:Brush up on the
partial
class keyword. This will help you break-up your class later. -
Class Constructor
Once the scaffolding brokerage methods are in place (overrides of the abstract base classes); you can focus on the class constructor. Here if you are using a brokerage SDK you can create a new instance of their library, and store it to a class variable for later use. You should make your constructor take all the arguments it needs. This is passed to you from code you implemented in the previous section, the
IBrokerageFactory
.Class Constructor Examples Interactive Brokers Launching external process to create brokerage. FXCM Simple example; saving class variables and setting defaults. OANDA Creating SDK instance and assigning internal event handlers. GDAX Offload constructor work to BrokerageFactory and use BaseWebsocketBrokerage
base class. -
string Name
The
Name
property is a human readable brokerage name for debugging and logging. For US Equity regulated brokerages convention states this name generally ends in the word "Brokerage". -
void Connect()
The
Connect
method triggers logic for establishing a link to your brokerage. Normally we do not do this in the constructor as it would make algorithms and brokerages die in the BrokerageFactory process. For most brokerages this will be simply calling the connect method on your SDK library.Connection Method Examples Interactive Brokers Connecting to external process with brokerage SDK. FXCM Using SDK to create event handlers and connection thread. OANDA Simple example, calling SDK. GDAX Establish websocket connection and monitoring in a thread. Exceptions
In the event of a soft failure like internet connection lost, or server 502 errors you should create a new
BrokerageMessageEvent
. By doing this you allow the algorithm to handle the brokerage messages. For example: Interactive Brokers resets socket connections at different times globally, so users in other parts of the world can get disconnected at strange times of the day. Knowing this, they may elect to have their algorithm ignore specific disconnection attempts.In the event of a hard-failure such as the password being incorrect or an unsupported API method you should throw a real exception with details of the error. You can see examples of using both of these techniques in the FXCM Connect method.
-
void Disconnect()
Disconnect is called at the end of the algorithm before shutting down LEAN.
-
bool IsConnected
A boolean property to indicate the state of the brokerage connection. Depending on your connection style this may be automatically handled for you and simply require passing back the value from your SDK. Alternatively you may need to maintain your own connection state flag in your brokerage.
-
bool PlaceOrder(Order order)
Send a new LEAN order to the brokerage and report back the success or failure. The
PlaceOrder
method accepts a genericOrder
object which is the base class for all order types. The first step of placing an order is often to convert it from LEAN format into the Brokerage SDK required format. Brokerage implementations should aim to support as many LEAN order types as possible. There may be other order types in the brokerage but implementing them is considered out of scope of a rev-0 brokerage implementation.Once the order type is converted you should send it to your brokerage submit API. Often you will receive an immediate reply indicating the order was successfully placed.
Part of the order conversion might be converting the brokerage ticker (e.g. LEAN name "EURUSD", OANDA name is "EUR/USD"). This is done with a
BrokerageSymbolMapper
class. We'll get to that in Step 3, for now you can pass a request for the brokerage ticker to the stub implementation.The
PlaceOrder()
method returns true when the order is accepted to the brokerage. If it is invalid, immediately rejected or there is an internet outage you can return false.Tip:Create an internal method:
BrokerOrder ConvertOrder(Order order)
for conversion to brokerage orders. You will use this over and over again. Converting order types is an error prone process and its recommended to carefully review each order after you've completed a port. Remember to check each required brokerage order property, brokers like InteractiveBrokers have hundreds of properties.Tip:Use the
IsConnected
boolean to check if you're connected before placing the order. If not throw an exception to halt the algorithm. -
bool UpdateOrder(Order order)
Transmit an update request to the API and return true if it was successfully processed. There are no simple examples here! Updating an order is one of the most tricky part of the brokerage implementations. You can easily run into synchronization issues.
Update Order Examples Interactive Brokers External application, updating multiple asset classes. FXCM Relatively simple example via async brokerage callback. OANDA v1, v2 Complex example with 2 async APIs, multiple parameters. GDAX Updating isn't supported; throw an exception. - bool CancelOrder(Order order)
- bool UpdateOrder(Order order)
- List<Order> GetOpenOrders()
- List<Holding> GetAccountHoldings()
- List<Cash> GetCashBalance()
- bool AccountInstantlyUpdated
- IEnumerable<BaseData> GetHistory(HistoryRequest request)
- bool AccountInstantlyUpdated
You can also see our Documentation and Videos. You can also get in touch with us via Discord.
Did you find this page helpful?