Checkout our new book!
Hands on AI Trading with Python, QuantConnect, and AWS
Learn More 
Short Availability
Key Concepts
Introduction
To short a security, you need to borrow shares from another investor or organization that owns the shares. A shortable provider is a model that tracks the number of shares that are available for you to borrow. A shortable provider can make your backtest results more realistic because it doesn't let you place a short trade if there are no shares available for you to borrow.
Set Providers
The brokerage model of your algorithm automatically sets the settlement model for each security, but you can override it. To manually set the shortable provider of a security, call the SetShortableProviderset_shortable_provider method on the Security object.
public override void Initialize()
{
var security = AddEquity("SPY");
// Get shortable stocks and quantity from local disk
security.SetShortableProvider(new LocalDiskShortableProvider("axos"));
} def initialize(self) -> None:
security = self.add_equity("SPY")
# Get shortable stocks and quantity from local disk
security.set_shortable_provider(LocalDiskShortableProvider("axos"))
You can also set the shortable provider in a security initializer. If your algorithm has a universe, use the security initializer technique. In order to initialize single security subscriptions with the security initializer, call SetSecurityInitializerset_security_initializer before you create the subscriptions.
public override void Initialize()
{
// Set the security initializer before requesting data to apply to all requested securities afterwards
SetSecurityInitializer(CustomSecurityInitializer);
AddEquity("SPY");
}
private void CustomSecurityInitializer(Security security)
{
security.SetShortableProvider(new LocalDiskShortableProvider("axos"));
}
def initialize(self) -> None:
# Set the security initializer before requesting data to apply to all requested securities afterwards
self.set_security_initializer(self.custom_security_initializer)
self.add_equity("SPY")
def custom_security_initializer(self, security: Security) -> None:
security.set_shortable_provider(LocalDiskShortableProvider("axos"))
If you call the SetSecurityInitializerset_security_initializer method, it overwrites the default security initializer. The default security initializer uses the security-level reality models of the brokerage model to set the following reality models of each security:
To extend upon the default security initializer instead of overwriting it, create a custom BrokerageModelSecurityInitializer.
public class BrokerageModelExampleAlgorithm : QCAlgorithm
{
public override void Initialize()
{
// In the Initialize method, set the security initializer to set models of assets.
// To seed the security with the last known prices, set the security seeder to FuncSecuritySeeder with the GetLastKnownPrices method.
var securitySeeder = new FuncSecuritySeeder(GetLastKnownPrices);
// In some case, for example, adding Options and Futures using Universes, we don't need to seed prices.
// To avoid the overhead of and potential timeouts for calling the GetLastKnownPrices method, set the security seeder to SecuritySeeder.Null.
securitySeeder = SecuritySeeder.Null;
SetSecurityInitializer(new MySecurityInitializer(BrokerageModel, securitySeeder));
}
}
public class MySecurityInitializer : BrokerageModelSecurityInitializer
{
public MySecurityInitializer(IBrokerageModel brokerageModel, ISecuritySeeder securitySeeder)
: base(brokerageModel, securitySeeder) {}
public override void Initialize(Security security)
{
// First, call the superclass definition.
// This method sets the reality models of each security using the default reality models of the brokerage model.
base.Initialize(security);
// Next, overwrite some of the reality models
security.SetShortableProvider(new LocalDiskShortableProvider("axos")); }
} class BrokerageModelExampleAlgorithm(QCAlgorithm):
def initialize(self) -> None:
# In the Initialize method, set the security initializer to set models of assets.
# To seed the security with the last known prices, set the security seeder to FuncSecuritySeeder with the get_last_known_prices method.
security_seeder = FuncSecuritySeeder(self.get_last_known_prices)
# In some case, for example, adding Options and Futures using Universes, we don't need to seed prices.
# To avoid the overhead of and potential timeouts for calling the get_last_known_prices method, set the security seeder to SecuritySeeder.NULL.
security_seeder = SecuritySeeder.NULL
self.set_security_initializer(MySecurityInitializer(self.brokerage_model, security_seeder))
# Outside of the algorithm class
class MySecurityInitializer(BrokerageModelSecurityInitializer):
def __init__(self, brokerage_model: IBrokerageModel, security_seeder: ISecuritySeeder) -> None:
super().__init__(brokerage_model, security_seeder)
def initialize(self, security: Security) -> None:
# First, call the superclass definition.
# This method sets the reality models of each security using the default reality models of the brokerage model.
super().initialize(security)
# Next, overwrite some of the reality models
security.set_shortable_provider(LocalDiskShortableProvider("axos"))
To view all the pre-built shortable providers, see Supported Providers.
Default Behavior
The brokerage model of your algorithm automatically sets the shortable provider for each security. The default brokerage model is the DefaultBrokerageModel, which uses the NullShortableProvider.
Provider Structure
Shortable providers must implement the IShortableProvider interface.
Shortable providers that implement the IShortableProvider interface must implement the FeeRatefee_rate, RebateRaterebate_rate, and ShortableQuantityshortable_quantity methods.
These methods receives a Symbol and the local time of the algorithm.
The FeeRatefee_rate method returns the borrow fee rate.
The RebateRaterebate_rate method returns the borrow rebate rate.
The ShortableQuantityshortable_quantity returns the shortable quantity.
public class CustomShortableProviderExampleAlgorithm : QCAlgorithm
{
public override void Initialize()
{
var security = AddEquity("SPY");
// Provide the selected security with your broker's shortable information
security.SetShortableProvider(new MyShortableProvider());
}
}
// Define the custom shortable provider
class MyShortableProvider : IShortableProvider
{
public decimal FeeRate(Symbol symbol, DateTime localTime)
{
return 0.0025m;
}
public decimal RebateRate(Symbol symbol, DateTime localTime)
{
return 0.0507m;
}
public long? ShortableQuantity(Symbol symbol, DateTime localTime)
{
return 10000;
}
} class CustomShortableProviderExampleAlgorithm(QCAlgorithm):
def initialize(self) -> None:
security = self.add_equity("SPY")
# Provide the selected security with your broker's shortable information
security.set_shortable_provider(MyShortableProvider())
# Define the custom shortable provider
class MyShortableProvider(NullShortableProvider):
def fee_rate(self, symbol: Symbol, local_time: datetime) -> float:
return 0.0025
def rebate_rate(self, symbol: Symbol, local_time: datetime) -> float:
return 0.0507
def shortable_quantity(self, symbol: Symbol, local_time: datetime) -> float:
return 10000
For a full example algorithm, see this backtestthis backtest.
Borrowing Costs
In live trading, your brokerage charges an interest fee when you borrow shares to short a security. In backtesting, we don't currently simulate short interest fees. You can evaluate the borrowing cost with the FeeRatefee_rate and RebateRaterebate_rate methods.
To get valid borrowing rates, use the InteractiveBrokersShortableProvider.
Examples
The following examples demonstrate common practices for using a shortable provider.
Example 1: Sizing By Shortable Quantity
The following algorithm trades arbitrage of class A and class C price divergence of Google stocks using cointegration analysis. However, the position sizing may be limited by the available quantity to be shorted by the brokerage, breaking the paired trade. Thus, we need to make use of the
ShortableQuantity
shortable_quantity
method of the shortable provider to calculate the quantity.
using Accord.Statistics;
using MathNet.Numerics.LinearRegression;
using QuantConnect.Data.Shortable;
public class ShortableProviderAlgorithm : QCAlgorithm
{
private Symbol _goog1, _goog2;
// The threshold that the spread/residual of the cointegrated series triggers a trade.
private decimal _thresold;
// Store the coefficient and intercept of the cointegrated series for calculating the spread of a new data point.
private decimal[] _coefficients = new[] { 0m, 0m };
// Store the price series of each symbol for cointegration calculation.
private Dictionary<Symbol, RollingWindow<decimal>> _windows = new();
public override void Initialize()
{
SetStartDate(2019, 1, 1);
SetEndDate(2023, 1, 1);
// Subscribe to 2 classes of Google stocks to trade their price divergence.
var goog1 = AddEquity("GOOGL", Resolution.Daily);
var goog2 = AddEquity("GOOG", Resolution.Daily);
_goog1 = goog1.Symbol; // Class A
_goog2 = goog2.Symbol; // Class C
// Set the shortable providers to simulate the available shares to trade realistically.
goog1.SetShortableProvider(new InteractiveBrokersShortableProvider());
goog2.SetShortableProvider(new InteractiveBrokersShortableProvider());
// Use rolling windows to save the price data for cointegration analysis.
_windows[_goog1] = new(252);
_windows[_goog2] = new(252);
// Warm up the rolling windows.
var history = History<TradeBar>(new[] { _goog1, _goog2 }, 252, Resolution.Daily);
foreach (var bar in history)
{
_windows[_goog1].Add(bar[_goog1].Close);
_windows[_goog2].Add(bar[_goog2].Close);
}
// Adjust the cointegration factor between the 2 classes' monthly price series.
Schedule.On(
DateRules.MonthStart(),
TimeRules.At(0, 1),
CalculateCointegration
);
CalculateCointegration();
}
public override void OnData(Slice slice)
{
if (slice.Bars.TryGetValue(_goog1, out var bar1) && slice.Bars.TryGetValue(_goog2, out var bar2))
{
// Update the rolling windows for cointegration analysis.
_windows[_goog1].Add(bar1.Close);
_windows[_goog2].Add(bar2.Close);
// Calculate the current cointegrated series spread.
var residual = _coefficients[0] * bar2.Close + _coefficients[1] - bar1.Close;
var quantity = CalculateOrderQuantity(_goog1, 0.2m);
// If the residual is lower than the negative threshold, it means class A price is much higher than what it should be compared to class C.
// We sell class A and buy class C to bet on their price convergence.
if (residual < -_thresold && !Portfolio[_goog1].IsShort)
{
// Since it will be limited by shortable quantity, we max{shortable, 20% of portfolio value}.
var shortableQuantity = Securities[_goog1].ShortableProvider.ShortableQuantity(_goog1, bar1.EndTime);
quantity = Math.Min(quantity, Convert.ToDecimal(shortableQuantity));
MarketOrder(_goog1, -quantity);
MarketOrder(_goog2, quantity * _coefficients[0]);
}
// If the residual is higher than the threshold, it means class A price is much lower than what it should be compared to class C.
// We buy class A and sell class C to bet on their price convergence.
else if (residual > _thresold && !Portfolio[_goog1].IsLong)
{
// Since it will be limited by shortable quantity, we max{shortable, 20% of portfolio value}.
var shortableQuantity = Securities[_goog2].ShortableProvider.ShortableQuantity(_goog2, bar2.EndTime);
quantity = Math.Min(quantity, Convert.ToDecimal(shortableQuantity));
MarketOrder(_goog1, quantity);
MarketOrder(_goog2, -quantity * _coefficients[0]);
}
// Close positions of the price are converged.
else if ((Portfolio[_goog1].IsShort && residual > 0m) || (Portfolio[_goog1].IsLong && residual < 0m))
{
Liquidate();
}
}
}
private void CalculateCointegration()
{
// Lag direction is unimportant; it is just a sign flip in the linear regression, so we don't need to flip the window order.
var y = _windows[_goog1].Select(x => (double)x).ToArray();
var x = _windows[_goog2].Select(x => (double)x).ToArray();
// Perform Linear Regression on both price series to investigate their relationship.
var regressionResult = SimpleRegression.Fit(x, y);
var intercept = regressionResult.Item1;
var slope = regressionResult.Item2;
// Calculate the residuals series to check if it is stationary, meaning if the 2 price series move together.
var residuals = new double[x.Length];
for (int i = 0; i < x.Length; i++)
{
residuals[i] = y[i] - (intercept + slope * x[i]);
}
// Check if the residuals are stationary using the augmented Dickey-Fuller test.
if (ADFTest(residuals))
{
// If cointegrated, update the positional sizing ratio and the spread threshold of the trade trigger.
_coefficients = new[] { Convert.ToDecimal(slope), Convert.ToDecimal(intercept) };
_thresold = 2m * Convert.ToDecimal(Measures.StandardDeviation(residuals));
}
else
{
// If not cointegrated, liquidate and set the size to zeros for no positions.
Liquidate();
_coefficients = new[] { 0m, 0m };
_thresold = 100000000m; // An arbitrarily large number that the class A price will never reach.
}
}
private static bool ADFTest(double[] series)
{
var n = series.Length;
var lagged = new double[n - 1];
var differences = new double[n - 1];
// Fit linear regression for the residual series on unit root: ΔY_t = α + βY_{t-1} + ε_t.
for (int i = 1; i < n; i++)
{
lagged[i - 1] = series[i - 1];
differences[i - 1] = series[i] - series[i - 1];
}
var regressionResult = SimpleRegression.Fit(lagged, differences);
var alpha = regressionResult.Item1; // Intercept
var beta = regressionResult.Item2; // Coefficient of lagged term
// Calculate the ADF statistic and check if the null hypothesis is rejected.
var adfStatistic = beta / Measures.StandardError(differences);
// Reject the null hypothesis of a unit root is present if test statistic <= -3.45 (approximate α=0.05 for n=250)
// This means no unit root exists for the difference series, and the residuals are stationary.
return adfStatistic <= -3.45d;
}
}
from sklearn.linear_model import LinearRegression
from statsmodels.tsa.stattools import adfuller
class ShortableProviderAlgorithm(QCAlgorithm):
# The threshold that the spread/residual of the cointegrated series triggers a trade.
threshold = 0
# Store the coefficient and intercept of the cointegrated series for calculating the spread of a new data point.
coefficients = [0, 0]
def initialize(self) -> None:
self.set_start_date(2019, 1, 1)
self.set_end_date(2023, 1, 1)
# Subscribe to 2 classes of Google stocks to trade their price divergence.
goog1 = self.add_equity("GOOGL", Resolution.DAILY) # Class A
goog2 = self.add_equity("GOOG", Resolution.DAILY) # Class C
self.goog1 = goog1.symbol
self.goog2 = goog2.symbol
# Set the shortable providers to simulate the available shares to trade realistically.
goog1.set_shortable_provider(InteractiveBrokersShortableProvider())
goog2.set_shortable_provider(InteractiveBrokersShortableProvider())
# Use rolling windows to save the price data for cointegration analysis.
goog1.window = RollingWindow(252)
goog2.window = RollingWindow(252)
# Warm up the rolling windows.
history = self.history[TradeBar]([self.goog1, self.goog2], 252, Resolution.DAILY)
for bar in history:
goog1.window.add(bar[self.goog1].close)
goog2.window.add(bar[self.goog2].close)
# Adjust the cointegration factor between the 2 classes' monthly price series.
self.schedule.on(
self.date_rules.month_start(),
self.time_rules.at(0, 1),
self.calculate_cointegration
)
self.calculate_cointegration()
def on_data(self, slice: Slice) -> None:
bar1 = slice.bars.get(self.goog1)
bar2 = slice.bars.get(self.goog2)
if bar1 and bar2:
# Update the rolling windows for cointegration analysis.
self.securities[self.goog1].window.add(bar1.close)
self.securities[self.goog2].window.add(bar2.close)
# Calculate the current cointegrated series spread.
residual = self.coefficients[0] * bar2.close + self.coefficients[1] - bar1.close
quantity = self.calculate_order_quantity(self.goog1, 0.2)
# If the residual is lower than the negative threshold, it means class A's price is much higher than it should be compared to class C.
# We sell class A and buy class C to bet on their price convergence.
if residual < -self.threshold and not self.portfolio[self.goog1].is_short:
# Since it will be limited by shortable quantity, we max{shortable, 20% of portfolio value}.
shortable_quantity = self.securities[self.goog1].shortable_provider.shortable_quantity(self.goog1, bar1.end_time)
quantity = min(quantity, shortable_quantity)
self.market_order(self.goog1, -quantity)
self.market_order(self.goog2, quantity * self.coefficients[0])
# If the residual is higher than the threshold, it means class A price is much lower than what it should be compared to class C.
# We buy class A and sell class C to bet on their price convergence.
elif residual > self.threshold and not self.portfolio[self.goog1].is_long:
# Since it will be limited by shortable quantity, we max{shortable, 20% of portfolio value}.
shortable_quantity = self.securities[self.goog2].shortable_provider.shortable_quantity(self.goog2, bar2.end_time)
quantity = min(quantity, shortable_quantity)
self.market_order(self.goog1, quantity)
self.market_order(self.goog2, -quantity * self.coefficients[0])
# Close positions of the price are converged.
elif (self.portfolio[self.goog1].is_short and residual > 0) or (self.portfolio[self.goog1].is_long and residual < 0):
self.liquidate()
def calculate_cointegration(self) -> None:
# Lag direction is unimportant; it is just a sign flip in the linear regression, so we don't need to flip the window order.
y = np.array(list(self.securities[self.goog1].window)).reshape(-1, 1)
x = np.array(list(self.securities[self.goog2].window)).reshape(-1, 1)
# Perform Linear Regression on both price series to investigate their relationship.
lr = LinearRegression().fit(x, y)
slope = lr.coef_[0]
intercept = lr.intercept_
# Calculate the residuals series to check if it is stationary, meaning if the 2 price series move together.
residuals = y - (intercept + slope * x)
# Check if the residuals are stationary using the augmented Dickey-Fuller test.
# Reject the null hypothesis of a unit root is present if test statistic <= -3.45 (approximate α=0.05 for n=250)
# This means no unit root exists for the difference series, and the residuals are stationary.
adf_reject = adfuller(residuals)[0] <= -3.45
if adf_reject:
# If cointegrated, update the positional sizing ratio and the spread threshold of the trade trigger.
self.coefficients = [slope, intercept]
self.threshold = 2 * np.std(residuals)
else:
# If not cointegrated, liquidate and set the size to zeros for no positions.
self.liquidate()
self.coefficients = [0, 0]
self.threshold = 100000000 # An arbitrarily large number that the class A price will never reach.
Other Examples
For more examples, see the following algorithms:
Demonstration Algorithms
InteractiveBrokersShortableProviderAlgorithm.py
Python
CustomShortableProviderRegressionAlgorithm.py
Python
AllShortableSymbolsCoarseSelectionRegressionAlgorithm.py
Python
InteractiveBrokersShortableProviderAlgorithm.cs
C#
CustomShortableProviderRegressionAlgorithm.cs
C#
AllShortableSymbolsCoarseSelectionRegressionAlgorithm.cs
C#
You can also see our Videos. You can also get in touch with us via Discord.
Did you find this page helpful?
Contribute to the documentation:
