Algorithm Framework
Insight Manager
Introduction
The InsightManager tracks all of the Insight objects in your algorithm. It's an InsightCollection, which stores all of your insights in an internal dictionary that has the security Symbol as the key and a list of Insight objects as the value. You can access the manager anywhere in your algorithm where you have reference to the algorithm class. If you want to use an InsightCollection in your algorithm that's seperate from the InsightManager, create a new InsightCollection.
Check Membership
To check if an insight exists in the InsightManager, call the Containscontains method.
// Check the algorithms central insight collection (InsightManager) to avoid emitting duplicates, reducing the processing work.
if (algorithm.Insights.Contains(insight))
{
} # Check the algorithms central insight collection (InsightManager) to avoid emitting duplicates, reducing the processing work.
if algorithm.insights.contains(insight):
pass
To check if the InsightManager has an insight for a Symbol, call the ContainsKeycontains_key method.
// Check the algorithms central insight collection (InsightManager) if there's already been an insight for this symbol across the other alphas.
if (algorithm.Insights.ContainsKey(symbol))
{
} # Check the algorithms central insight collection (InsightManager) if there's already been an insight for this symbol across the other alphas.
if algorithm.insights.contains_key(symbol):
pass
To check if the InsightManager has active insights for a Symbol at a specific Coordinated Universal Time (UTC), call the HasActiveInsightshas_active_insights method.
// Check if there are active insights for a specific symbol at the given time to determine if alpha should emit more insights.
if (algorithm.Insights.HasActiveInsights(symbol, utcTime))
{
} # Check if there are active insights for a specific symbol at the given time to determine if alpha should emit more insights
if algorithm.insights.has_active_insights(symbol, utc_time):
pass
Get Insights
To get the insights for a Symbol, index the InsightManager with the Symbol.
if (algorithm.Insights.ContainsKey(symbol))
{
var insights = algorithm.Insights[symbol];
} if algorithm.insights.contains_key(symbol):
insights = algorithm.insights[symbol]
To get the insights that pass a filter, call the GetInsightsget_insights method.
var insights = algorithm.Insights.GetInsights(insight => insight.Direction == InsightDirection.Up);
insights = algorithm.insights.get_insights(lambda insight: insight.direction == Insightdirection.up)
To iterate through the InsightManager, call the GetEnumeratorget_enumerator method.
var enumerator = algorithm.insights.get_enumerator();
enumerator = algorithm.insights.get_enumerator()
To get all of the insights that will be active at a certain UTC time, call the GetActiveInsightsget_active_insights method.
var activeInsights = algorithm.Insights.GetActiveInsights(utcTime);
active_insights = algorithm.insights.get_active_insights(utc_time)
Remove Insights
Only the Portfolio Construction model should remove insights from the InsightManager. It should remove insights when the insights expire and when the corresponding security leaves the universe.
To remove an insight from the InsightManager, call the Removeremove method.
var removeSuccessful = algorithm.Insights.Remove(insight);
remove_successful = algorithm.insights.remove(insight)
To remove all the insights for a set of Symbol objects, pass a list of Symbol objects the Clearclear method.
algorithm.insights.clear(symbols);
algorithm.insights.clear(symbols)
To remove all the insights that will be expired at a certain UTC time, call the RemoveExpiredInsightsremove_expired_insights method.
var expiredInsights = algorithm.Insights.RemoveExpiredInsights(utcTime);
expired_insights = algorithm.insights.remove_expired_insights(utc_time)
Cancel Insights
In some cases, you may want to cancel an Insight. For example, if a Risk Management model in your algorithm liquidates a security, it should also cancel all of the active insights for the security. If you don't cancel the insights, the Portfolio Construction model might create a new PortfolioTarget to re-enter the position.
Another example of a situtation where you would want to cancel an Insight is when an Alpha model in your algorithm determines the trading opportunity has pre-maturely ended. For instance, say you want your algorithm to enter a long position in a security when its Relative Strength Index (RSI) moves below 20 and then liquidate the position when the security's RSI moves above 30. If you emit an Insight that has a duration of 30 days when the RSI moves below 20 but the RSI moves above 30 in 10 days, you should cancel the Insight when the RSI moves above 30.
To cancel insights, call the Cancelcancel/Expire method with a list of Insight objects.
algorithm.insights.cancel(insights)
algorithm.insights.cancel(insights);
To cancel all the insights for some securities, call the Cancelcancel/Expire method with a list of Symbol objects.
algorithm.insights.cancel(symbols)
algorithm.insights.cancel(symbols);
When you cancel an active insight, it's CloseTimeUtcclose_time_utc property is set to one second into the past.
Add Insights to Self-Managed Insight Collections
To add insights to the InsightManager, return a list of Insight objects from the Updateupdate method of your Alpha model or call the EmitInsightsemit_insights method. If you manage an InsightCollection that's seperate from the InsightManager, there are some methods to add Insight objects to it.
To add an insight to an InsightCollection, call the Addadd method.
// You can manually add an Insight to an InsightCollection object to organize, fetch, or cancel insights an alpha has generated. _insightCollection.Add(insight);
# You can manually add an Insight to an InsightCollection object to organize, fetch, or cancel insights an alpha has generated. self.insight_collection.add(insight)
To add a list of insights, call the AddRangeadd_range method.
// You can manually add a list of Insights to an InsightCollection object to organize, fetch, or cancel insights an alpha has generated. _insightCollection.AddRange(insights);
# You can manually add a list of Insights to an InsightCollection object to organize, fetch, or cancel insights an alpha has generated. self.insight_collection.add_range(insights)
Preserve Insights Between Deployments
Follow these steps to use the Object Store to preserve the algorithm state across live deployments:
- Create an algorithm that defines a storage key and adds insights to the Insight Manager.
- At the top of the algorithm file, add the following imports:
- In the OnEndOfAlgorithmon_end_of_algorithm event handler of the algorithm, get the Insight objects and save them in the Object Store as a JSON object.
- At the bottom of the
Initializeinitializemethod, read the Insight objects from the Object Store and add them to the Insight Manager.
public class ObjectStoreChartingAlgorithm : QCAlgorithm
{
private string _insightKey;
public override void Initialize()
{
_insightKey = $"{ProjectId}/insights";
SetUniverseSelection(new ManualUniverseSelectionModel(QuantConnect.Symbol.Create("SPY", SecurityType.Equity, Market.USA)));
SetAlpha(new ConstantAlphaModel(InsightType.Price, InsightDirection.Up, TimeSpan.FromDays(5), 0.025, null));
}
} class ObjectStoreChartingAlgorithm(QCAlgorithm):
def initialize(self):
self.insight_key = f"{self.project_id}/insights"
self.set_universe_selection(ManualUniverseSelectionModel([ Symbol.create("SPY", SecurityType.EQUITY, Market.USA) ]))
self.set_alpha(ConstantAlphaModel(InsightType.PRICE, InsightDirection.UP, timedelta(5), 0.025, None))
from Newtonsoft.Json import JsonConvert from System.Collections.Generic import List
Insight objects are a C# objects, so you need the preceding C# libraries to serialize and deserialize them.
public override void OnEndOfAlgorithm()
{
var insights = Insights.GetInsights(x => x.IsActive(UtcTime));
ObjectStore.SaveJson(_insightKey, insights);
} def on_end_of_algorithm(self):
insights = self.insights.get_insights(lambda x: x.is_active(self.utc_time))
content = ','.join([JsonConvert.SerializeObject(x) for x in insights])
self.object_store.save(self.insight_key, f'[{content}]')
if (ObjectStore.ContainsKey(_insightKey))
{
var insights = ObjectStore.ReadJson<List<Insight>>(_insightKey);
Insights.AddRange(insights);
} if self.object_store.contains_key(self.insight_key):
insights = self.object_store.read_json[list[Insight]](self.insight_key)
self.insights.add_range(insights)
The following algorithm provides a full example of preserving the Insight state between deployments:
public class ObjectStoreInsightsAlgorithm : QCAlgorithm
{
private string _insightsKey = "insights";
public override void Initialize()
{
UniverseSettings.Resolution = Resolution.Daily;
SetStartDate(2023, 4, 1);
SetEndDate(2023, 4, 11);
SetCash(100000);
_insightsKey = $"{ProjectId}/{_insightsKey}";
// Read the file with the insights
if (ObjectStore.ContainsKey(_insightsKey))
{
// Load cached JSON key value pair from object store, then add to algorithm insights.
var insights = ObjectStore.ReadJson<List<Insight>>(_insightsKey);
Log($"Read {insights.Count} insight(s) from the Object Store");
Insights.AddRange(insights);
// Delete the key to reuse it
ObjectStore.Delete(_insightsKey);
}
SetUniverseSelection(new ManualUniverseSelectionModel(QuantConnect.Symbol.Create("SPY", SecurityType.Equity, Market.USA)));
SetAlpha(new ConstantAlphaModel(InsightType.Price, InsightDirection.Up, TimeSpan.FromDays(5), 0.025, null));
SetPortfolioConstruction(new EqualWeightingPortfolioConstructionModel(Resolution.Daily));
}
public override void OnEndOfAlgorithm()
{
// Get all active insights.
var insights = Insights.GetInsights(x => x.IsActive(UtcTime));
// If we want to save all insights (expired and active), we can use.
// var insights = Insights.GetInsights(x => true);
Log($"Save {insights.Count} insight(s) to the Object Store.");
ObjectStore.SaveJson(_insightsKey, insights);
}
} class ObjectStoreInsightsAlgorithm(QCAlgorithm):
def initialize(self) -> None:
self.universe_settings.resolution = Resolution.DAILY
self.set_start_date(2023,4,1)
self.set_end_date(2023,4,11)
self.set_cash(100000)
self.insights_key = f"{self.project_id}/insights"
# Read the file with the insights
if self.object_store.contains_key(self.insights_key):
# Load cached JSON key value pair from object store, then add to algorithm insights.
insights = self.object_store.read_json[list[Insight]](self.insights_key)
self.log(f"Read {len(insights)} insight(s) from the Object Store")
self.insights.add_range(insights)
# Delete the key to reuse it
self.object_store.delete(self.insights_key)
self.set_universe_selection(ManualUniverseSelectionModel([ Symbol.create("SPY", SecurityType.EQUITY, Market.USA) ]))
self.set_alpha(ConstantAlphaModel(InsightType.PRICE, InsightDirection.UP, timedelta(5), 0.025, None))
self.set_portfolio_construction(EqualWeightingPortfolioConstructionModel(Resolution.DAILY))
def on_end_of_algorithm(self) -> None:
# Get all active insights.
insights = self.insights.get_insights(lambda x: x.is_active(self.utc_time))
# If we want to save all insights (expired and active), we can use
# insights = self.insights.get_insights(lambda x: True)
self.log(f"Save {len(insights)} insight(s) to the Object Store.")
content = ','.join([JsonConvert.SerializeObject(x) for x in insights])
self.object_store.save(self.insights_key, f'[{content}]')
Examples
The following examples demonstrate some common practices for using insight manager.
Example 1: Cancel Insight
If the risk management model only submits the PortfolioTarget to liquidate the security, it will be re-ordered immediately due to the active insight. Hence, we also need to add logic to cancel the insight in the ManageRiskmanage_risk method of the risk management model. The below example only permits 1-day holding time per each insight through a custom risk management model.
public class FrameworkInsightManagerAlgorithm : QCAlgorithm
{
public override void Initialize()
{
SetStartDate(2024, 8, 12);
SetEndDate(2024, 9, 1);
SetCash(1000000);
// Add a universe of the most liquid stocks since their trend is more capital-supported.
AddUniverseSelection(new QC500UniverseSelectionModel());
// Emit insights on EMA cross, indicating the trend changes. We use short-term versus medium-term for more trade opportunities.
AddAlpha(new EmaCrossAlphaModel(20, 60, Resolution.Daily));
// Equal weighting on each insight is needed to dissipate capital risk evenly.
SetPortfolioConstruction(new EqualWeightingPortfolioConstructionModel());
// Add a risk management model that controls maximum holding time.
AddRiskManagement(new MaximumHoldingTimeRiskManagementModel(TimeSpan.FromDays(5)));
}
private class MaximumHoldingTimeRiskManagementModel : RiskManagementModel
{
private TimeSpan _maximumHoldingTime;
// A dictionary to hold the holding time of each position.
private Dictionary<Symbol, DateTime> _holdingTime = new();
public MaximumHoldingTimeRiskManagementModel(TimeSpan maximumHoldingTime)
{
_maximumHoldingTime = maximumHoldingTime;
}
public override IEnumerable<IPortfolioTarget> ManageRisk(QCAlgorithm algorithm, IPortfolioTarget[] _)
{
var targets = new List<PortfolioTarget>();
foreach (var (symbol, holding) in algorithm.Portfolio)
{
if (holding.Invested)
{
if (!_holdingTime.TryGetValue(symbol, out var time))
{
// For newly open positions, save the entry time.
_holdingTime[symbol] = time = algorithm.Time;
}
else if (algorithm.Time - time > _maximumHoldingTime)
{
// Liquidate the position if the holding time exceeds the permitted level.
targets.Add(new PortfolioTarget(symbol, 0m));
// Cancel the insight to avoid repeated re-ordering.
algorithm.Insights.Cancel(new[] { symbol });
// Remove from dictionary.
_holdingTime.Remove(symbol);
}
}
}
return targets;
}
}
} class FrameworkInsightManagerAlgorithm(QCAlgorithm):
def initialize(self) -> None:
self.set_start_date(2024, 8, 12)
self.set_end_date(2024, 9, 1)
self.set_cash(1000000)
# Add a universe of the most liquid stocks since their trend is more capital-supported.
self.add_universe_selection(QC500UniverseSelectionModel())
# Emit insights on EMA cross, indicating the trend changes. We use short-term versus medium-term for more trade opportunities.
self.add_alpha(EmaCrossAlphaModel(20, 60, Resolution.DAILY))
# Equal weighting on each insight to dissipate capital risk evenly.
self.set_portfolio_construction(EqualWeightingPortfolioConstructionModel())
# Add a risk management model that controls maximum holding time.
self.add_risk_management(MaximumHoldingTimeRiskManagementModel(timedelta(5)))
class MaximumHoldingTimeRiskManagementModel(RiskManagementModel):
def __init__(self, maximum_holding_time: timedelta) -> None:
self.maximum_holding_time = maximum_holding_time
# A dictionary to hold the holding time of each position.
self.holding_time = {}
def manage_risk(self, algorithm: QCAlgorithm, targets: list[PortfolioTarget]) -> list[PortfolioTarget]:
targets = []
for symbol, holding in algorithm.portfolio.items():
if holding.invested:
if symbol not in self.holding_time:
# For newly open positions, save the entry time.
self.holding_time[symbol] = algorithm.time
elif algorithm.time - self.holding_time[symbol] > self.maximum_holding_time:
# Liquidate the position if the holding time exceeds the permitted level.
targets.append(PortfolioTarget(symbol, 0))
# Cancel the insight to avoid repeated re-ordering.
algorithm.insights.cancel([symbol])
# Remove from the dictionary.
self.holding_time.pop(symbol)
return targets
You can also see our Videos. You can also get in touch with us via Discord.
Did you find this page helpful?
