Few questions

Back

Hi all,

I have a mean-reversion algo that's running using pylivetrader, it's doing well, but the backtesting from modifying on Quantopian and then modifying locally etc... QuantConnect is appealing because of all the reasons you're all here. 

There's a lot in the filtering, but the first 2 filtrations are a SMA of the volume; the 50 day being over 50,000, and then  current closing price being over the SMA-150 of closing.

I'm starting simple with just the SMA volume. I don't seem to get any results back - though now the logging limit (I've upgraded) has just screwed me so I'm losing the day because the logs are so limited.

 

private class SelectionData
{
private readonly Symbol _symbol;
private readonly QCAlgorithm _algorithm;

public readonly SimpleMovingAverage SMAClosing;
public readonly SimpleMovingAverage SMAVolume;
public readonly RelativeStrengthIndex RSI;
public readonly NormalizedAverageTrueRange NormalizedATR;

public SelectionData(QCAlgorithm algorithm, Symbol symbol)
{
_algorithm = algorithm;
_symbol = symbol;

RSI = new RelativeStrengthIndex(4);
SMAClosing = new SimpleMovingAverage(150);
SMAVolume = new SimpleMovingAverage(50);
NormalizedATR = new NormalizedAverageTrueRange(10);
}

public bool Update(DateTime time, decimal closePrice, decimal volume)
{
//_algorithm.Log($"symbol={_symbol.Value}, dt: {time}, close={closePrice}, volume={volume}");
return RSI.Update(time, closePrice)
&& SMAClosing.Update(time, closePrice)
&& SMAVolume.Update(time, volume);
}
}

public override void Initialize()
{
UniverseSettings.Resolution = Resolution.Daily;

SetStartDate(2020, 01, 01);
SetEndDate(2020, 04, 20);
SetCash(10000);
SetWarmUp(TimeSpan.FromDays(200));
SetBrokerageModel(BrokerageName.Alpaca);

AddUniverse(CoarseSelectionFunction, FineSelectionFunction);
}

public IEnumerable<Symbol> CoarseSelectionFunction(IEnumerable<CoarseFundamental> coarse)
{
// Moving above the 150 day SMA
return (from cf in coarse
let sel = _selection.GetOrAdd(cf.Symbol, sym => new SelectionData(this, cf.Symbol))
where cf.HasFundamentalData && cf.AdjustedPrice > 1
where sel.Update(cf.EndTime, cf.AdjustedPrice, cf.Volume)
where sel.SMAVolume > 50000
select cf.Symbol)
.Take(100);
}

private IEnumerable<Symbol> FineSelectionFunction(IEnumerable<FineFundamental> fine)
{
return fine
.Where(f => f.SecurityReference.IsPrimaryShare)
.Select(x => x.Symbol);
}

 

That's some of the code I'm using.

- How often does Initialize() run? 

- The docs or examples don't seem to address the best way of doing this kind of filtering. As you can see in my `SelectionData` class, there are more elements I want to screen the initial list of symbols on. There's more filtering but that will be in a scheduled buy/sell that runs daily.

I'm trying to now use a different way; which is:

private List<SecurityFilter> _securities = new List<SecurityFilter>();
private SecurityChanges _changes = SecurityChanges.None;

/// <summary>
///
/// </summary>
private struct SecurityFilter {
public Security Security;
public SimpleMovingAverage Volume;
}

public override void Initialize()
{
UniverseSettings.Resolution = Resolution.Daily;

SetStartDate(2020, 04, 01);
SetEndDate(2020, 04, 20);
SetCash(10000);
SetBrokerageModel(BrokerageName.Alpaca);

SetWarmUp(TimeSpan.FromDays(200));
AddUniverse(CoarseSelectionFunction, FineSelectionFunction);
}

/// <summary>
/// https://www.quantconnect.com/docs/algorithm-reference/universes#Universes-Coarse-Universe-Selection
/// https://www.quantconnect.com/forum/discussion/1515/question-about-universe-history/p1
/// </summary>
/// <param name="coarse"></param>
/// <returns></returns>
private IEnumerable<Symbol> CoarseSelectionFunction(IEnumerable<CoarseFundamental> coarse)
{
return (from cf in coarse
where cf.HasFundamentalData && cf.AdjustedPrice > 1
select cf.Symbol)
.Take(100);
}

private IEnumerable<Symbol> FineSelectionFunction(IEnumerable<FineFundamental> fine)
{
return fine
.Where(f => f.SecurityReference.IsPrimaryShare)
.Select(x => x.Symbol);
}

// this event fires whenever we have changes to our universe
public override void OnSecuritiesChanged(SecurityChanges changes)
{
_changes = changes;

if (changes.AddedSecurities.Count > 0) {
AddSecurities(changes.AddedSecurities);
}
}

/// <summary>
///
/// </summary>
/// <param name="securities"></param>
private void AddSecurities(IEnumerable<Security> securities)
{
var securityFilters = new List<SecurityFilter>();
foreach (var s in securities) {
var ef = new SecurityFilter {
Security = s,
Volume = new SimpleMovingAverage(50),
};

var history = History(ef.Security.Symbol, 200, Resolution.Daily);
foreach (var hist in history) {
ef.Volume.Update(hist.EndTime, hist.Volume);
}

securityFilters.Add(ef);
}

_securities = securityFilters
.Where(
f => f.Volume > 500000
)
.OrderBy(f => f.Volume)
.Take(10)
.ToList();

Debug($"Added securities: {securities}");
}

But now because I have no logs, I guess I'm losing the weekend.

Is there a list of best practices?

Thanks!

Update Backtest







 
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Is there also a guide for creating a separate project in VisualStudio to put our code/algorithms in? Or a Lean Nuget package which we can add as a reference into a new project to run? I can rebase into my branch from master, but having the code all mixed up with the examples, etc, is a little annoying.

I want to separate out my different classes into different files and all, so having a separate project, at a minimum, would be ideal.

0

I think it looks like I need to use insights? But it's not clear how to actually use it or add the parameters

0

To answer one of your questions, Initialize() runs once at the beginning of your algorithm,

0

I'm sorry if I've missed this but I've read through most of the docs - what's missing is a document that outlines how this all flows together - what the context of Initialize() is, universe creationg timing, and how often OnData() is called, if it's adjustable, how to adjust it - meaning if I set the Universe time to daily/minutely or something, does that affect OnData? How do I change that? 

And then there are alphas and insights, how those flow, what they do, how to set them up (full examples, not just the premade algorithms or just isolated code snippets). For example, insights look useful and that's probably what I need. But how do I actually use them? Do they somehow flow into the securities in OnSecuritiesChanged()? Or in OnData?

Under "Good Design Patterns" in Alpha Creation, there's "1. Use Assets Defined By Universe Selection Model", but ok... how's that then used in OnData() or for a scheduled run? I feel like there's a lot of peices here and there, but there's not really something that breaks things down (in the docs) at least.

https://www.quantconnect.com/docs/algorithm-framework/portfolio-construction

That starts to connect some stuff like insights, but then it's just standlone snippets that don't really mean anything.

0

Kyle, thanks, I read that, but I think my question was too vague. How does that relate to the universe creation/generation? Does that mean the universe, along with any Course/Fine adjustments, run only once?

0

So my 2nd method isn't working:

var history = History<IndicatorDataPoint>(ef.Security.Symbol, 200, Resolution.Daily);
foreach (var hist in history) {
ef.Volume.Update(hist);
ef.Atr.Update(hist);
ef.Rsi.Update(hist);
}

221 | 18:50:37:

BacktestingRealTimeHandler.Run(): There was an error in a scheduled event EveryDay: SPY: 10 min after MarketOpen. The error was The specified security is not of the requested type. Symbol: A RPTMYV3VC57P Requested Type: IndicatorDataPoint Actual Type: TradeBar

222 | 18:50:38:

Runtime Error: In Scheduled Event 'EveryDay: SPY: 10 min after MarketOpen', The specified security is not of the requested type. Symbol: A RPTMYV3VC57P Requested Type: IndicatorDataPoint Actual Type: TradeBar (Open Stacktrace)

 

I don't see in the docs how to use the indicators for selection or 'ad-hoc' queries. I'm trying to filter out securities where the 10-Day Normalized ATR is  > 4 and the 4-Day RSI is < 40. The docs talk about explicitly added equities, but not how to wire them up with the universe. 

0

Hi Nabeel,

I'll breakup this response up into the different questions that have been asked throughout the thread. Going forward though, we would appreciate focusing each discussion thread on a single topic.

Organizing Project Files
We no longer support our VS Plugin. Instead, we recommend moving to our new solution, Skylight. It automatically synchronizes your QC account to disk, giving you control over your project file structure.

Insights
We recommend using Insights when building algorithms. Thorough explanations on how to use them and their parameters can be found in our Algorithm Framework Overview and Alpha Creation documentation. Another great resource is our Algorithm Framework Bootcamp lesson.

Universe Creation Timing
By default, the universe selection is run once a day, at midnight. We can perform universe selection at a different interval with the ScheduledUniverseSelectionModel.

Resolution
OnData is called when a new data slice is available in the algorithm. We can control the resolution of the data through UniverseSettings.

More Code Examples
We have our software open-source on GitHub. For more example algorithms instead of just code snippets, check out our repository.

Runtime Error
I was unable to produce the same error. Please attach a backtest. It will make the error easier to resolve.

Best,
Derek

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thanks, Derek. I'll go through the bootcamp. There's no links to Skylight in the docs page(s), particularly: 

https://www.quantconnect.com/docs/key-concepts/developing-in-the-ide

I'm trying to attach a backtest, but it seems I can't attach one with a runtime error. I've playing with the MACD example learn the consolidated indicators. 

using System;
using System.Collections.Generic;
using QuantConnect.Brokerages;
using QuantConnect.Data.Consolidators;
using QuantConnect.Data.Market;
using QuantConnect.Indicators;
using QuantConnect.Interfaces;

namespace QuantConnect.Algorithm.CSharp
{
public class NabeelMacd: QCAlgorithm, IRegressionAlgorithmDefinition
{
private MovingAverageConvergenceDivergence _macd;
private NormalizedAverageTrueRange _atr;
private readonly string _symbol = "SPY";
private TradeBar _last;

/// <summary>
/// Initialise the data and resolution required, as well as the cash and start-end dates for your algorithm. All algorithms must initialized.
/// </summary>
public override void Initialize()
{
SetStartDate(2019, 01, 01);
SetEndDate(2020, 01, 01);
SetCash(10000);

SetBrokerageModel(BrokerageName.Alpaca);
SetWarmUp(TimeSpan.FromMinutes(26 * 60));

AddSecurity(SecurityType.Equity, _symbol, Resolution.Minute);

var barConsolidator = new TradeBarConsolidator(TimeSpan.FromMinutes(60));
// var barConsolidator = new TradeBarConsolidator(TimeSpan.FromHours(4));
barConsolidator.DataConsolidated += ConslidatedBarHandler;
SubscriptionManager.AddConsolidator(_symbol, barConsolidator);

// define our daily macd(12,26) with a 9 day signal
_macd = MACD(_symbol, 12, 26, 9, MovingAverageType.Exponential, Resolution.Hour);
_atr = new NormalizedAverageTrueRange(10);
}

/// <summary>
/// OnData event is the primary entry point for your algorithm. Each new data point will be pumped in here.
/// </summary>
/// <param name="data">TradeBars IDictionary object with your stock data</param>
public void OnData(TradeBars data)
{
}

/// <summary>
/// This is our event handler for our 30 minute trade bar defined above in Initialize(). So each time the consolidator
/// produces a new 30 minute bar, this function will be called automatically. The 'sender' parameter will be the
/// instance of the IDataConsolidator that invoked the event, but you'll almost never need that!
/// </summary>
private void ConslidatedBarHandler(object sender, TradeBar consolidated)
{
_macd.Update(consolidated);
_atr.Update(consolidated);

var holding = Portfolio[_symbol];

var signalDeltaPercent = (_macd - _macd.Signal)/_macd.Fast;
const decimal tolerance = 0.0025m;

if (_atr < 4) {
return;
}

// if our macd is greater than our signal, then let's go long
if (holding.Quantity <= 0 && signalDeltaPercent > tolerance) { // 0.01%
// longterm says buy as well
SetHoldings(_symbol, 1.0);
}
// of our macd is less than our signal, then let's go short
else if (holding.Quantity >= 0 && signalDeltaPercent < -tolerance) {
Liquidate(_symbol);
}

// plot both lines
Plot("MACD", _macd, _macd.Signal);
Plot(_symbol, "Open", consolidated.Open);
Plot(_symbol, _macd.Fast, _macd.Slow);

_last = consolidated;
}

/// <summary>
/// This is used by the regression test system to indicate if the open source Lean repository has the required data to run this algorithm.
/// </summary>
public bool CanRunLocally { get; } = true;

/// <summary>
/// This is used by the regression test system to indicate which languages this algorithm is written in.
/// </summary>
public Language[] Languages { get; } = { Language.CSharp, Language.Python };

/// <summary>
/// This is used by the regression test system to indicate what the expected statistics are from running the algorithm
/// </summary>
public Dictionary<string, string> ExpectedStatistics => new Dictionary<string, string>
{
{"Total Trades", "84"},
{"Average Win", "4.79%"},
{"Average Loss", "-4.16%"},
{"Compounding Annual Return", "2.963%"},
{"Drawdown", "34.700%"},
{"Expectancy", "0.228"},
{"Net Profit", "37.907%"},
{"Sharpe Ratio", "0.274"},
{"Probabilistic Sharpe Ratio", "0.399%"},
{"Loss Rate", "43%"},
{"Win Rate", "57%"},
{"Profit-Loss Ratio", "1.15"},
{"Alpha", "0.034"},
{"Beta", "-0.04"},
{"Annual Standard Deviation", "0.113"},
{"Annual Variance", "0.013"},
{"Information Ratio", "-0.234"},
{"Tracking Error", "0.214"},
{"Treynor Ratio", "-0.775"},
{"Total Fees", "$443.74"},
{"Fitness Score", "0.013"},
{"Kelly Criterion Estimate", "0"},
{"Kelly Criterion Probability Value", "0"},
{"Sortino Ratio", "0.216"},
{"Return Over Maximum Drawdown", "0.085"},
{"Portfolio Turnover", "0.024"},
{"Total Insights Generated", "0"},
{"Total Insights Closed", "0"},
{"Total Insights Analysis Completed", "0"},
{"Long Insight Count", "0"},
{"Short Insight Count", "0"},
{"Long/Short Ratio", "100%"},
{"Estimated Monthly Alpha Value", "$0"},
{"Total Accumulated Estimated Alpha Value", "$0"},
{"Mean Population Estimated Insight Value", "$0"},
{"Mean Population Direction", "0%"},
{"Mean Population Magnitude", "0%"},
{"Rolling Averaged Population Direction", "0%"},
{"Rolling Averaged Population Magnitude", "0%"},
{"OrderListHash", "-1703572248"}
};
}
}

 

0

Hi Nabeel,

The runtime error the algorithm was experiencing is a result of how the algorithm is updating the indicators with the consolidated data. Instead of calling

_macd.Update(consolidated);
_atr.Update(consolidated);

in ConsolidatedBarHandler, we can register the indicators to be updated with 30 minute bars by writing

RegisterIndicator(_symbol, _macd, TimeSpan.FromMinutes(30));
RegisterIndicator(_symbol, _atr, TimeSpan.FromMinutes(30));

We can then ensure the indicators are ready in ConsolidatedBarHandler with

if (!_macd.IsReady || !_atr.IsReady) {
return;
}

See the attached backtest for the full solution file.

 

John,
To see what member functions are available for our API classes, we can utilize the LEAN documentation. For example, this page in the documentation shows all the members and methods of the SecurityPortfolioManager class.

Best,
Derek

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Got it, thanks, so I was missing the RegisterIndicator()... the bootcamp has been very helpful. Next step is learning how to incorporate it with the Universe but looks like the Alphas are the way to go... off to the next bootcamp.

0

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed