QUANTCONNECT COMMUNITY

No Results

Join Our Discord Channel

Join QuantConnect's Discord server for real-time support, where a vibrant community of traders and developers awaits to help you with any of your QuantConnect needs.

pending review This research is under review. To publish this research attract three community upvotes.

Draft Discussions

Bookmarked Discussions

Share New Research

Start New Discussion Sign up

SEARCH DISCUSSIONS

TOP 5 Research PUblications

About Quant League

The Open-Quant League is a quarterly competition between universities and investment clubs for the best performing strategy. Previous quarter's code is open-sourced, and competitors must adapt to survive.

competition rules

See the competition code of conduct and rules for participation in prizes.

Read Rules

previous competitions

Browse strategies and organization entries from previous quarter's competitions.

STRATEGY

284,900 Quants.

Become a Quant

VOTE FOR UPCOMING FEATURES

Share your input and vote on our future direction.

LEAN Roadmap

Create an account on QuantConnect for the latest delivered to your inbox.

Question regarding Coarse Universe Selection and lean CLI

Thanks for adding ```failed-data-requests``` and ```succeeded-data-requests``` files to lean cli backtests, much helpful!

I am building a strat that using ```CoarseSelectionFunction``` and executing in minute data.

I have noticed in the ```failed-data-requests``` file that whenever a ticker is once added to the Universe then LEAN is trying to access its minute data even though it is not actually in the current trading day (i.e. removed for today).
I am downloading only the necessary minute data selected by the previous day DollarVolume so I don't have all of the tickers minute data.

Why is that happening?

Update Backtest

person upvoted this people upvoted this

Super Richman

| |

Accepted Answer

Update Backtest

Notebook

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Louis Szeto

STAFF ,

Hi Super Richman

You'll also need the coarse universe data downloaded for that functionality. Either use the cli data downloader helper to download in advance, or download the needed data alongside with the backtest.

Best
Louis

Super Richman

290 ,

Thanks Louis. Coarse folder is in place as well as custom mapping and factor files according to the tickmap specification. the algo purpose is just to test which data is accessed since I am only downloading the relevant minute data.

/*
 * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
 * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
*/

using QuantConnect.Interfaces;
using System.Collections.Generic;
using System.Linq;
using QuantConnect.Data.Market;
using QuantConnect.Data.UniverseSelection;
using QuantConnect.Orders;
using System.IO;
using System.Collections;
using System;
using QuantConnect.Packets;

namespace QuantConnect.Algorithm.CSharp
{
    /// <summary>
    /// In this algorithm we demonstrate how to use the coarse fundamental data to
    /// define a universe as the top dollar volume
    /// </summary>
    /// <meta name="tag" content="using data" />
    /// <meta name="tag" content="universes" />
    /// <meta name="tag" content="coarse universes" />
    /// <meta name="tag" content="regression test" />
    public class CoarseFundamentalTop3Algorithm : QCAlgorithm, IRegressionAlgorithmDefinition
    {
        private const int NumberOfSymbols = 9;       
        private const string strFolderName = "QCiSAweSome";

        // initialize our changes to nothing
        private SecurityChanges _changes = SecurityChanges.None;
        TradingCalendar tc;
        ArrayList alCurrTickers;

        public override void Initialize()
        {
            alCurrTickers = new ArrayList();
            UniverseSettings.Resolution = Resolution.Minute;
            SetStartDate(2023, 1, 4); // Set Start Date // Day before since our universe need to be saved
            SetEndDate(2023, 1, 14); // Set end Date
            SetCash(100000);
            // this add universe method accepts a single parameter that is a function that
            // accepts an IEnumerable<CoarseFundamental> and returns IEnumerable<Symbol>
            AddUniverse(CoarseSelectionFunction);
        }

        // sort the data by daily dollar volume and take the top 'NumberOfSymbols'
        public static IEnumerable<Symbol> CoarseSelectionFunction(IEnumerable<CoarseFundamental> coarse)
        {
            // sort descending by daily dollar volume
            var sortedByDollarVolume = coarse.OrderByDescending(x => x.DollarVolume);

            // take the top entries from our sorted collection
            var top = sortedByDollarVolume.Take(NumberOfSymbols);

            var symbols = top.Select(x => x.Symbol.Value);            
            
            // we need to return only the symbol objects
            return top.Select(x => x.Symbol);
        }

        //Data Event Handler: New data arrives here. "TradeBars" type is a dictionary of strings so you can access it by symbol.
        public void OnData(TradeBars data)
        {
            // This algo is just for data check so nothing to do here
        }

        // this event fires whenever we have changes to our universe
        public override void OnSecuritiesChanged(SecurityChanges changes)
        {
            Console.WriteLine("Added:");
            foreach (var added in changes.AddedSecurities) { Console.WriteLine(added.Symbol.ID.ToString() + "\t" + added.Symbol.Value.ToString()); }
            Console.WriteLine("Removed:");
            foreach (var removed in changes.RemovedSecurities) { Console.WriteLine(removed.Symbol.ID.ToString() + "\t" + removed.Symbol.Value.ToString()); }
        }

        public override void OnOrderEvent(OrderEvent fill)
        {
           // Log($"OnOrderEvent({UtcTime:o}):: {fill}");
        }

        /// <summary>
        /// This is used by the regression test system to indicate if the open source Lean repository has the required data to run this algorithm.
        /// </summary>
        public bool CanRunLocally { get; } = true;

        /// <summary>
        /// This is used by the regression test system to indicate which languages this algorithm is written in.
        /// </summary>
        public Language[] Languages { get; } = { Language.CSharp, Language.Python };

        /// <summary>
        /// Data Points count of all timeslices of algorithm
        /// </summary>
        public long DataPoints => 78091;

        /// <summary>
        /// Data Points count of the algorithm history
        /// </summary>
        public int AlgorithmHistoryDataPoints => 0;

        /// <summary>
        /// This is used by the regression test system to indicate what the expected statistics are from running the algorithm
        /// </summary>
        public Dictionary<string, string> ExpectedStatistics => new Dictionary<string, string>
        {
            {"Total Trades", "12"},
            {"Average Win", "0.55%"},
            {"Average Loss", "-0.26%"},
            {"Compounding Annual Return", "16.717%"},
            {"Drawdown", "1.700%"},
            {"Expectancy", "0.850"},
            {"Net Profit", "0.637%"},
            {"Sharpe Ratio", "1.088"},
            {"Probabilistic Sharpe Ratio", "50.223%"},
            {"Loss Rate", "40%"},
            {"Win Rate", "60%"},
            {"Profit-Loss Ratio", "2.08"},
            {"Alpha", "0.198"},
            {"Beta", "0.741"},
            {"Annual Standard Deviation", "0.118"},
            {"Annual Variance", "0.014"},
            {"Information Ratio", "2.294"},
            {"Tracking Error", "0.097"},
            {"Treynor Ratio", "0.173"},
            {"Total Fees", "$27.94"},
            {"Estimated Strategy Capacity", "$200000000.00"},
            {"Lowest Capacity Asset", "AAPL R735QTJ8XC9X"},
            {"Fitness Score", "0.28"},
            {"Kelly Criterion Estimate", "0"},
            {"Kelly Criterion Probability Value", "0"},
            {"Sortino Ratio", "3"},
            {"Return Over Maximum Drawdown", "9.559"},
            {"Portfolio Turnover", "0.308"},
            {"Total Insights Generated", "0"},
            {"Total Insights Closed", "0"},
            {"Total Insights Analysis Completed", "0"},
            {"Long Insight Count", "0"},
            {"Short Insight Count", "0"},
            {"Long/Short Ratio", "100%"},
            {"Estimated Monthly Alpha Value", "$0"},
            {"Total Accumulated Estimated Alpha Value", "$0"},
            {"Mean Population Estimated Insight Value", "$0"},
            {"Mean Population Direction", "0%"},
            {"Mean Population Magnitude", "0%"},
            {"Rolling Averaged Population Direction", "0%"},
            {"Rolling Averaged Population Magnitude", "0%"},
            {"OrderListHash", "de456413f89396bd6f920686219ed0a5"}
        };
    }
}

Louis Szeto

STAFF ,

Hi Superrich Man

Sorry for the delay. Could you please also tell us which files you have in your local data directory? It will help a lot on the debugging since we cannot reproduce your problem.

Best
Louis

Super Richman

290 ,

Hi Louis,

Thanks for replying! Delay is always good, I hope you were on vacation or did something fun ;)

I am adding some files that will help you understand, I hope it will be good for you.

succeeded-data-requests-20230118140038721.txt

/equity/usa/fundamental/coarse/20230103.csv
/equity/usa/fundamental/coarse/20230104.csv
/equity/usa/fundamental/coarse/20230105.csv
/equity/usa/fundamental/coarse/20230106.csv
/equity/usa/fundamental/coarse/20230109.csv
/equity/usa/fundamental/coarse/20230110.csv
/equity/usa/fundamental/coarse/20230111.csv
/equity/usa/fundamental/coarse/20230112.csv
/equity/usa/fundamental/coarse/20230113.csv
/equity/usa/minute/aapl/20230104_trade.zip
/equity/usa/minute/pg/20230104_trade.zip
/equity/usa/minute/pg/20230104_quote.zip
/equity/usa/minute/aapl/20230104_quote.zip
/equity/usa/minute/msft/20230104_quote.zip
/equity/usa/minute/tsla/20230104_trade.zip
/equity/usa/minute/cvx/20230104_quote.zip
/equity/usa/minute/amzn/20230104_trade.zip
/equity/usa/minute/msft/20230104_trade.zip
/equity/usa/minute/tsla/20230104_quote.zip
/equity/usa/minute/amzn/20230104_quote.zip
/equity/usa/minute/meta/20230104_quote.zip
/equity/usa/minute/meta/20230104_trade.zip
/equity/usa/minute/nvda/20230104_quote.zip
/equity/usa/minute/cvx/20230104_trade.zip
/equity/usa/minute/nvda/20230104_trade.zip
/equity/usa/minute/meta/20230105_trade.zip
/equity/usa/minute/nvda/20230105_quote.zip
/equity/usa/minute/amzn/20230105_trade.zip
/equity/usa/minute/cvx/20230105_trade.zip
/equity/usa/minute/nvda/20230105_trade.zip
/equity/usa/minute/meta/20230105_quote.zip
/equity/usa/minute/tsla/20230105_trade.zip
/equity/usa/minute/cvx/20230105_quote.zip
/equity/usa/minute/msft/20230105_quote.zip
/equity/usa/minute/tsla/20230105_quote.zip
/equity/usa/minute/amzn/20230105_quote.zip
/equity/usa/minute/aapl/20230105_quote.zip
/equity/usa/minute/pg/20230109_quote.zip
/equity/usa/minute/pg/20230109_trade.zip
/equity/usa/minute/jnj/20230104_trade.zip
/equity/usa/minute/jnj/20230104_quote.zip
/equity/usa/minute/nvda/20230106_trade.zip
/equity/usa/minute/amzn/20230106_trade.zip
/equity/usa/minute/meta/20230106_quote.zip
/equity/usa/minute/tsla/20230106_quote.zip
/equity/usa/minute/msft/20230106_quote.zip
/equity/usa/minute/nvda/20230106_quote.zip
/equity/usa/minute/cvx/20230106_trade.zip
/equity/usa/minute/tsla/20230106_trade.zip
/equity/usa/minute/msft/20230105_trade.zip
/equity/usa/minute/amzn/20230106_quote.zip
/equity/usa/minute/cvx/20230106_quote.zip
/equity/usa/minute/aapl/20230105_trade.zip
/equity/usa/minute/aapl/20230106_quote.zip
/equity/usa/minute/meta/20230106_trade.zip
/equity/usa/hour/spy.zip
/equity/usa/minute/jnj/20230105_trade.zip
/equity/usa/minute/jnj/20230105_quote.zip
/equity/usa/minute/nvda/20230109_trade.zip
/equity/usa/minute/amzn/20230109_trade.zip
/equity/usa/minute/tsla/20230109_quote.zip
/equity/usa/minute/msft/20230109_quote.zip
/equity/usa/minute/cvx/20230109_trade.zip
/equity/usa/minute/nvda/20230109_quote.zip
/equity/usa/minute/tsla/20230109_trade.zip
/equity/usa/minute/meta/20230109_quote.zip
/equity/usa/minute/cvx/20230109_quote.zip
/equity/usa/minute/amzn/20230109_quote.zip
/equity/usa/minute/meta/20230109_trade.zip
/equity/usa/minute/aapl/20230109_quote.zip
/equity/usa/minute/aapl/20230106_trade.zip
/equity/usa/minute/msft/20230106_trade.zip
/equity/usa/minute/jnj/20230106_quote.zip
/equity/usa/minute/jnj/20230106_trade.zip
/equity/usa/minute/v/20230105_trade.zip
/equity/usa/minute/v/20230105_quote.zip
/equity/usa/minute/v/20230106_trade.zip
/equity/usa/minute/v/20230106_quote.zip
/equity/usa/minute/msft/20230109_trade.zip
/equity/usa/minute/tsla/20230110_trade.zip
/equity/usa/minute/amzn/20230110_quote.zip
/equity/usa/minute/meta/20230110_trade.zip
/equity/usa/minute/cvx/20230110_quote.zip
/equity/usa/minute/aapl/20230110_quote.zip
/equity/usa/minute/nvda/20230110_trade.zip
/equity/usa/minute/jnj/20230110_trade.zip
/equity/usa/minute/jnj/20230110_quote.zip
/equity/usa/minute/amzn/20230110_trade.zip
/equity/usa/minute/tsla/20230110_quote.zip
/equity/usa/minute/msft/20230110_quote.zip
/equity/usa/minute/cvx/20230110_trade.zip
/equity/usa/minute/v/20230109_trade.zip
/equity/usa/minute/aapl/20230109_trade.zip
/equity/usa/minute/v/20230109_quote.zip
/equity/usa/minute/aapl/20230111_quote.zip
/equity/usa/minute/nvda/20230111_trade.zip
/equity/usa/minute/aapl/20230110_trade.zip
/equity/usa/minute/msft/20230111_quote.zip
/equity/usa/minute/cvx/20230111_trade.zip
/equity/usa/minute/amzn/20230111_trade.zip
/equity/usa/minute/meta/20230110_quote.zip
/equity/usa/minute/tsla/20230111_quote.zip
/equity/usa/minute/nvda/20230110_quote.zip
/equity/usa/minute/msft/20230110_trade.zip
/equity/usa/minute/v/20230110_quote.zip
/equity/usa/minute/v/20230110_trade.zip
/equity/usa/minute/aapl/20230111_trade.zip
/equity/usa/minute/msft/20230111_trade.zip
/equity/usa/minute/pg/20230111_trade.zip
/equity/usa/minute/pg/20230111_quote.zip
/equity/usa/minute/cvx/20230111_quote.zip
/equity/usa/minute/meta/20230111_trade.zip
/equity/usa/minute/aapl/20230112_quote.zip
/equity/usa/minute/nvda/20230112_trade.zip
/equity/usa/minute/aapl/20230112_trade.zip
/equity/usa/minute/amzn/20230112_trade.zip
/equity/usa/minute/cvx/20230112_trade.zip
/equity/usa/minute/tsla/20230112_quote.zip
/equity/usa/minute/msft/20230112_quote.zip
/equity/usa/minute/msft/20230112_trade.zip
/equity/usa/minute/meta/20230112_trade.zip
/equity/usa/minute/cvx/20230112_quote.zip
/equity/usa/minute/tsla/20230111_trade.zip
/equity/usa/minute/nvda/20230111_quote.zip
/equity/usa/minute/amzn/20230111_quote.zip
/equity/usa/minute/v/20230111_trade.zip
/equity/usa/minute/amzn/20230112_quote.zip
/equity/usa/minute/v/20230111_quote.zip
/equity/usa/minute/meta/20230111_quote.zip
/equity/usa/minute/tsla/20230112_trade.zip
/equity/usa/minute/meta/20230112_quote.zip
/equity/usa/minute/cvx/20230113_quote.zip
/equity/usa/minute/nvda/20230112_quote.zip
/equity/usa/minute/v/20230112_trade.zip
/equity/usa/minute/amzn/20230113_quote.zip
/equity/usa/minute/aapl/20230113_trade.zip
/equity/usa/minute/nvda/20230113_trade.zip
/equity/usa/minute/v/20230112_quote.zip
/equity/usa/minute/amzn/20230113_trade.zip
/equity/usa/minute/cvx/20230113_trade.zip
/equity/usa/minute/pg/20230112_trade.zip
/equity/usa/minute/msft/20230113_trade.zip
/equity/usa/minute/tsla/20230113_quote.zip
/equity/usa/minute/meta/20230113_trade.zip
/equity/usa/minute/pg/20230112_quote.zip
/equity/usa/minute/msft/20230113_quote.zip
/equity/usa/minute/tsla/20230113_trade.zip
/equity/usa/minute/pg/20230113_trade.zip
/equity/usa/minute/nvda/20230113_quote.zip
/equity/usa/minute/meta/20230113_quote.zip
/equity/usa/minute/pg/20230113_quote.zip
/equity/usa/minute/aapl/20230113_quote.zip
/equity/usa/minute/aal/20230113_trade.zip
/equity/usa/minute/aal/20230113_quote.zip

failed-data-requests-20230118140038722.txt

/equity/usa/minute/pg/20230105_trade.zip
/equity/usa/minute/pg/20230105_quote.zip
/equity/usa/minute/pg/20230106_quote.zip
/equity/usa/minute/pg/20230106_trade.zip
/equity/usa/minute/jnj/20230109_trade.zip
/equity/usa/minute/jnj/20230109_quote.zip
/equity/usa/minute/pg/20230110_trade.zip
/equity/usa/minute/pg/20230110_quote.zip
/equity/usa/minute/jnj/20230111_trade.zip
/equity/usa/minute/jnj/20230111_quote.zip
/equity/usa/minute/jnj/20230112_trade.zip
/equity/usa/minute/jnj/20230112_quote.zip
/equity/usa/minute/jnj/20230113_trade.zip
/equity/usa/minute/jnj/20230113_quote.zip
/equity/usa/minute/v/20230113_trade.zip
/equity/usa/minute/v/20230113_quote.zip

data-monitor-report-20230118140045884.json

{"succeeded-data-requests-count":154,"failed-data-requests-count":16,"succeeded-universe-data-requests-count":9,"failed-universe-data-requests-count":0,"total-data-requests-count":170,"failed-data-requests-percentage":9.0,"total-universe-data-requests-count":9,"failed-universe-data-requests-percentage":0.0,"data-request-rates":[0.0]}

Partial Log:

2023-01-04 09:31:00 2023-01-04
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	AAPL SRW990ATUCMD	126.8900
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	AMZN SRW990ATUCMD	86.5100
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	CVX SRW990ATUCMD	170.9500
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	JNJ SRW990ATUCMD	178.8900
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	FB V6OIPNZEM8V9	127.3800
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	MSFT SRW990ATUCMD	232.0200
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	NVDA SRW990ATUCMD	145.6600
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	PG SRW990ATUCMD	151.2500
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	TSLA UNV35CB9Z6G5	109.1100
2023-01-05 00:00:00 Added:
2023-01-05 00:00:00 V U12VRGLO8PR9	V
2023-01-05 00:00:00 Removed:
2023-01-05 00:00:00 PG SRW990ATUCMD	PG
2023-01-05 09:31:00 2023-01-05

As deducted from log, for 5.1 PG is not needed, and yet we have a failed data request for minute data for PG

I have all the files needed per day of Coarse by DollarVolume - so the coarse slected top 10 minute data will be downloaded for tomorrow.

All of the “needed” data is in place, the failed requests are for days that are not in the coarse and should not actually be requested.

To reproduce:

Create a strat which uses coarse selection
load a minute data the day after (automatically by coarse strat)
the day after the stock was “removed” see the failed (or succeeded) data request for minute data even if unnecessary.

Feel free to correct me if I am wrong :P What do you think?

Louis Szeto

STAFF ,

Hi Super Richman

Do you have the “failed requests” files in position? Maybe you'll also need `lean data download … --overwrite` if some existing/random generated files are there.

Best
Louis

Super Richman

290 ,

Hi again Louis, I deliberately do not have the unneeded files since they are not required for the strategy.

Louis Szeto

STAFF ,

Hi Super Richman

I mean you have the needed files, but are they holding the data required? Like `equity/usa/daily/spy.csv`, it might only hold a period of data which is insufficient for your backtest. You'll need the --overwrite option to download and replace this outdated file.

Best
Louis

Super Richman INVESTOR

Update Backtest

Notebook

person upvoted this people upvoted this

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!

Organization

Organization Website

Update Competition

Team

Clone Strategy

Copy this strategy code to your QuantConnect account and deploy it live with your brokerage.

Clone

Previous Ranking

Browse strategies and organization entries from previous quarter's competitions.

Author:

Platform

Radically Open-Source Algorithmic Trading Engine

Join Our Discord Channel

Quarterly Open-Source Trading Competition

Draft Discussions

Bookmarked Discussions

SEARCH DISCUSSIONS

TOP 5 Research PUblications

About Quant League

competition rules

previous competitions

284,900 Quants.

VOTE FOR UPCOMING FEATURES

Question regarding Coarse Universe Selection and lean CLI

Organization

Team

Clone Strategy

Previous Ranking

IN THIS RESEARCH

PARTICIPANTS

Discussion Awards

Actions

Join QuantConnect for Free

Platform

SIGN IN

Radically Open-Source Algorithmic Trading Engine

Join Our Discord Channel

Quarterly Open-Source Trading Competition

Draft Discussions

Bookmarked Discussions

SEARCH DISCUSSIONS

TOP 5 Research PUblications

About Quant League

competition rules

previous competitions

284,900 Quants.

VOTE FOR UPCOMING FEATURES

Question regarding Coarse Universe Selection and lean CLI

Organization

Team

Clone Strategy

Previous Ranking

IN THIS RESEARCH

PARTICIPANTS

Discussion Awards

SHARE RESEARCH

SHARE DISCUSSION

SHARE ARTICLE

SHARE

Actions

Join QuantConnect for Free