I'm receiving the following error when trying to cycle through Fundamental data for the universe of symbols. This error started over the weekend and is continued to cause errors on Monday
System.Exception: N7parquet38ParquetInvalidOrCorruptedFileExceptionE (message: 'Invalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.') in ParquetFundamentalDataProvider.cs:line 321
---> QuantConnect.Exceptions.SystemExceptionInterpreter+SanitizedException: N7parquet38ParquetInvalidOrCorruptedFileExceptionE (message: 'Invalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.')
at ParquetSharp.ExceptionInfo.Check(IntPtr exceptionInfo)
at ParquetSharp.ExceptionInfo.Return[TArg0,TValue](IntPtr handle, TArg0 arg0, GetFunction`2 getter)
at ParquetSharp.ExceptionInfo.Return[TArg0,TValue](ParquetHandle handle, TArg0 arg0, GetFunction`2 getter)
at ParquetSharp.ParquetFileReader..ctor(Stream stream, ReaderProperties readerProperties, Boolean leaveOpen)
at QuantConnect.Data.Common.ParquetFundamentalDataProvider.TryGetSecurityIndex(DateTime time, SecurityIdentifier key, FundamentalProperty name, Int32& index) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/LeanCloud.Data/QuantConnect.Data.Common/ParquetFundamentalDataProvider.cs:line 321
at QuantConnect.Data.Common.ParquetFundamentalDataProvider.Get[T](DateTime time, SecurityIdentifier securityIdentifier, FundamentalProperty name) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/LeanCloud.Data/QuantConnect.Data.Common/ParquetFundamentalDataProvider.cs:line 136
at QuantConnect.Data.UniverseSelection.FundamentalService.Get[T](DateTime time, SecurityIdentifier securityIdentifier, FundamentalProperty name) in Common/Data/UniverseSelection/FundamentalService.cs:line 84
at QuantConnect.Data.Fundamental.ValuationRatios.get_BookValueYield() in Common/Data/Fundamental/Generated/ValuationRatios.cs:line 137
Code snippet
var fundamentals = new List<Fundamental>();
foreach (var equity in data.ETFConstituentsFiltered)
{
if (algorithm.Securities.TryGetValue(equity, out var security))
{
fundamentals.Add(security.Fundamentals);
}
}
universeEquities.UpdateFundamentals(fundamentals);
public void UpdateFundamentals(IEnumerable<Fundamental> fine)
{
foreach (var fineFundamental in fine.Where(x => x.HasFundamentalData))
{
var valuationRatios = fineFundamental.ValuationRatios;
var bookValueYield = valuationRatios.BookValueYield;
var totalYield = valuationRatios.TotalYield;
var symbolData = SymbolData[fineFundamental.Symbol];
symbolData.BookValueYield = bookValueYield;
symbolData.TotalYield = totalYield;
}
}
Mia Alissi
The error message you're seeing suggests that there's an issue with the Parquet file you're trying to read. Parquet is a columnar storage file format that is optimized for use with big data processing frameworks. The error message 'Invalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.' typically indicates that the file you're trying to read is either not a Parquet file or is corrupted.
However, based on the information provided in the extracts, it's not clear what could be causing this issue in your specific case. The code snippets you've shared are related to accessing and updating fundamental data, but they don't seem to directly involve reading from a Parquet file.
Unfortunately, without more information, it's difficult to provide a more specific solution. I would recommend checking the Parquet files you're trying to read to ensure they're not corrupted and are indeed Parquet files. If the issue persists, you might want to reach out to the QuantConnect community or support for more specific assistance.
Please note that the information provided is based on the extracts given and might not fully solve your issue. For more detailed help, consider providing more context or details about your problem.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Mark Fairbrother
As mentioned in the response I'm not accessing the Parquet files directly, simply using the Fundamentals property of the Securities[Symbol].Fundamentals.ValuationRatios.BookValueYield provided by the QuantConnect api (up until the weekend this has been working fine for nearly ~12 months). The code checks to ensure HasFundamentalData is true for the security yet still seems to be failing for all the Constituents of the SPY ETF Universe.
Mia Alissi
Based on the information provided in the extracts, it's important to note that not all assets have fundamental data. For example, ETFs like SPY do not have fundamental data. This is mentioned in the Corporate Fundamentals documentation.
You can check if fundamental data is available for an asset using the
HasFundamentalData
property. However, even ifHasFundamentalData
returns true, it doesn't necessarily mean that all types of fundamental data are available. For example, theValuationRatios.BookValueYield
property might not be available for all securities, even if they have some fundamental data.Here is how you can check if fundamental data is available for a security:
If you're trying to access fundamental data for ETFs like SPY, you might encounter issues because these types of securities typically do not have fundamental data. If you need to use fundamental data in your algorithm, you might need to consider using a different type of security that has this data available.
If the issue persists, it might be a problem with the data provider or a temporary issue with the QuantConnect platform. In this case, reaching out to QuantConnect support might be the best course of action.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Mark Fairbrother
Also note that this started failing when performing FineFundamental filtering of the universe with the same Parquet file error on the BookValueYield property so I tried moving the updates into the public override void OnData(Slice slice) method using the code above but it still fails with the same error.
Mia Alissi
Based on the information provided in the extracts, it's difficult to pinpoint the exact cause of the issue you're experiencing. The error message you're seeing is typically associated with issues reading a Parquet file, but it's not clear why this would be happening in your case if you're not directly interacting with Parquet files.
One possibility could be that there's an issue with the underlying data for the
BookValueYield
property. If the data for this property is stored in a Parquet file that is corrupted or not properly formatted, it could potentially cause this error when you try to access the property.However, without more information, it's difficult to provide a more specific solution. I would recommend reaching out to the QuantConnect community or support for more specific assistance. They might be able to provide more insight into why you're seeing this error and how to resolve it.
Please note that the information provided is based on the extracts given and might not fully solve your issue. For more detailed help, consider providing more context or details about your problem.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Mark Fairbrother
Issue appears to be specific to the BookValueYield property of the ValuationRatios since when I skip that I'm able to access the TotalYield property without any issues.
Mia Alissi
Based on the information provided in the extracts, it seems like there might be an issue with the
BookValueYield
property specifically. If you're able to access other properties likeTotalYield
without any issues, it suggests that the problem might be specific toBookValueYield
.It's possible that there's an issue with the underlying data for the
BookValueYield
property. If the data for this property is stored in a Parquet file that is corrupted or not properly formatted, it could potentially cause this error when you try to access the property.However, without more information, it's difficult to provide a more specific solution. I would recommend reaching out to the QuantConnect community or support for more specific assistance. They might be able to provide more insight into why you're seeing this error and how to resolve it.
Please note that the information provided is based on the extracts given and might not fully solve your issue. For more detailed help, consider providing more context or details about your problem.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Mark Fairbrother
I'm guessing this was probably a data issue but looks like it is working again
Mia Alissi
I'm glad to hear that it's working again! It's possible that there was a temporary issue with the data for the
BookValueYield
property. If you encounter similar issues in the future, don't hesitate to report them. QuantConnect has a system designed to track, fix, and notify users when data issues are fixed. You can report data issues through the Data Explorer Issues page. Remember to provide specific details like dates, times, contracts, and the type of issue to help the team resolve the issue more efficiently.The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Mark Fairbrother
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!