Back

New Data Source! Tiingo News Data

Hello Everyone! 

We're happy to share we've completed an integration with Tiingo to add their historical news data to QuantConnect's platform! This will enable a new suite of news events and sentiment style strategies able to quickly apply the news to trading!

The engineering behind our new custom data source integrations was pretty epic. We've worked behind the scenes to make it smooth for months - ensuring tickers were properly mapped, dealing with collisions of symbol objects and finally getting new data production ready!

This is an [Alpha Streams Approved] data source. For any alpha streams or competition use, the data is free to consume! For individual live trading, you will need to register for a paid account and get an API token on Tiingo.

With Tiingo you can pull in the news titles, descriptions, tags about the article and other companies mentioned in the news. We're excited to see the awesome new applications possible with the new data!

Check out the example below doing some basic NLP to assign sentiment to words of the news! See more information in the new Alternative Data / Tiingo News Documentation! 

Happy Coding,

Jared

 

 

Update Backtest






The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.



0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


This looks like a great data source. 

Will you also consider adding more text mining packages, such as gensim, fasttext?

A request for these packages has been sent to support. 

2

Great idea thank you Zhiqiang
1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


great!

1

Nice!

0

@Zhiqiang we merged in the new NLP processing packages. We had an issue with FastText so we couldn't get that in for this round but maybe next month. You can see the full list of supported libraries here.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Great. Thanks Jared.
0

Is this data available for Research? 

0

Yes! Research is now immediately updated with LEAN's latest version; so all new LEAN features are available in Research.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


What is the time resolution of news events? 

Is it delivered once daily at the close, or like instanteous tick data in backtesting?

1

@Kieran the data is "point in time" - streamed to your algorithm as it happens (so its closer to tick data).

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thanks for the addition of this capability! I have attempted to incorporate it in my Alpha, but the algo is running pretty slow with the addition of the Tiingo news. Why is this? Any advice on how to improve the speed? The Tiingo news section starts at line 92. 

from clr import AddReference
AddReference("System")
AddReference("QuantConnect.Algorithm")
AddReference("QuantConnect.Indicators")
AddReference("QuantConnect.Common")
from datetime import datetime, timedelta
import numpy as np
from System import *
from QuantConnect import *
from QuantConnect.Data import *
from QuantConnect.Algorithm import *
from QuantConnect.Indicators import *
from System.Collections.Generic import List
from QuantConnect.Algorithm.Framework import *
from QuantConnect.Algorithm.Framework.Alphas import *
from Risk.MaximumSectorExposureRiskManagementModel import MaximumSectorExposureRiskManagementModel
from QuantConnect.Data.Custom.Tiingo import *

class Compbasealgo(QCAlgorithm):

def Initialize(self):
'''Initialise the data and resolution required, as well as the cash and start-end dates for your algorithm. All algorithms must initialized.'''

self.SetStartDate(2015, 8, 4) # 5 years up to the submission date
self.SetCash(1000000) # Set $1m Strategy Cash to trade significant AUM
self.SetBenchmark('SPY') # SPY Benchmark
self.SetBrokerageModel(AlphaStreamsBrokerageModel())
self.UniverseSettings.Resolution = Resolution.Daily
self.AddAlpha(MixedAlphaModel())
self.SetRiskManagement(MaximumDrawdownPercentPerSecurityCustom(0.01))
self.SetExecution(ImmediateExecutionModel())
self.SetUniverseSelection(LiquidETFUniverse())
self.SetPortfolioConstruction(InsightWeightingPortfolioConstructionModel())

class MixedAlphaModel(AlphaModel):
def __init__(self,
fastPeriod = 12,
slowPeriod = 26,
signalPeriod = 9,
movingAverageType = MovingAverageType.Simple,
resolution = Resolution.Daily):
self.fastPeriod = fastPeriod
self.slowPeriod = slowPeriod
self.signalPeriod = signalPeriod
self.movingAverageType = movingAverageType
self.resolution = resolution
self.insightPeriod = Time.Multiply(Extensions.ToTimeSpan(resolution), fastPeriod)
self.bounceThresholdPercent = 0.1
self.symbolData2 = {}
resolutionString = Extensions.GetEnumString(resolution, Resolution)
movingAverageTypeString = Extensions.GetEnumString(movingAverageType, MovingAverageType)
self.Name = '{}({},{},{},{},{})'.format(self.__class__.__name__, fastPeriod, slowPeriod, signalPeriod, movingAverageTypeString, resolutionString)
self.direction = InsightDirection.Flat


self.newsData = {}
self.wordScores = {
"bad": -0.5, "good": 0.5, "negative": -0.5,
"great": 0.5, "growth": 0.5, "fail": -0.5,
"failed": -0.5, "success": 0.5, "nailed": 0.5,
"beat": 0.5, "missed": -0.5, "profitable": 0.5,
"beneficial": 0.5, "right": 0.5, "positive": 0.5,
"large":0.5, "attractive": 0.5, "sound": 0.5,
"excellent": 0.5, "wrong": -0.5, "unproductive": -0.5,
"lose": -0.5, "missing": -0.5, "mishandled": -0.5,
"un_lucrative": -0.5, "up": 0.5, "down": -0.5,
"unproductive": -0.5, "poor": -0.5, "wrong": -0.5,
"worthwhile": 0.5, "lucrative": 0.5, "solid": 0.5,
"beat": 0.5, "missed": -0.5

}
self.currentWindow=0.1
self.sentimentMean=0.1
self.symbol2="EURUSD"

def Update(self, algorithm, data):

insights = []

for key, sd in self.symbolData2.items():
self.symbol=sd.Security.Symbol
self.history = algorithm.History(self.symbol, 100, Resolution.Daily)
if not self.history.empty:
self.historySymbol = str(self.symbol)
try:
self.symbolData2[self.symbol].WarmUpIndicators2(self.history)#.loc[self.historySymbol])
except KeyError:
pass
#if sd.Security.Price == 0:
# continue

tiingoNews=algorithm.Securities[self.symbol].Data.GetAll(TiingoNews)
for article in tiingoNews:
words = article.Description.lower().split(" ")
score = sum([self.wordScores[word] for word in words
if word in self.wordScores])
if score == 0.0:
continue
symbol = article.Symbol.Underlying
self.newsData[symbol].Window.Add(score)
algorithm.Debug("score "+ str(score))
algorithm.Debug("Count "+ str(self.newsData[symbol].Window.Count))
if self.newsData[symbol].Window.Count < 3:
return insights
self.currentWindow = self.newsData[symbol].Window[0]
self.sentimentMean = sum(self.newsData[symbol].Window)/self.newsData[symbol].Window.Count
algorithm.Debug("Sentiment "+ str(symbol))
algorithm.Debug("MACD "+ str(self.symbol))
algorithm.Debug("score "+ str(score))
algorithm.Debug("Count "+ str(self.newsData[symbol].Window.Count))

normalized_signal = sd.MACD.Signal.Current.Value / sd.Security.Price
normalized_slope=sd.Reg.Slope.Current.Value/sd.Security.Price
algorithm.Debug("MACD "+ str(normalized_signal))
algorithm.Debug("slope "+ str(sd.Reg.Slope.Current.Value))
algorithm.Debug("currentwindow "+ str(self.currentWindow))
algorithm.Debug("sentimentmean "+ str(self.sentimentMean))
#if self.direction == sd.PreviousDirection:
# continue
if normalized_signal > self.bounceThresholdPercent and normalized_slope>0 and self.currentWindow > (self.sentimentMean*1.1):
insight = Insight.Price(sd.Security.Symbol, timedelta(hours = 1), InsightDirection.Up, None, None, None, 0.05)
self.direction = InsightDirection.Up
insights.append(insight)
sd.PreviousDirection = insight.Direction

elif normalized_signal < -self.bounceThresholdPercent and normalized_slope<-0 and self.currentWindow < -1:
self.direction = InsightDirection.Down
insight = Insight.Price(sd.Security.Symbol, timedelta(hours = 1), InsightDirection.Down, None, None, None, 0.05)
insights.append(insight)
sd.PreviousDirection = insight.Direction

elif normalized_signal>-self.bounceThresholdPercent and normalized_signal<self.bounceThresholdPercent:
self.direction = InsightDirection.Flat
insight = Insight.Price(sd.Security.Symbol, timedelta(hours = 1), InsightDirection.Flat)
insights.append(insight)

return insights


def OnSecuritiesChanged(self, algorithm, changes):
for added in changes.AddedSecurities:
self.symbol=added.Symbol
self.symbolData2[added.Symbol] = SymbolData2(algorithm, added, self.fastPeriod, self.slowPeriod, self.signalPeriod, self.movingAverageType, self.resolution)

for removed in changes.RemovedSecurities:
data = self.symbolData2.pop(removed.Symbol, None)
if data is not None:
algorithm.SubscriptionManager.RemoveConsolidator(removed.Symbol, data.Consolidator)

for security in changes.AddedSecurities:
symbol = security.Symbol
newsAsset = algorithm.AddData(TiingoNews, symbol)
self.newsData[symbol] = NewsData(newsAsset.Symbol)

for security in changes.RemovedSecurities:
newsData = self.newsData.pop(security.Symbol, None)
if newsData is not None:
algorithm.RemoveSecurity(newsData.Symbol)

class SymbolData2(object):
def __init__(self, algorithm, security, fastPeriod, slowPeriod, signalPeriod, movingAverageType, resolution):
self.Security = security
self.MACD = MovingAverageConvergenceDivergence(fastPeriod, slowPeriod, signalPeriod, movingAverageType)
self.Reg = RegressionChannel(50, 2)
self.Consolidator = algorithm.ResolveConsolidator(security.Symbol, resolution)
algorithm.RegisterIndicator(security.Symbol, self.MACD, self.Consolidator)
algorithm.RegisterIndicator(security.Symbol, self.Reg, self.Consolidator)
self.PreviousDirection = None

def WarmUpIndicators2(self, history):
for index, row in history.loc[str(self.Security.Symbol)].iterrows():
self.MACD.Update(index, row["close"])
self.Reg.Update(index, row["close"])


class MaximumDrawdownPercentPerSecurityCustom(RiskManagementModel):

def __init__(self, maximumDrawdownPercent = 0.001, reenter_after_days = 1):
self.maximumDrawdownPercent = -abs(maximumDrawdownPercent)
self.maximumProfitPercent = abs(0.002)
self.liquidated_dates_by_symbol = {}
self.reenter_after_days = timedelta(reenter_after_days)

def ManageRisk(self, algorithm, targets):
symbols_to_pop = []
for symbol, liquidation_date in self.liquidated_dates_by_symbol.items():
if algorithm.Time - liquidation_date >= self.reenter_after_days:
symbols_to_pop.append(symbol)
for symbol in symbols_to_pop:
self.liquidated_dates_by_symbol.pop(symbol, None)

targets = []
for kvp in algorithm.Securities:
security = kvp.Value

pnl = security.Holdings.UnrealizedProfitPercent
if pnl > self.maximumProfitPercent or pnl < self.maximumDrawdownPercent or security.Symbol in self.liquidated_dates_by_symbol:
# liquidate
targets.append(PortfolioTarget(security.Symbol, 0))
if algorithm.Securities[security.Symbol].Invested:
self.liquidated_dates_by_symbol[security.Symbol] = algorithm.Time

return targets

class NewsData():
def __init__(self, symbol):
self.Symbol = symbol
self.Window = RollingWindow[float](100)

 

0

Hi Michael,

The thing slowing down the algorithm the most is the fact that it makes a History call for each security in the universe when a new data point is passed to the Update method. To resolve this, we can move the indicator warm-up logic to inside the SymbolData2 constructor.

class SymbolData2(object):
def __init__(...):
...
history = algorithm.History(security.Symbol, 100, Resolution.Daily)
self.WarmUpIndicators2(history)

With this change, we just need to add

if not (sd.MACD.IsReady and sd.Reg.IsReady):
continue

to the Update method after the algorithm analyzes the TiingoNews data.

Secondly, the backtesting speed can be increase by removing most of the imports from the top of the file. Without explicitly importing the python modules, LEAN will utilize the C# equivalents.

Finally, we can speed up the OnSecuritiesChanged method by only looping through `changes.AddedSecurities` and `changes.RemovedSecurities` once.

See the attached backtest which implements the suggestions above.

Best,
Derek Melchin

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Wow it's way faster! Thanks for the fundamental knowledge!

0

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed