Calculate technical indicators every period using ta_lib package

Back

Hi,

I'm new to QC platform.

I want to incorporate my ML model saved as pkl file in backtesting and later live trading on quantconnect.

My first ptoblem is that I don't know how to calculate ta_lib indicators. I create simple example that does not work:

import pandas as pd
import numpy as np
from talib.abstract import (
DEMA, EMA, MIDPRICE, SMA, T3, TEMA, TRIMA, WMA,
ADX, ADXR, AROONOSC, BOP, CMO, DX, MFI, MINUS_DM, MOM, ROC, RSI,
TRIX , WILLR, ATR, NATR, BBANDS, AROON, STOCHRSI,
HT_TRENDLINE, AD, OBV, HT_DCPERIOD, HT_DCPHASE, HT_TRENDMODE,
TRANGE, AVGPRICE, MEDPRICE, TYPPRICE, WCLPRICE, ULTOSC,
MAMA, SAR, SAREXT, APO, MACD, ADOSC,
HT_PHASOR, HT_SINE, STOCHF, STOCH
)

# GLOBALS
periods = [5, 30, 60, 300, 480, 2400, 12000, 96000]

class CalibratedResistanceAtmosphericScrubbers(QCAlgorithm):

def Initialize(self):
self.SetStartDate(2019, 12, 31) # Set Start Date
self.SetCash(100000) # Set Strategy Cash
self.AddEquity("SPY", Resolution.Minute)

model = self.Download("https://github.com/MislavSag/trademl/blob/master/trademl/modeling/rf_model.pkl")


def OnData(self, data):
'''OnData event is the primary entry point for your algorithm. Each new data point will be pumped in here.
Arguments:
data: Slice object keyed by symbol containing the stock data
'''

open_ = self.Securities["SPY"].Open
high_ = self.Securities["SPY"].High
low_ = self.Securities["SPY"].Low
close_ = self.Securities["SPY"].Close
volume_ = self.Securities["SPY"].Volume

dema = DEMA(np.array(close_), 30)

self.Debug(f'DEMA value is equal to: {dema}')

If I ran this, it returns an error:

Runtime Error: TypeError : Argument 'real' has incorrect type (expected numpy.ndarray, got NoneType) at OnData in main.py:line 81 TypeError : Argument 'real' has incorrect type (expected numpy.ndarray, got NoneType)

 

Update Backtest







 
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hi Mislav,

I've attached a backtest which shows how we can implement an algorithm that utilizes talib. Instead of constructing a DEMA object with an array containing 1 element, we need to ensure we pass DEMA an array that is atleast twice in length of the timeperiods parameter (30). We can accomplish this by saving a rolling window of closing prices and pass this rolling window to DEMA whenever we want the indicator calculated.

Since we are needing a rolling window, we should also make sure we warm up the rolling window before the algorithm starts trading. As our documentation shows, we can accomplish this with

self.SetWarmUp(dema_period * 2)

in this situation.

See that attached backtest for the full solution file.

Best,
Derek Melchin

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Derek Melchin ,

Thank you, thats very helpfull. 

Why is warm up period two times longer than lookback, shouldn't it be the same as lookback? We wait wor 3 periods and than we can strat calculating indicator. It's not so importnant in my case because I have one minute resolution (so lots of data will go in the backtest), I am jsut curious.

Now, if you can help me with additional problem it would solve most of my problems!. I have written a function that automaticly calculates lots of features (cca 315 of them) based on pandas data frame (I think it can be a numpy too.). here are the functions (you don't have to inspect this, please cont read above);

def add_ind(ohlcv, f, n, periods):
"""
Add technical indicator to pd.DataFrame

Parameters
----------
f : function
function from ta_lib package.
n : str
Nme prefix.

Returns
-------
pd.Data.Frame.

"""
ind = pd.concat([f(ohlcv, p).rename(n + str(p)) for p in periods],
axis=1)
return ind


def add_ind_df(ohlcv, f, n, periods):
"""
Add technical indicator to pd.DataFrame when indicator has multiplie
outputs.

Parameters
----------
f : function
function from ta_lib package.
n : str
Nme prefix.

Returns
-------
pd.Data.Frame.

"""
ind = [f(ohlcv, p).add_prefix((f._Function__namestr + '_' + str(p) + '_'))
for p in periods]
# ind = [f(ohlcv, p).
# set_axis((f._Function__namestr + '_' +
# pd.Series(f.output_names) + '_' + str(p)), axis=1)
# for p in periods]
ind = pd.concat(ind, axis=1)
return ind


def add_technical_indicators(data, periods):
"""Add tecnical indicators as featues.

Arguments:
data {pd.DataFrame} -- Pandas data frame with OHLC data
periods {list} -- List that contain periods as arguments.

Returns:
pd.dataFrame -- Pandas data frame with additional indicators
"""
# add technical indicators for variuos periods when ind has 1 output
indsList = [DEMA, EMA, MIDPRICE, SMA, T3, # MIDPOINT
TEMA, TRIMA, WMA, # KAMA memory intensive!
ADX, ADXR, AROONOSC, BOP, CMO, DX, MFI, MINUS_DM, MOM, ROC, RSI,
TRIX , WILLR, # CCI NE RADI (VALJDA)
ATR, NATR]
inds = [add_ind(data, f, f._Function__name.decode('ascii'), periods)
for f in indsList]
inds = pd.concat(inds, axis=1)
data = pd.concat([data, inds], axis=1)

# add technical indicators for variuos periods when ind has multiplie
# outputs
indsList = [BBANDS, AROON, STOCHRSI]
inds = [add_ind_df(data, f, f._Function__name.decode('ascii'), periods)
for f in indsList]
inds = pd.concat(inds, axis=1)
data = pd.concat([data, inds], axis=1)

# add tecnical indicators with no arguments
indsList = [HT_TRENDLINE, AD, OBV, HT_DCPERIOD, HT_DCPHASE, HT_TRENDMODE,
TRANGE, AVGPRICE, MEDPRICE, TYPPRICE, WCLPRICE,
ULTOSC]
inds = [f(data).rename(f._Function__name.decode('ascii')) for f in indsList]
inds = pd.concat(inds, axis=1)
data = pd.concat([data, inds], axis=1)

# add other indicators
data[['MAMA', 'FAMA']] = MAMA(data) # MAVP ne radi
data[['MAMA_25', 'FAMA_25']] = MAMA(data, fastlimit=0.25, slowlimit=0.02) # MAVP ne radi
data[['MAMA_5', 'FAMA_5']] = MAMA(data, fastlimit=0.5, slowlimit=0.05) # MAVP ne radi
data['SAR'] = SAR(data)
data['SAR_1'] = SAR(data, acceleration=0.01, maximum=0.01)
data['SAR_2'] = SAR(data, acceleration=0.02, maximum=0.02)
data['SAREXT'] = SAREXT(data)
startvalue, offsetonreverse, accelerationinitlong, accelerationlong,\
accelerationmaxlong, accelerationinitshort, accelerationshort,\
accelerationmaxshort = np.random.uniform(low=0.01, high=0.4, size=8)
data['SAREXT_rand'] = SAREXT(data, startvalue=startvalue,
offsetonreverse=offsetonreverse,
accelerationinitlong=accelerationinitlong,
accelerationlong=accelerationlong,
accelerationmaxlong=accelerationmaxlong,
accelerationinitshort=accelerationinitshort,
accelerationshort=accelerationshort,
accelerationmaxshort=accelerationmaxshort)
data['APO'] = APO(data)
data['APO_1'] = APO(data, fastperiod=24, slowperiod=52, matype=0)
data['APO_2'] = APO(data, fastperiod=50, slowperiod=100, matype=0)
data['APO_3'] = APO(data, fastperiod=100, slowperiod=200, matype=0)
data['APO_4'] = APO(data, fastperiod=200, slowperiod=400, matype=0)
data['APO_5'] = APO(data, fastperiod=12000, slowperiod=24000, matype=0)
data['ADOSC'] = ADOSC(data)
data[['MACD', 'MACDSIGNAL', 'MACDHIST']] = MACD (data)
# data[['MACD_24', 'MACDSIGNAL_24', 'MACDHIST_24']] = MACD (data,
# fastperiod=24,
# slowperiod=52,
# signalperiod=18)
# data[['MACD_48', 'MACDSIGNAL_48', 'MACDHIST_48']] = MACD (data,
# fastperiod=48,
# slowperiod=104,
# signalperiod=36)
# data[['MACD_200', 'MACDSIGNAL_200', 'MACDHIST_200']] = MACD (data,
# fastperiod=200,
# slowperiod=300,
# signalperiod=50)
# data[['MACDEXT', 'MACDEXTSIGNAL', 'MACDEXTHIST']] = MACD (data)
# data[['MACDEXT_', 'MACDEXTSIGNAL_', 'MACDEXTHIST_']] = MACD (data,
# fastperiod=200, fastmatype=0,
# slowperiod=100,
# slowmatype=0,
# signalperiod=50,
# signalmatype=0)
# data[['MACDFIX', 'MACDFIX SIGNAL', 'MACDFIXHIST']] = MACDFIX(data)
# data[['MACDFIX_18', 'MACDFIX SIGNAL_18',
# 'MACDFIXHIST_18']] = MACDFIX(data, 18)
# data[['MACDFIX_50', 'MACDFIX SIGNAL_50',
# 'MACDFIXHIST_50']] = MACDFIX(data, 50)
# data[['MACDFIX_200', 'MACDFIX SIGNAL_200',
# 'MACDFIXHIST_200']] = MACDFIX(data, 200)
# data[['MACDFIX_12000', 'MACDFIX SIGNAL_12000',
# 'MACDFIXHIST_12000']] = MACDFIX(data, 12000)
data[['inphase', 'quadrature']] = HT_PHASOR(data)
data[['sine', 'leadsine']] = HT_SINE(data)
data[['fastk', 'fastd']]= STOCHF(data)
data[['fastk_20', 'fastd_20']]= STOCHF(data, fastk_period=20, fastd_period=9, fastd_matype=0)
data[['fastk_200', 'fastd_200']]= STOCHF(data, fastk_period=200, fastd_period=80, fastd_matype=0)
data[['fastk_3600', 'fastd_3600']]= STOCHF(data, fastk_period=3600, fastd_period=400, fastd_matype=0)
data[['slowk', 'slowd ']]= STOCH(data)
data[['slowk_30', 'slowd_30']]= STOCH(data, fastk_period=30, slowk_period=15,
slowk_matype=0, slowd_period=9, slowd_matype=0)

return data

Now, when I want to calculate the functions, I just execute:

periods = [5, 30, 60, 300, 480, 2400, 12000, 96000]

data = add_technical_indicators(data, periods)

where data is pandas dataframe with OHLCV columns. Now, here is my main question: is it possible to use append pandas df, similiar as you did with numpy in your example, to calculate all this indicators in every step, using above function? Or I have to rewrite everything and doing it indicator by indivator?

If you can help me with that most of my problems would be solved. Than I would just need to apply my ml model on those features and get prdictions.

0

Hi Mislav,

In order to receive at least 1 non-nan data point, the DEMA indicator requires the numpy array that is passed to it (the first argument) to be at least twice as long as the `timeperiod` (the second argument). This is why we required the `self.closes` array to be twice the length of the DEMA timeperiod in the previous algorithm I shared.

We can definitely replace the numpy array we used earlier with a pandas DataFrame. See the backtest I attached for a working example. After we update the DataFrame with the latest closing price, we pass the list of closing prices to each talib indicator method. The results from calling each indicator method are then added as a column in our DataFrame. This produces a time series of indicator values. To retrieve the latest value for each indicator, we just need to slice the bottom row with

self.rolling_window.iloc[-1]

As time is a constraint, I was not able to rework and debug the `add_ind`, `add_ind_df`, and `add_technical_indicators` methods that were published. However, the backtest I attached demonstrates the principle of storing multiple indicator time series values in a single DataFrame. For a more involved 1-1 support session, I recommend checking out our new Tutoring Initiative.

Best,
Derek Melchin

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed