Back

Removing NAN's from Minute Data

I've been trying to port my python algorithms from Quantopian to QC. For reasons that are not clear to me, the same functionality that I used in python at QP does not work at QP. 

https://www.quantconnect.com/terminal/processCache?request=embedded_backtest_1e30c81c3121cbcabf705974ca173e2c.html

Try running this algo with resolution in "take_profit"  function set to "Minute".  It runs fine for a while then runs in to what I presume are nan's.  This is my error:

Runtime Error: Python.Runtime.PythonException: KeyNotFoundException : 'TMF' wasn't found in the Slice object, likely because there was no-data at this moment in time and it wasn't possible to fillforward historical data. Please check the data exists before accessing it with data.ContainsKey("TMF")

So I presume it cannot find a corresponding value in one of the columns of the history dataframe. My ported code had originally contained command ").bfill().ffill()"  to fill out any NAN's in the data, but this bombed.

My final recourse was to use try / except to trap for the error and skip over it if encountered but that is obviously not an acceptable solution. I've got to find out how to forwad and backfill missing values.

Runtime Error: Python.Runtime.PythonException: AttributeError : '1, Culture=neutral, PublicKeyToken=null]]' object has no attribute 'bfill'

Also, I have been developing in the online IDE which does not allow me to inspect objects and variables and do any serious debugging. I have installed LEAN on the Google Cloud Engine but not yet figured out how to run my tests from there. I've set up Jypyter to run from my browser off of Google Cloud install but not clear to me how to point this to QC server to run and debut programs there. Any tips/tutorials would be much appreciated.

Update Backtest








I found one workaround. Basically it looks like I was trying to work with a dictionary when a dataframe was needed. Added this function:

    def get_history(self, symbol, num,resolution ):
        data = {}
        dates, price = [],[]
        sym = List[Symbol]()
        sym.Add(symbol)
        history = self.History(sym,num,resolution)
        for i in map(lambda x: x[symbol],history):
            dates.append(i.EndTime)
            price.append(np.log(float(i.Close)))
        # return a pandas DataFrame contain log price and python datetime
        df = pd.DataFrame(price,index = dates, columns = ['price'])
        return df

 

then called it as follows

                # Get last 10 bars of any positioin held , at specific resolution.
                hist_df=self.get_history(s,10,Resolution.Minute).bfill().ffill()

      
                #create a list from dataframe since slope function requires a List
                histlist = hist_df.values.tolist()
                if self.slope(histlist)      > 0: continue
                if self.slope(histlist[-5:]) > 0: continue
                if histlist[-1] > histlist[-2]: continue

 

Now if someone would like to point me to a fabulous Jypyter tutorial on debugging on headless servers my day would be perfect!

BTW. This is hardly elegant code. I'm sure I'm pulling a load of unnecessary resources into the Class. I'll clean that up later. 

0

In Lean/QuantConnect, the History method can return a pandas.DataFrame:

https://www.quantconnect.com/forum/discussion/2423/pandasdataframe-from-history-method/p1

It will make your code simpler:

def take_profit(self): # Close some positions to take profit
pos = [ x.Symbol for x in self.Portfolio.Values if x.Invested ]
if len(pos) == 0: return

# Returns a DataFrame with closing price of symbols of securities that are invested
# .unstack(level = 0) puts the symbol in the column
df = self.History(pos, 10, Resolution.Minute).close.unstack(level = 0)

# Only loop through the symbols that are found in the DataFrame
for s in df.columns:
history = df[s]
if history[-1] > history[-2]: continue
prc = history[-1]
amt = self.Portfolio[s].Quantity
1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed