Back

Calling Python Experts - Proposed New QC Python API

We want QuantConnect to be a truly multi-language platform - where language is not a barrier to writing quant strategies. Currently our Python API follows the C# method and variable names -- which is a little counter intuitive for python programmers.

To fix this we started a project to provide a 'pythonista' style to the API, trying to follow the patterns and capitalization python programmers are familiar with. Its been a few months work but its almost complete and we'd love to get your feedback -- please see the two comments below and vote to pick your favorite.

Comment 1 - The current style which follows the C# API.
Comment 2 - The proposed style using python style conventions.

Please vote on the comment style you like best!
---

Update Backtest






The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.



Current Style:

Overall Algorithm Structure: 

def Initialize(self):
    self.SetStartDate(2013,10,07)   #Set Start Date
    self.SetEndDate(2013,10,11)     #Set End Date
    self.SetCash(100000)             #Set Strategy Cash
    self.AddSecurity(SecurityType.Equity, "SPY", 
        Resolution.Second)


def OnData(self, data):
    if not self.Portfolio.Invested:
        self.SetHoldings("SPY", 1)

Indicators

def init(self):
    self.__macd = None
    self.__Symbol = "SPY"


def Initialize(self):
    # define our daily macd(12,26) with a 9 day signal
    self.__macd = self.MACD(self.__Symbol, 9, 26, 9, 
        MovingAverageType.Exponential, Resolution.Daily)


def OnData(self, data):
    if not self.__macd.IsReady: return    

    tolerance = 0.0025;
    holdings = self.Portfolio[self.__Symbol].Quantity        

    signalDeltaPercent =
        (self.__macd.Current.Value - 
         self.__macd.Signal.Current.Value)/ 
         self.__macd.Fast.Current.Value

    if holdings <= 0 and signalDeltaPercent > tolerance:
        self.SetHoldings(self.__Symbol, 1.0)
    elif holdings >= 0 and signalDeltaPercent < -tolerance:
        self.Liquidate(self.__Symbol)


Universe

def __init__(self):
    self.__numberOfSymbols = 5
    self.__changes = SecurityChanges.None


def Initialize(self):
    self.UniverseSettings.Resolution = Resolution.Daily        
    self.AddUniverse(self.CoarseSelectionFunction)


def OnData(self, data):
    if self.__changes == SecurityChanges.None: return

    for security in self.__changes.RemovedSecurities:
        if security.Invested:
            self.Liquidate(security.Symbol)

    for security in self.__changes.AddedSecurities:
        self.SetHoldings(security.Symbol, Decimal(0.2))    

    self.__changes = SecurityChanges.None;


def OnSecuritiesChanged(self, changes):
    self.__changes = changes


def CoarseSelectionFunction(self, coarse):
    sortedByDollarVolume = 
        sorted(coarse, key=lambda x: x.DollarVolume, reverse=True) 
    top5 = sortedByDollarVolume[:self.__numberOfSymbols]
    return [x.Symbol for x in top5]

Order

def __init__(self):
    self.__Symbol = Symbol.Create("SPY", SecurityType.Equity, "USA")
    self.__openMarketOnOpenOrders = []


def OnData(self, data):
    if self.TimeIs(8, 12 + 2, 0):
        self.Log("Submitting MarketOnOpenOrder")

        newTicket = self.MarketOnOpenOrder(self.__Symbol, 50)
        self.__openMarketOnOpenOrders.append(newTicket)

    if len(self.__openMarketOnOpenOrders) == 1 
        and datetime(self.Time).minute == 59:
        ticket = self.__openMarketOnOpenOrders[0]

        if ticket.Status == OrderStatus.Filled:
            self.__openMarketOnOpenOrders = []
            return

        quantity = ticket.Quantity + 1
        self.Log("Updating quantity  - New Quantity: {0}"
            .format(quantity))

        updateOrderFields = UpdateOrderFields()
        updateOrderFields.Quantity = quantity
        updateOrderFields.Tag = "Update #{0}"
            .format(len(ticket.UpdateRequests) + 1)
        ticket.Update(updateOrderFields)


def OnOrderEvent(self, orderEvent):
    order = self.Transactions.GetOrderById(orderEvent.OrderId)
    self.Log("{0}: {1}: {2}"
        .format(self.Time, order.Type, orderEvent))

Setter

def Initialize(self):
    self.SetBrokerageMessageHandler(
        CustomBrokerageMessageHandler(self))


class CustomBrokerageMessageHandler(IBrokerageMessageHandler):
    def __init__(self, algo):
        self._algo = algo


    def handle(self, message):
        toLog = "{0} Event: {1}"
            .format(self._algo.Time, message.Message)
        self._algo.Debug(toLog) 
        self._algo.Log(toLog)

 

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Proposed Style:

Overall Algorithm Structure:

def initialize(self):
    self.set_start_date(2013,10,07)   #Set Start Date
    self.set_end_date(2013,10,11)     #Set End Date
    self.set_cash(100000)             #Set Strategy Cash
    self.add_security(PySecurityType.equity, "SPY", 
        PyResolution.second)


def on_data(self, data):
    if not self.portfolio.invested:
        self.set_holdings("SPY", 1)

Indicators

def init(self):
    self.__macd = None
    self.__symbol = "SPY"


def initialize(self):
    self.security = self.add_security(PySecurityType.equity, 
        self.__symbol)

    # define our daily macd(12,26) with a 9 day signal
    self.__macd = self.macd(self.__symbol, 9, 26, 9, 
        PyMovingAverageType.exponential, PyResolution.daily)


def on_data(self, data):
    if not self.__macd.is_ready: return    

    tolerance = 0.0025;
    holdings = self.security.holdings.quantity 

    signal_delta_percent =
        (self.__macd.current.value - 
         self.__macd.signal.current.value)/ 
         self.__macd.fast.current.value

    if holdings <= 0 and signal_delta_percent > tolerance:
        self.set_holdings(self.__Symbol, 1.0)
    elif holdings >= 0 and signal_delta_percent < -tolerance:
        self.liquidate(self.__Symbol)

Universe

def __init__(self):
    self.__changes = PySecurityChanges.none


def initialize(self):
    self.PyUniverseSettings.resolution = PyResolution.daily        
    self.add_universe(self.coarse_selection_function)


def on_data(self, data):
    if self.__changes == PySecurityChanges.none: return

    for security in self.__changes.removed_securities:
        if security.invested:
            self.liquidate(security.symbol)

    for security in self.__changes.added_securities:
        self.set_holdings(security.symbol, Decimal(0.2))    

    self.__changes = PySecurityChanges.none;


def on_securities_changed(self, changes):
    self.__changes = changes


def coarse_selection_function(self, coarse):
    sorted_by_dollar_volume = 
        sorted(coarse, key=lambda x: x.dollar_volume, reverse=True) 

    top5 = sorted_by_dollar_volume [:5]
    return [x.symbol for x in top5]

 

Order

def __init__(self):
    self.__symbol = PySymbol.create("SPY", PySecurityType.equity,
        "USA")
    self.__open_market_on_open_orders = []


def on_data(self, data):
    if self.TimeIs(8, 12 + 2, 0):
        self.log("Submitting MarketOnOpenOrder")

        new_ticket = self.market_on_open_order(self.__symbol , 50)
        self.__open_market_on_open_orders.append(new_ticket)

    if len(self.__open_market_on_open_orders) == 1 
        and datetime(self.time).minute == 59:
        ticket = self.__open_market_on_open_orders[0]

        if ticket.status == PyOrderStatus.filled:
            self.__open_market_on_open_orders = []
            return

        quantity = ticket.quantity + 1
        self.log("Updating quantity  - New Quantity: {0}"
            .format(quantity))

        update_order_fields = update_order_fields()
        update_order_fields.quantity = quantity
        update_order_fields.tag = "Update #{0}"
            .format(len(ticket.update_requests) + 1)
        ticket.update(update_order_fields)


def on_order_event(self, order_event):
    order = self.transactions.get_order_by_id(order_event.order_id)
    self.log("{0}: {1}: {2}"
        .format(self.time, order.type, order_event))


Setter

def initialize(self):
    self.set_brokerage_message_handler(
        CustomBrokerageMessageHandler(self))


class CustomBrokerageMessageHandler(PyBrokerageMessageHandler):
    def __init__(self, algo):
        self._algo = algo


    def handle(self, message):
        to_log = "{0} Event: {1}"
            .format(self._algo.time, message.message)
        self._algo.debug(to_log) 
        self._algo.log(to_log)

 

4

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Comment 2. Python style might be better for Python users since we are used to a lot of the open source trading libraries out there.
1

No, no, no. Keep the same names and style regardless of language. Don't let strange, rudimentary pythonic styles creep into your excellent API.

Nothing in the Python Manifesto (>import self) says that python vars and methods should be named in lower case with underscores as delimiters. That's just some nonsense dreamed up by a bunch of developers who wanted to make a foolish statement.
WORDS_SEPARATED_BY_UNDERSCORES_ARE_CONSTANTS = "Period"!

 

2

Very interesting point of view @AnonyMole - we were trying to keep it to the official guides. We always wondered if the unfamiliar style scared away python users and were experimenting with this as a potential fix. Can you point out some python projects which might use code styles similar to QC's current API?

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Guido only wanted to make his new baby shockingly different when he adopted that broken model of my_favorite_variable_name nonsense. Seriously, would you suddenly start writing ALL CAPS FOR A LANGUAGE THAT STIPULATED IT? Or what if when you wrote Spanish or German is was exepected that you s_e_p_a_r _ a _ t _ e  a _ ll v _ o _ u _ ls w _ i _ th u _ nd _ e _ rsc _ o _ r _ e _ s? You would do neither. C C++,C#,Java, JavaScript, VisualBasic, and no doubt many others can all be written with the same noun or verb style. why_must_python_be_the_only_stand_out():?

I wrote all of my Quantopian code (dozens of offered examples) using the common language standard style. I wrote all my Computational Finance code (which many people fully enjoyed) using the CLSS too. I have no QC examples, no, or github examples. But I would never let trite, ill conceived rules tell me how to write my code. But, it appears that adhearing to such rules, in order to be accepted, is a trend that cannot be denied. I've fought this battle with others where I have worked and rarely win. But I never cowdown. Stubborn I suppose.

 

0

I like the Python style better. 

2

The proposed Python style requires more letters per identifier that the c# style and thus uses more visible line space in my IDE editor - i prefer the succinct C# style.

1

Any update on this? 

I understand Anony's point. I've actually always wondered why the German language puts the verb at the end of each sentence. French people don't do that, nor English people as far as I know. Now, would you write the German version of QC's documentation in a different German than what is used nowadays? Probably not because it just doesn't make sense. Even if you think you're right about where a verb should be placed. 

It's a fact that can't be denied that most (almost all actually) libraries in Python are written in the "pythonista" style. As a (possibly dumb) Python user who adopted the majority view on Python's style without giving it much thinking, I find it quite frustrating to switch to a new style when using QC. Old habits are hard to break I guess, but the pain of switching back and forth is simply not worth it IMO. And I'm certainly not going to rewrite all my code base and all the packages out there to match QC's style. 

C# coders are probably pleased to write Python code in C# style, but I doubt this is the case for Python programmers. 

1

"Pythonista" -- I like that -- like the Python crowd is a communist plot to take over your coding style. <grin>

In the end, does it really matter? I'll write all my code the way I want, and so will you, and so will all of us. What any one vendor chooses as their style... has little impact on any one programmer's coding style decisions.

0

Thanks for the feedback guys. We're waiting for a few more python users to chip in before making a decision one way or the other. There's no "right" or "wrong"; we're just looking to make it easy for people to learn.

Making it easier can go both ways: 1) similar style to other python libraries, but different to QC so users can't really copy paste C# algorithms. or 2) similar to QC's C# style which isn't like other python -- but more or less allows you to copy and paste from C# code. 

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


I don't agree with some of the assessment on why Guido chose the style he did but that's irrelevant (I actually prefer it). 

I would argue that you should not waste any engineering effort on this task though.

Yes the style will perhaps be different from what Python developers are used to but that's ok. I think keeping it as close as possible to C# allows one to understand other people's examples and even copy and paste (or translate) algorithms easier. 

Also this reduces the engineering overhead in supporting these differences, I'd rather have futures support for example :)

If you've ever used Twisted, which existed before PEP8, you are familiar with dealing with a different code style. No big deal.

Besides people will code however they want to code anyway, that I do agree with.

0

Great analysis! Thank you Aaron,
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


The issue I take with the pythonista style of variable/method/object naming is that such a style contradicts with how the brain works. When you read, you don't read letters, you read a blocks of squiggles that together represent words. When you read the word "dog", your brain doesn't concentrate on the d or the o or the g. It actually pattern matches on the combination of letters, the whole word. And what separates these blocks of squiggles are spaces, namely white space between word patterns. Now when you read the word dogfood, your brain initially identifies that this new pattern is a contraction of two patterns. But in future reading of that same pattern, dogfood, your brain will see that pattern as a single entity.

If you, on the other hand, were to divide those two patterns by a space "dog food" your brain will continue to see them as two separate patterns. I hold that the underscore is an equivalent replacement for the space. That the variable name "dog_food" remains TWO words. And that being two words your brain must do extra duty to identify this as a SINGLE thing. 

DogFoodWithGravy -- can be interpreted, by your brain, as a single entity.

dog_food_with_gravy -- will ALWAYS be interpreted as four separate words that your brain must parse EVERY TIME you read it.

This, this is the primary reason that pythonic naming is a failed style.

 

 

0

I do like the second style more (it is easier to follow from my perspecrtive). As long as the dev logic is consistent regardless of language, implementation is a secondary factor though - I care more about explicit documentation with more examples.

Are there any plans regarding importing external libraries (pandas)? :)

0

@Jared Broad: How much additional work would switching to the proposed style require from you guys? According to Alexandre Catarino, most of the work is done already. If it still requires a lot of work, then of course I agree with Aaron Todd that there are more important things to allocate resources to (like Futures? :). 

Back to the topic... Most of our time is spent coding algorithms, not copying and pasting other people's algos. So the occasional frustration of having to translate C# code into Python is nothing compared to the permanent frustration of switching styles or having inconsistent styles when using QC's API with other libraries. It's a small thing but it's annoying. 

Lastly, because most people on this forum seem to be C# coders, I'm a little worried that this will skew results towards the C# style. To correct this bias, I suggest that we ask the same question to C# coders: should we use the Python style for C# code? Then let's sum the votes for "same style for C# and Python" and "keep the original language style" from both topics and we'll have our answer. :-) 

 

0

Why would you change the syntax of the language that is so widely accepted amongst Python programmers? For sake of consistency please implement the proposed version!!!

1

The second style aligns with python style standards (PEP8). I recommend going with that if you want to call it a python API.

1

As a Python programmer, I would not want to look at C# conventions in my Python code.
0

Python has a style guide and most Python programmers follow these strictly. It is part of the joy of working with the language. If you remove that joy, no serious Python programmer will want to use the API.
1

It does really matters, what would be the point in creating a python API if it will look like another language. To make it intuitive to Python programmers it should look like python code.
0

I feel like a lot of people are perhaps missing the point. They aren't "creating" a new Python API. The Lean implementation is accessed via IronPython's dotnet integration http://ironpython.net/documentation/dotnet/

More than likely what they are proposing is some wrapper API that hides this detail. Of course Python developers, of which I count myself, are going to want the proposed API but it doesn't change the fact underneath your code is calling C# code.

If I was writing something unrelated to QC that integrates with .NET framework classes (e.g a winforms app) I wouldn't expect the imports/API to differ and in fact they don't.

I appreciate that the QC team is trying to make this more accessible and familiar to Python developers but I just don't see enough benefit in maintaining some wrapper API (even if it can be auto-generated) just because the style differs.

QC team feel free to correct me if I've made an incorrect assumption somewhere.

0

Aaron, from an engineering standpoint you're probably right. But what you seem to be missing is that adoption by the Python community is a MASSIVE benefit. Not a technical benefit, but a social one (look at Quantopian for instance). 

0

Aaron is correct. It will be a auto-generated wrapper.

I do not want to believe that python coders are not coming to QuantConnect because of its API. We do have benefits over other platforms that should be more important than style.

Nonetheless, we want to provide this API to make people "feel at home".
The point of this thread is: "Do they care and/or want it?".
 

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


@Aaron you're right, technically that is the implementation. Like other decisions we make regarding the LEAN stack though we try to make the right decision regardless of difficulty which is why we started this wrapper project. From your detailed analysis though it looks like there are no good ways to support Numpy/Panda - either fix IronPython or create a raw python api using ZMQ messaging to talk to LEAN. Personally I like how IPy compiled dll's run lightening fast so lean towards that solution. 

Ultimately we want LEAN-Python to support all the built in libraries; so regardless of the API we have some thinking to do. The best course now is probably to spend more time seeing if IronPython can support Pandas/Numpy/SciPy - then continue the wrapper project. 

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


@Alexis Adoption and the underlying reason was not lost on me, as a

developer I understand and appreciate the tradeoffs they are making though.



@Alexandre, absolutely that is partially my point, the style makes little

difference, the capabilities are more important.



@Jared Yes, I can appreciate that, I don't mean to derail the conversation

either since the point of this thread was to simply take a vote, so

apologies. I was just trying to make sure everyone understood why the

current API looks the way it does and it wasn't a choice between creating

it one way or another necessarily. To your point maybe that isn't their

concern and if offering a Python API is possible and people want it then by

all means. And yes I would agree figuring out the numpy/pandas/scipy

integration is perhaps more important to people but that's a whole

different problem.



Anyway to hopefully put the thread back on track my vote is to keep it the

way it is ala C#.
0

I find the capital letter on method calls a bit annoying, I like the second style better: https://www.python.org/dev/peps/pep-0008/#function-names

Then on placing order the method  MarketOnOpenOrder was not really clear to me. Maybe a name improvement. Is it placing an order as Market?

0

Hi Lucas,

In the new style, MarketOnOpenOrder would be called market_on_open_order. The style wouldn't make it clearer for you, right?
The name logic is the following: order is the substantive and market on open is the adjective. It is a market on open, since it will be executed when the exchange opens at the market price of the security. I think it would be too wordy to call it MarketOnExchangeOpenOrder.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thank you all for the feedback. Lets call the discussion paused for now until we can resolve a way of getting the science/math libraries to talk to C#!

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed