New Optimizer Feature Discussion Thread

SUGGESTIONS CONTINUED:

• This one is for the forum web app but I almost didn't post this thread because I hit Backspace while I was :focus'd on the page and not on the text-area where I was writing my post, and Firefox page-backed to the Forum Index page. If your devs can be bothered, please try yeeting this javascript into your /forum/post/discussion - Stack Overflow - Warn user before leaving web page with unsaved changes, I think the "Long, Correct Answer:" can be copied and pasted verbatim since it's targeting an event, not an id etc.. bless
• also for the forum but why is there a 5 minute time limit on editing :'( I made a mistake pasting my screenshot in the OP and now it's confusing
• I could see pushback on this suggestion, but I'm legitimately curious. Is the reason for an alternative pricing model for Optimization a technical reason or a business reason? Not much of a cloud guy but I don't se the technical difference between running an optimization on 1 Node and running the optimization as a set of queue'd up backtests on a Backtest Compute node, which I am already paying for. Perhaps since this is the equivalent of "queueing a backtest", and for business reasons QC has elected to make that a feature that you must be a TEAM (not a RESEARCHER) to use, I could understand only allowing people to run optimizations on 1 Backtest node that they are currently paying for a TEAM license. That said, if I'm paying $20/month for a TEAM seat and $10-$80/month for a Backtest Node, I would be dissapointed to have to "pay twice" for this optimization feature, since in it's current state users must pay "hourly" to optimize in a way that has no correlation to their existing node payments.

• Continuing with the previous suggestion, I do not think allowing users to do a "slow" optimization using the resources they are already paying for will cannibalize the Optimzer business model. If Users have a need to see optimization results more quickly, then they can elect for pay-for-what-you-need nodes, up to 24 of them running "asyncronously". Similar to airline seats, I'm sure this compromise would allow QC to have a cheap, worst option for individuals and a more expensive but dramatically improved option for power users and businesses. As an added benefit, users that are new to QC can see the power of the Optimizer without being scared away by increasing the amount they are paying for it. I think this would allow a "trial" of sorts, where users that are paying a minimum of $30/mo can grasp the power of a feature like the optimizer, and after they make big bucks with their newly discovered 10+ Calmar algo, can come back and pay for improved Optimizer performance now that they understand it's potential.
• That said, as a pre-10-Calmar QC user, I think the pricing as-is is reasonable for small amounts of data. It's really not terrible. I'm just dissapointed because I would like to try implementing extensive Walk-Forward testing at various data resolutions and larger time spans, and while I would not mind setting up an experiment to run on my 1 backtest node overnight, I would mind paying $5-10 for a single optimization experiment to prove that an algorithm recently posted in our QC Community is overfit and dangerous.

As always, I love this work. My overarching review is positive, as is my impression of all the devs working hard to improve this platform! But there is always room for growth, and I wanted to start that conversation.    

I suscribe to the pricing model suggestion as Austin says, adding that it can be the other way around, the optimizer may cannibalize fixed node subscriptions depending on your yearly CPU consumption. Fixed nodes should be able to get pooled into optimization to make them more attractive.

I think maximum flexibility would be ideal.  Expose the underlying backtest scheduling and dynamic scaling as an API.  Then expose an open source optimization framework (maybe within LEAN), so people can implement an optimization as code just like backtests.  This code can again be local or in your web IDE.  This will greatly increase the proliferation of different optimization options beyond what is achievable through a point-and-click UI. For example the framework can expose places to plug in parameter generation, the optimization function (e.g. sharpe ratio, or something custom), even report generation.  If the implementation is open-sourced, the community can contribute useful version of these things (much like how indicators can get added in LEAN today).  This has the added benefit where possible more advanced optimization algos (e.g. beyond grid search) could be outsourced to existing libraries with robust implementations.

I also agree with the points above about a hybrid model where optimization/automated backtests and manual backtests can share the same resources, so they can run either on monthly pre-paid nodes or scale out to the shared node pool.

Thanks for the feedback all, it's a quite different technology that powers the two node types but I appreciate the desire to want to reuse your existing subscriptions. We've come up with a "slider" concept where it'll use your initial nodes available and you can "slide" it up the scale to rope in more nodes on a per hour basis. This should give you the speed of the original nodes, and scaling of the new optimization nodes when you need it. 

It'll take some time to implement with Christmas but hopefully, in the next 10 days, the wizard will reflect this new feature. 

"We've come up with a "slider" concept where it'll use your initial nodes available and you can "slide" it up the scale to rope in more nodes on a per hour basis. This should give you the speed of the original nodes, and scaling of the new optimization nodes when you need it."

That would be awesome, glad to hear this feature request is already in the works!

It reminded me of what I heard in this Ernie Chan video (23 minutes in), he mentions how if you pay for cloud computing and you think a test is likely to fail, there is a psycological barrier that makes it less likely for you to run failing tests. I'm glad that in the future, I'll have the ability to run tests on slower/unoptimized nodes for tests that I believe will fail, but still want to test a hypothesis. :)

Also, will Optimizer runs be queueable, like backtests are? I only have a Researcher seat so I don't currently have access to backtest queueing but would likely upgrade if it did.

I think there are several use cases, but my primary one would be for Walk-Forward analysis, in this example one could queue 5 optimizations at once and use the results to then queue 5 walk-forward backtests:

This would greatly help me avoid the pitfall Jared made at the end of the announcement thread - "PS: It may be tempting to use this to hyper tune the backtest but resist the urge! It is generally meaningless to look at a single backtest with this analysis tool."