AD interface compatibile with both ArrayDiff and ExaModels - #3046
AD interface compatibile with both ArrayDiff and ExaModels#3046blegat wants to merge 12 commits into
Conversation
From MadNLP's copy (parameter-aware, accumulating into the output)
The applicable() guard in _inner_constraint_linearity did not recurse: num_constraints on a layer is applicable but throws when the innermost evaluator (for example, a legacy NLPBlock evaluator) does not implement the query. Add _try_num_constraints, which returns nothing in that case at any depth of the stack.
An evaluator is only required to implement jacobian_structure and hessian_lagrangian_structure when it supports :Jac and :Hess. Solvers such as Ipopt query the structures eagerly even for models whose legacy NLPBlock evaluator supports neither (for example, an objective-only evaluator with feature :Grad), so the layers must not call into the inner evaluator in that case.
exploits_structure(backend) returns true when the model returned by model(backend) natively accepts ScalarAffine/ScalarQuadratic functions (no ModelWithQuad layer), so consumers such as NLPModelsJuMP know to pass all functions unparsed. Defined in MOI.Nonlinear because a trait spanning two package extensions (an AD backend and a consumer) has no reliable home in either: extension load order is unspecified. _has_objective falls back to true for unknown evaluators, which is conservative: the solver then calls eval_objective, and evaluators without an objective return zero.
Solvers such as MadNLP call the Jacobian-transpose product for dual computations even when :JacVec is not advertised, so erroring in the presence of oracle constraints is not an option. The oracle layer now materializes each oracle's Jacobian (or Hessian) to answer the product callbacks, which can be slow on large oracles; the :JacVec/:HessVec features remain removed from features_available so that solvers do not rely on them for performance. Also forward MOI.is_valid for constraint indices and MOI.ListOfSupportedNonlinearOperators through the model layers (needed by NLopt), and add the Nonlinear.exploits_structure trait.
|
Without looking at the details, it's worth investigating, but I'm pretty strongly against this. My experience with madsuite-org/ExaModels.jl#290 would suggest that writing a new We should have a very, very high bar for making the nonlinear interface in MOI any more complicated. |
|
I like the For the other components, I don't know enough MOI to provide feedbacks. Oscar is the expert 🙂 |
|
It was always my intention to move something like https://github.com/jump-dev/Ipopt.jl/blob/master/ext/IpoptMathOptInterfaceExt/utils.jl into MOI. I'd accept that as a separate PR. |
That's a good first step
I agree we shouldn't complicate it too much but I feel that at the moment,
Actually it won't. How would you make madsuite-org/ExaModels.jl#237 work ? GenOpt generator functions can be directly given to ExaModels without needing to be expanded. If they have to go through
It's fine as a one time thing but then once you add something new like |
3 participants
In #2989, I tried doing a simple change to the AD interface for ArrayDiff to serve as an AD backend.
The idea was to have
modelreturn a custom model type when given the AD backend as argument so that ArrayDiff could deviate fromMOI.Nonlinear.Model.The conclusion at the JuMP-dev call was to improve
MOI.Nonlinear.Modelso that we don't need to deviate.The issue with this is that it forces us to commit the changes to MOI so it means we can't do breaking changes which makes it hard to play around and try things. ArrayDiff is then blocked in a state where it cannot be tried by user until we just commit to an interface that just cannot be changed anymore.
Another proof that the AD interface is too rigid is the fact that ExaModels couldn't interface itself as a new AD backend and had to define itself as a meta-solver.
With AI getting more powerful, I think the features of MOI that allows people to extend it independently in different package is quite important. Trying to have a centralized
MOI.Nonlinear.Modelthat would cover all need is probably too strict.So this PR attempt to create a new interface so that both ArrayDiff and ExaModels can be defined as AD backends.
On major difference between ExaModels and ReverseAD is that ReverseAD is that Ipopt and NLPModelsJuMP don't give the quadratic constraints to ReverseAD while ExaModels takes the quadratic constraints in its AD.
So for the Ipopt wrapper to work for both, it would need to give the quadratic constraints to both and ReverseAD should also handle quadratic constraints.
The way this PR does it is to refactor the handling of quadratic constraints and
VectorNonlinearOracleof Ipopt as layers that we can stack on top of an existing AD.This is continuing the approach started in jump-dev/Ipopt.jl#520
As a side effect, this reduces the large code duplication of the handling of quadratic constraints and nonlinear oracles in Ipopt, NLopt, MadNLP, NLPModelsJuMP, etc... Again, with AI, we might see a lot of new solvers appear, so having all of them copy the Ipopt wrapper is probably not a good idea. Because when we'll modify the Ipopt wrapper, they won't follow and it will create inconsistent behavior across solvers. Even if many vibe-coded solvers won't be any good, natural selection will do its job and natural selection will work better if they are all easily accessible through JuMP.
So Claude Fable 5 create this PR which seems reasonable although I haved looked at all details yet.
We could probably open a PR for 459c20a and merge it independently if we decide we want to go for this.
We needed linearity information for NLPModelsJuMP and Ipopt so I reused the linearity of MOI.Nonlinear.ReverseAD, just added quadratic in the mix. The quadratic detection might have some false negative at the moment but it's find since the quadratic given as quadratic in JuMP are already reported by the WithQuad layer.
Let's try it with the downstream packages to see if it works