Optimization improvements

[torfjelde]

IMO the current optimization we provide is quite lacking. In particular, it seems (I couldn't get it to work with constrained models) as if we support the Optimizations.jl interface through SciMLBase, but it's not really exposed nor documented. Moreover, it's exposed through this optim_problem (which is also exported btw) that returns several things that might not be obvious to the user.

I suggest the following:

1. Clean up the src/optimization module.
2. Define two methods maximum_a_posteriori(model, solver; kwargs...) and maximum_likelihood(model, solver; kwargs...), where the solver is whatever Optimizations.jl accepts (and similarly the kwargs).
3. Fix whatever is going on with the constrained models not working.
4. Make adtype have some default type. Currently, if you try to use something like Optim.Newton, it will error with some obscure message which really just comes down to the fact that adtype=SciMLBase.NoAD() (which is contrary to everything else we do in Turing.jl, which is all "ad is enabled by default", no?)

Related: #2164

@mhauru

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 0.00%. Comparing base (c29d36e) to head (902ed18).
Report is 9 commits behind head on master.

Files	Patch %	Lines
src/optimisation/Optimisation.jl	0.00%	6 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #2214    +/-   ##
=======================================
  Coverage    0.00%   0.00%            
=======================================
  Files          21      22     +1     
  Lines        1367    1534   +167     
=======================================
- Misses       1367    1534   +167     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[yebai]

Maybe also take a look at #2152 to see why it is causing issues along the process.

[torfjelde]

Maybe also take a look at #2152 to see why it is causing issues along the process.

I don't think this has anything to do with this issue. Isn't that just because of incompatibilities wrt. ADTypes v1? Some downstream packages are still not up-to-date IIRC.

[yebai]

Superseded by #2221