Transitioning from ECOS to Clarabel as the default open-source IPM

[PTNobel]

This is my attempt to summarize various private conversations I've had over the last few weeks about transitioning CVXPY from using ECOS to using Clarabel as the default open-source IPM and solver.

Background

ECOS has been the default IPM for solving SOCPs in CVXPY for longer than I have been aware of this project. It is not actively maintained, and has known issues with numerical stability and performance in some cases.

A new IPM that has a few small changes to the ECOS algorithm (including quadratic objectives) is being developed at @oxfordcontrol/Clarabel.rs. There are two versions of Clarabel: Clarabel.rs and Clarabel.jl; the first is written in Rust, and has a Python interface that CVXPY currently supports; the second is written in Julia and has no Python interface.

A C API for Clarabel.rs is in development. Drake is looking to add Clarabel as a solver using this interface; see here. Clarabel can solve problems with quadratic objectives and equality, inequality, SOC, exponential cone, power cone, or SDP constraints.

Generally, features are prototyped in the Julia version and then ported to the Rust version; though the Rust version supports all the cones ECOS supports and SDP cones making it already feature-superior to ECOS.

Clarabel.jl already supports the 3D and ND power cones; when this is ported to the Rust version, this would allow us to partially address #1222 by eliminating the need for the Exotic2Common reduction in many cases.

In general, it is reasonable to expect that Clarabel.rs will be able to give better SDP accuracy than our current default open-source SDP solver, SCS, because it is an IPM instead of an ADMM-variant.

Since CVXPY 1.3, we have supported using Clarabel for SOCPs. We have merged support for using Clarabel for SDPs and will release this with version 1.4. The maintainers decided against backporting SDP support to the 1.3 branch.

There are backward compatibility concerns with:

1. removing as a mandatory dependency a currently installed-by-default solver.
2. reordering the solvers that are called.
3. introducing new default dependencies, especially those that require new toolchains.
   Notably Rust is a system programming language that is supported by all
   platforms supported by LLVM, whose compiled artifacts require no runtime beyond
   the C library; when distributed as binaries (which they are for most common
   platforms) the user experience is the same as a new C or C++ dependency.

Proposal

Part 1: CVXPY should aim to change its default open-source solver for SOCPs from ECOS to Clarabel. I think it is safe to expect Clarabel to be a strict upgrade in performance, numerical stability, and accuracy.

Part 2: CVXPY should aim to change its default open-source solver for SDPs from SCS to Clarabel. I think preferring more accurate solutions to potential better performance is worth considering.

Part 3: CVXPY should treat adding a new dependency in an LLVM-supported language with effectively no runtime (so at this point C, C++, Rust, and Zig [tho Zig is trying to get away from LLVM, so we'll see what the future holds]), as a minor-version-allowable backwards compatible change. This was the approach taken by the devs of python-cryptography [who do not follow SemVer, but highlighted that this was compatible with a minor version bump to argue that SemVer compliance would not have impacted the change] to some blowback: see here for a thread which had a big pile on.

It has been a few years, and I suspect the ecosystem is much more used to Rust now. I also suspect CVXPY is used in fewer exotic contexts than cryptography. The Cryptography FAQ notes that with recent versions of pip, precompiled versions of the library are used instead of building it from source which alleviates the need for a Rust compiler. This only affects users who are building all of our dependencies from scratch.

Further, we can direct users who cannot or do not want to build Rust libraries, to install cvxpy-base and ECOS instead.

Part 4: We should add Clarabel as a default dependency---without changing its priority---in 1.4 and aim to release 1.4 in the Fall quarter of 2023. We should encourage users to try out Clarabel in the release notes and the developer call, and encourage them to file bugs or regressions.

Part 5: Barring a major and easy-to-hit regression, we should move Clarabel above ECOS in priority for solving SOCPs in 1.5 and aim to release 1.5 in the Spring quarter of 2024.

Part 6: Barring a major and easy-to-hit regression, we should move Clarabel above SCS in priority for solving SDPs in 1.5.

Open Questions

How does Clarabel SDP runtime performance compare to SCS?
Paul has expressed interest in better performance benchmarks, and has offered to prioritize benchmarking to help us make a decision in Part 6 if we adopt this proposal.

Should we deprecate ECOS and eventually remove it as a dependency?
We could, for example, start raising a deprecation warning when calling ECOS in 1.6 and remove it in 1.7.

CC: @goulart-paul @rileyjmurray @phschiele @SteveDiamond

[PTNobel]

Discussion from Sep 10 Community Call:

• Clarabel will be added as a dependency in 1.4
• The plan regarding SOCPs is:
  • In 1.4, we will raise a warning when .solve is called without explicitly specifying a solver
  • In 1.5, we will switch the default solver to be Clarabel and raise a warning when explicitely calling ECOS about the dependency change
  • In 1.6, we will remove ECOS as an installation dependency
• We will probably consider switching SDPs to default to Clarabel on the same timeline, but are less certain right now while we wait for more SDP performance results
• We'd like to see QP benchmarking data before considering Clarabel replaces OSQP
• We agreed there should be an error if ECOS is called and it is not installed
  • Parth suggested adding a wrapper package called ecos-explicit that silences the warning in 1.5 about needing to install ECOS explicitly
  • Akshay opposed ever having solve(cp.ECOS) call Clarabel with a warning in 1.6

CC: @goulart-paul

[PTNobel]

We will probably consider switching SDPs to default to Clarabel on the same timeline, but are less certain right now while we wait for more SDP performance results

My views have since evolved, and I don't think we should consider changing the SDP default with the same 1.4 warning/1.5 change cadence, since Clarabel SDP support is shipping for the first time in 1.4 and I think we should let it bake longer.