Exposing API for CVXPY's conic form

[phschiele]

Hi,
I wanted to start a discussion to define a path that allows us to define a unified and stable API for exposing CVXPY's conic (or potentially other standard) forms. This is useful in many projects (at least 4 that I know about) and was brought up independently today by @tschm and @TobiaMarcucci.

Here is where we are right now:

1. We describe the conic form in the Contributing section. This form is no longer the most general form available, as SCS and Clarabel support quadratic objectives.
2. Getting the standard form is described in the Advanced Features section, but it is solver dependent.

Here is a rough plan of how we could go about this:

1. Create a dedicated section for CVXPY's standard form and how to obtain it. It should include the updated standard form with quadratic objectives.
2. In the experimental namespace, expose this standard form to users. All necessary ingredients should be available in the ParamProb classes. Perhaps, this would be a good time to unify the code in this area, now that the joint parent of the conic program with a linear objective and quadratic program exists.
3. If no changes are required or the API has stabilized, move it to the main namespace in a future release.

Happy to hear your thoughts on this!

[SteveDiamond]

That sounds fine to me, other than we should make sure we don't slow down QP canonicalization when we unify the QP and conic + quadratic objective paths.

[SteveDiamond]

Also we may want to think about a file format as well. This could be the same format that JuMP uses. Their format allows parameters (or will soon).

[tschm]

I think the precise file format could be an enum argument. We can start with a proprietary format and then support JuMP (and others)

[SteveDiamond]

Ok, maybe the proprietary format would just be pickle. I don't think we want to design our own format.

[tschm]

Ok, got it. Please don't use pickle. It's not for long term storage :-) An people will build crazy stuff relying on it. Better JuMP

[rileyjmurray]

I fully support this. We'd probably want to keep it in an experimental namespace for a while though.

[rileyjmurray]

I definitely don’t want a proprietary format beyond pickled data structures. Lots of people have worked hard on this and no one will benefit from yet another file format.

…

On Fri, Jun 30, 2023 at 10:08 AM Steven Diamond ***@***.***> wrote: Ok, maybe the proprietary format would just be pickle. I don't think we want to design our own format. — Reply to this email directly, view it on GitHub <#2167 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACRLIFBVN7OZCHYZTV3ALM3XN4B2JANCNFSM6AAAAAAZY7W2WE> . You are receiving this because you commented.Message ID: ***@***.***>

[TobiaMarcucci]

@phschiele Thank you for starting this discussion!

In my specific case, I'd like to convert a DCP into conic form, apply a few change of variables to the conic problem, and then solve the modified conic problem. Specifically, I'd like to define my problem as
prob = cp.Problem(cp.Minimize(cost), constraints)
and then reduce prob to a conic program conic_prob, and have a simple mapping between the variables in prob and the entries of the matrices in conic_prob.

I've tried
conic_prob = cp.reductions.Dcp2Cone(prob).reduce()
but this doesn't seem to let me access the matrices of the conic problem.
I've also seen the suggestion in https://www.cvxpy.org/tutorial/advanced/index.html#getting-the-standard-form but it seems suboptimal to call SCS for doing this?

If it's useful today I can put together a simple example and my best effort solution to this problem. Thanks again!

[TobiaMarcucci]

Basically I would need a way to get the matrices of the conic program corresponding to my DCP, and also have some way to pair the variables in my DCP to the right columns in the matrices of the conic program.

I've managed to achieve this with the (probably very inefficient) following code. I hope this clarifies!

def dcp_to_conic(cost, constraints):
    """
    Transforms the problem
    minimize cost subject to constraints
    into a conic program of the form
    minimize c x + d subject to Ax + b in K.

    Returns:
    c : dictionary of vectors with keys equal to the ids of the variables in x
    d : scalar
    A : list of dictionaries, each organized as c
    b : list of vectors
    K : list of cones
    aux_variables : list of variables introduced in the conic reduction.
    """

    # construct conic program from given DCP
    prob = cp.Problem(cp.Minimize(cost), constraints)
    solver_opts = {"use_quad_obj": False}
    chain = prob._construct_chain(solver_opts=solver_opts)
    chain.reductions = chain.reductions[:-1]
    cone_prob = chain.apply(prob)[0]

    # store auxiliary variables introduced in the reduction
    ids = [x.id for x in prob.variables()]
    aux_variables = [x for x in cone_prob.variables if x.id not in ids] 

    # map variables in the conic program to their indices in c and A
    ranges = {}
    for v in cone_prob.variables:
        start = cone_prob.var_id_to_col[v.id]
        stop = start + v.size
        ranges[v.id] = range(start, stop)

    # cost function
    cd = cone_prob.c.toarray().flatten()
    c = {v_id: cd[v_range] for v_id, v_range in ranges.items()}
    d = cd[-1]

    # constraints
    cols = cone_prob.c.shape[0]
    Ab = cone_prob.A.toarray().reshape((-1, cols), order='F')
    A = []
    b = []
    K = []
    first_row = 0
    for cone in cone_prob.constraints:
        last_row = first_row + cone.size
        A.append({v_id: Ab[first_row:last_row, v_range] for v_id, v_range in ranges.items()})
        b.append(Ab[first_row:last_row, -1])
        K.append(type(cone))
        first_row = last_row

    return c, d, A, b, K, aux_variables

[SteveDiamond]

I see, that's reasonable. We would be exposing the ParamConeProb class here

cvxpy/cvxpy/reductions/dcp2cone/cone_matrix_stuffing.py

Line 123 in 245defa

class ParamConeProg(ParamProb):

That has the info you need. It is part of the output of get_problem_data I believe.

[TobiaMarcucci]

Thanks! You mean the output of get_problem_data as in here? Is the one in this link is a stable solution? I ask since it requires passing a solver, which in principle should not be necessary

[SteveDiamond]

That's right, it's in data["param_prob"]. Agreed specifying a solver should not be necessary but at the moment there's no way to specify a target standard form except via a solver. The ParamConeProb is sort of an intermediate standard form.

[tschm]

There shouldn't be the need to call a protected function such as _construct_chain. Does it also take care of caching the chain?

[tschm]

I think the get_solution_data function is a bit too powerful and its name is somewhat misleading. Imho it would be nice if we isolate a function called canonicalize which essentially fills up the cache with the canonical form. It could literally take a few lines from the get_solution_data and get_solution_data could call the canonicalize function. For the backend it would be great if we could use an enum to simplify switching between those backends? @phschiele @SteveDiamond