How to use C++ Backend with Python Optimization API

[tbretl]

The SymForce diagram in the README implies that it is possible to use the C++ Backend — i.e., to use runtime C++ code for factors as produced by codegen — with the Python Optimization API. How is this done? Can you point me to an example?

Thanks!

(And thanks also for the work you put into this project - it's very nice.)

[nathan-skydio]

I'm not sure if there's a full example of this unfortunately. Looking at our examples folder I see examples of using the python optimizer with python factors and examples of using the C++ optimizer with C++ factors, but I don't think we have an example of using C++ factors with they python optimizer (though this would be nice to have if you make one!).

If you wanted to do this, I believe you would have to pybind wrap the C++ factor yourself so that it's callable from python, then create a NumericFactor with the wrapped function, then pass the NumericFactor to the python optimizer.

Note that the "python optimizer" is just a wrapped version of the C++ optimizer though, so if you're trying to reduce solve time while still kicking off the optimization from python, it would likely make the most sense to pass your C++ factor to the C++ optimizer directly inside a pybind wrapped function. E.g. pybind wrap a helper function which takes your optimization problem inputs as an argument, builds your C++ factors and runs the C++ optimizer, and then returns the optimization result.

[aaron-skydio]

I will say that this isn't something we've done much at all, and I can't really say whether or in what cases you'll get better performance from doing this than using python-generated factors, possibly compiled with numba

[tbretl]

I understand, and thanks for the quick response.

The reason I asked is that I was surprised that it was necessary to rewrite optimization code in C++ after prototyping in python. It would be nice if codegen also turned python optimization code into C++ optimization code. Is there a reason you didn't do this? (Maybe it's harder than it seems.)

[aaron-skydio]

Yeah there are a couple things we can do here, some of which we just haven't gotten to yet.

I think generating code to build up the Values is generally something we can't do as much of - we have some ideas here for generating structures to mirror a Python Values if you've set one of those up, but nothing that would fill out the whole structure for you. My sense is that there's typically enough application-specific surrounding logic that something more there would be hard to make generally useful.

We can definitely generate more of the factor graph though. I think there are a couple different axes here:

1. For problems that are fixed size at codegen time, the Python OptimizationProblem class is designed to model these and be able to generate a single factor representing the entire problem (with all its sparsity). We don't have a tutorial for how to do this because it isn't as polished as we'd like right now, but that's what it's for.
2. For problems whose structure isn't known until runtime (e.g. with varying numbers of timesteps, poses, measurements, etc) we can definitely generate something that will give you a sym::Factor directly to add to a problem, instead of just generating the actual linearization function like we do now. I don't have much in mind beyond this that I think would be useful here, but very open to ideas.

I'm a little curious about your use case - are you more trying to speed up optimizing from Python, or make it easier to deploy the optimization in a pure C++ environment?