Residual to Factor map

[asa]

In debugging the contribution of factors to the overall error I would like to get a map from residual entry in Linearization to factor and additionally to the residual entry in the factor. I have made something that works by replicating the logic in sym::Linearizer::BuildInitialLinearization(), but I fear that will go out of date as symforce evolves.

Describe the solution you'd like
would it be possible to return this mapping in Linearization or in OptimizationStats when we have the Optimizer debug flag be true?

Describe alternatives you've considered
I have a current implementation that does the job. I think this is a general problem(and there may be way to do this already) that could benefit others who are looking for fine grained residual -> factor mapping.

[aaron-skydio]

How are you identifying the factors? By index in the factors list passed to the optimizer, or by some other tag? We should be able to make something that returns that mapping in order at least (they should all be in the same order in the residual as they are in the factor list, so you just need the residual size for each factor - although at some point we may have put all the sparse factors at the end of the residual if you were mixing sparse and dense factors)

[asa]

Yes, I am keeping the array of factors around I submit to the solve. (actually I have my own factor representation from which the sym::Factors are created so that I can keep type information for each factor, and I lookup into this array). Getting this back in the same order would be great. Agree on appending sparse after dense.

[aaron-skydio]

Yeah I think we could do this pretty easily, as a function that returns the offset and residual size for a given factor index, since we should already have that information stored. We could instead compute the whole mapping and return it as a vector or something, but that seems unnecessary. Should have a test that checks it as well. Feel free to submit a PR (or anyone reading this feel free to tackle it), one of us might also do this soon

[asa]

Happy to contribute to this when I have some time. Thanks for taking a look.