How to implement a different loss function in the SR3 optimizer?

[adboon]

In my research, I have started using SINDy, and it is working quite well, thank you for making this package. I have run into a problem, however. I want to change the optimizer so the loss function penalizes errors in the extremes more. To achieve this goal I want to change the loss function in SINDy from $$0.5|y-Xw|^2 $$ to $$0.5|y-Xw|^\gamma $$ where $\gamma$ is a power of 2 (based on Chang 2021 https://arxiv.org/pdf/2007.04441.pdf)

I have not been able to implement this change in the code. I chose the SR3 regression framework and have been playing around to figure out how I would be able to implement a different loss function. I expected line 248 (R2 = (y - np.dot(x, coef_full)) ** 2) in pysindy/optimizers/sr3.py to be the loss function, but it does not affect the optimization at all. Changing the variable "x_transpose_y" does seem to result in changes in the optimization, but I haven't been able to make sense of how this variable fits into the framework described in Champion 2020 (https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9194760) or Zheng 2019 (https://arxiv.org/abs/1807.05411). I also haven't been able to figure it out with the help of the SINDy documentation or other issues here.

Could you share some ideas on how I would be able to implement a different loss function in the SR3 optimizer?

Thanks in advance!

[Jacob-Stevens-Haas]

Great question. You're on the right approach - either directly changing the SR3 code or subclassing and overriding the _objective() method. I don't know why your change to that line isn't working, but you might start by checking out the most recent commit - that line is now 253. If it's still not changing the optimization result, check info about the cvxpy optimizer - was it finding a good fit, or taking one of the other paths through the code because it couldn't find a fit? If the latter, you might need to specify a better problem.

The trapping-plumes branch contains my efforts at standardizing some of the optimizers that use cvxpy. It might be easier to work off of, but there's a long merge conflict taking place right now, so maybe not.

I don't have time to take a look right now whether the SR3 module correctly implements those papers. If not, it wouldn't be the first time - I'd be very grateful if you'd take a deeper look, but regardless I can spend more time in a week or so. The first step is probably to finish the merge of trapping in order to not have a conflicting branch out in the wild.

[adboon]

Thank you for your quick response Jacob-Stevens-Haas!
I have looked at your reply and started working on your suggestion. However, as it goes with research, I discovered some other problems. Those need to be solved before this implementation will become relevant for my work, so I am focussing on them now.
Hopefully, I can get back to this in some time