Initial Guess - SR3 - Custom Library [BUG]

[IamKFJ]

I am using pysindy to identify the systemdynamics of a system with three states and one control input.
I am running into a problem when trying to provide initial guesses for the parameters.

When i provide a set of parameters in the dimension 3x5 an error message prompting me to provide initial guesses in the dimension 3x4 is sent.
When i rpovide initial guesses in the dimension 3x4 the error message prompts me to provide initial guesses in the dimesion 3x5.

The code is provided below.
The custom library is currently shrunken down because i wanted to replicate this with system as simple as possible.

I am fairly new to using pysindy.
Am I missing something?

Thanks a lot for your help.

Reproducing code example:

import pysindy as ps
from pysindy import SINDy

feature_names = ['x', 'v', 'F', 'u']

x_library_functions = [
    lambda: 1,
    lambda x: x,
]
x_library_functions_names = [
    lambda: 1,
    lambda x: x,
]

initial_guess_manual = np.array([[0, 0, 1, 0, 0], [1, 1, 1, 1, 0], [0, 0, 0, 0, 1]])
lib = ps.feature_library.CustomLibrary(library_functions=x_library_functions,function_names=x_library_functions_names,interaction_only=True)
model = SINDy(feature_library= lib,
              feature_names=feature_names,
              optimizer=ps.SR3(initial_guess=initial_guess_manual, normalize_columns=False, threshold=opt_threshold, thresholder='l1', max_iter=50))
model.fit(x=np.transpose(X_Data), t=dT, u=np.transpose(U_Data), library_ensemble=True, ensemble=True, quiet=True)

Error message:

Traceback (most recent call last):
File "./...file_path.../WP5000_fnn_MPC.py", line 241, in
sdot_new, t_SysIdent, coeffs_arr, method_characteristics = SINDYc(X_Data=X_Data, U_Data=U_Data, **args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./...file_path...//WP5000_functions_encapsuled.py", line 357, in SINDYc
model.fit(x=np.transpose(X_Data), t=dT, u=np.transpose(U_Data), library_ensemble=True, ensemble=True, quiet=True)
File "./...file_path.../lib/python3.11/site-packages/pysindy/pysindy.py", line 414, in fit
self.model.fit(x, x_dot)
File "./...file_path.../lib/python3.11/site-packages/sklearn/base.py", line 1152, in wrapper
return fit_method(estimator, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./...file_path.../lib/python3.11/site-packages/sklearn/pipeline.py", line 427, in fit
self._final_estimator.fit(Xt, y, **fit_params_last_step)
File "./...file_path.../lib/python3.11/site-packages/pysindy/optimizers/sindy_optimizer.py", line 60, in fit
self.optimizer.fit(x, y)
File "./...file_path.../lib/python3.11/site-packages/pysindy/optimizers/base.py", line 178, in fit
self._reduce(x_normed, y, **reduce_kws)
File "./...file_path.../lib/python3.11/site-packages/pysindy/optimizers/base.py", line 347, in _reduce
self.opt.fit(x_ensemble, y_ensemble)
File "./...file_path.../lib/python3.11/site-packages/pysindy/optimizers/base.py", line 168, in fit
raise ValueError(
ValueError: initial_guess shape is incompatible with training data. Expected: (3, 4). Received: (3, 5).

PySINDy/Python version information:

1.7.5 3.11.5 (main, Sep 11 2023, 08:31:25) [Clang 14.0.6 ]

[Jacob-Stevens-Haas]

Good catch, and thanks for providing this! I can reproduce on 1.7.5 and on master. The bug was introduced when creating EnsembleOptimizer was added to replace the old method of ensembling via model.fit(). It appears that the inner optimizer requires a 3x5 initial guess (correctly), whereas the EnsembleOptimizer requires a 3x4 (incorrectly), which is why you were getting errors for either one. See next comment.

Modified your script, since yours had some name errors:

import pysindy as ps
from pysindy import SINDy
import numpy as np

feature_names = ['x', 'v', 'F', 'u']

x_library_functions = [
    lambda: 1,
    lambda x: x,
]
x_library_functions_names = [
    lambda: 1,
    lambda x: x,
]
X_Data = np.zeros((5, 3))
U_Data = np.zeros((5, 1))
dT = .1

initial_guess_manual = np.array([[0, 0, 1, 0, 0], [1, 1, 1, 1, 0], [0, 0, 0, 0, 1]])
lib = ps.feature_library.CustomLibrary(library_functions=x_library_functions,function_names=x_library_functions_names,interaction_only=True)
model = SINDy(feature_library= lib,
              feature_names=feature_names,
              optimizer=ps.SR3(initial_guess=initial_guess_manual, normalize_columns=False, threshold=.1, thresholder='l1', max_iter=50))
model.fit(x=X_Data, t=dT, u=U_Data, library_ensemble=True, ensemble=True, quiet=True)

But even obeying the deprecation warning and creating the EnsembleOptimizer yourself gives the same error, and that error is on master as well. E.g.:

opt_inner = ps.SR3(initial_guess=initial_guess_manual, normalize_columns=False, threshold=.1, thresholder='l1', max_iter=50)
opt_outer = ps.EnsembleOptimizer(opt_inner, bagging=True, library_ensemble=True)

model = SINDy(feature_library= lib,
              feature_names=feature_names,
              optimizer=opt_outer)
model.fit(x=X_Data, t=dT, u=U_Data, quiet=True) # remove quiet=True for `master` branch

[Jacob-Stevens-Haas]

Actually, it looks like the bug predates EnsembleOptimizer. I believe it happens when dropping terms out of a library ensemble, but not adjusting the initial guess. I'm not sure what the right solution is. Should library ensembling w/initial guess (a) be allowed, and EnsembleOptimizer removes terms that correspond to features left out of the library bag, or (a) be disallowed, since it's unclear whether an initial guess is meaningful when modified in that manner? Obviously the error only occurs in optimizers that actually use the provided initial guess.

I looked at the Ensemble-SINDy paper of @urban-fasel, and the only time it mentions an initial guess is the one STLSQ generates for itself. So it feels like both (a) and (b) are reasonable - (a) allows people to investigate if this method would be useful, but (b) makes it easier for newer users to identify what SINDy methods make sense for their problem.

@urban-fasel @akaptano - what are your thoughts on the matter? And @IamKFJ, can you let us know, in your use case, how you chose library ensembling, SR3, and your initial guess?

[IamKFJ]

Thanks for the quick reply!

I chose library ensembling because i saw that it can lead to better approximation in one of the youtube tutorials. I didn't think about how ensembling interfeers with the initial guesses.
I don't remember why I chose the SR3. I think i got it from building on an existing code i found online.
I want to use initial guesses because i wanted to prime the algorithm towars a choice of parameters i know to be somewhat physical (mainly setting them positive/negative/zero). The ones in the initial guesse i put into the code are just placeholders.