[Major] L-BFGS Optimizer

[karl-richter]

🔬 Background

• Why is this change needed? Is there a related issue or a new feature to be added?

🔮 Key changes

• Explain the main changes introduced by this pull request for the reviewer.

📋 Review Checklist

• I have performed a self-review of my own code.
• I have commented my code, added docstrings and data types to function definitions.
• I have added pytests to check whether my feature / fix works.

Please make sure to follow our best practices in the Contributing guidelines.

[github-actions]

Model Benchmark

Benchmark	Metric	main	current	diff
AirPassengersBFGS	MAE_val	-	18.9926	-
AirPassengersBFGS	RMSE_val	-	23.2203	-
AirPassengersBFGS	Loss_val	-	0.0028	-
AirPassengersBFGS	MAE	-	6.3498	-
AirPassengersBFGS	RMSE	-	7.8142	-
AirPassengersBFGS	Loss	-	0.0002	-
AirPassengersBFGS	time	-	26.32	-
YosemiteTemps	MAE_val	1.71298	18.9074	1003.77%	❌
YosemiteTemps	RMSE_val	2.26069	21.7646	862.74%	❌
YosemiteTemps	Loss_val	0.00095	0.08728	9062.98%	❌
YosemiteTemps	MAE	1.45187	17.9308	1135.02%	❌
YosemiteTemps	RMSE	2.16721	21.1658	876.64%	❌
YosemiteTemps	Loss	0.00066	0.06248	9411.61%	❌
YosemiteTemps	time	131.13218	104.07	-20.64%	🎉
PeytonManning	MAE_val	0.58159	2.47694	325.89%	❌
PeytonManning	RMSE_val	0.72216	2.98224	312.96%	❌
PeytonManning	Loss_val	0.01239	0.20553	1558.37%	❌
PeytonManning	MAE	0.41671	2.16775	420.21%	❌
PeytonManning	RMSE	0.55961	2.85653	410.45%	❌
PeytonManning	Loss	0.00612	0.13845	2162.59%	❌
PeytonManning	time	16.25471	43.7	168.85%	❌
AirPassengers	MAE_val	15.26985	232.253	1420.99%	❌
AirPassengers	RMSE_val	19.42088	250.753	1191.15%	❌
AirPassengers	Loss_val	0.00195	0.318	16235.9%	❌
AirPassengers	MAE	9.82902	153.771	1464.46%	❌
AirPassengers	RMSE	11.70055	162.458	1288.47%	❌
AirPassengers	Loss	0.00056	0.11342	20287.23%	❌
AirPassengers	time	5.76797	18.13	214.32%	❌
YosemiteTempsBFGS	MAE_val	-	1.587	-
YosemiteTempsBFGS	RMSE_val	-	2.1777	-
YosemiteTempsBFGS	Loss_val	-	0.0009	-
YosemiteTempsBFGS	MAE	-	1.4415	-
YosemiteTempsBFGS	RMSE	-	2.1732	-
YosemiteTempsBFGS	Loss	-	0.0006	-
YosemiteTempsBFGS	time	-	573.76	-

Model training plots

Model Training

PeytonManning

YosemiteTemps

AirPassengers