[Major] Dataloader: Just-In-Time tabularization

[ourownstory]

Complete re-write of dataloader: Instead of tabularizing samples ahead of batching, samples are tabularized and features computed during batch creation. Trades Memory use for Compute.

This is an early implementation that works, but that needs to be compute-optimized.

• Addresses high data consumption when loading global models with many IDs and large datasets. See Drastically increased memory load #1469

[codecov]

Codecov Report

Attention: 25 lines in your changes are missing coverage. Please review.

Comparison is base (5a3fabf) 88.51% compared to head (2f38531) 88.24%.

Files	Patch %	Lines
neuralprophet/time_dataset.py	93.31%	20 Missing ⚠️
neuralprophet/forecaster.py	58.33%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1529      +/-   ##
==========================================
- Coverage   88.51%   88.24%   -0.27%     
==========================================
  Files          41       41              
  Lines        5330     5344      +14     
==========================================
- Hits         4718     4716       -2     
- Misses        612      628      +16     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

Model Benchmark

Benchmark	Metric	main	current	diff
YosemiteTemps	MAE_val	0.56412	0.60324	6.94%	⚠️
YosemiteTemps	RMSE_val	0.83161	0.88159	6.01%	⚠️
YosemiteTemps	Loss_val	0.0004	0.00045	12.29%	❌
YosemiteTemps	MAE	0.98449	0.98956	0.52%	✅
YosemiteTemps	RMSE	1.75389	1.73958	-0.82%	✅
YosemiteTemps	Loss	0.00132	0.0013	-1.5%	✅
YosemiteTemps	time	34.5426	833.36	2312.56%	❌
EnergyPriceDaily	MAE_val	5.41157	5.65526	4.5%	⚠️
EnergyPriceDaily	RMSE_val	6.71538	6.86834	2.28%	✅
EnergyPriceDaily	Loss_val	0.02525	0.02645	4.75%	⚠️
EnergyPriceDaily	MAE	5.94936	6.2678	5.35%	⚠️
EnergyPriceDaily	RMSE	7.9833	8.34813	4.57%	⚠️
EnergyPriceDaily	Loss	0.02579	0.02826	9.59%	❌
EnergyPriceDaily	time	15.1859	268.35	1667.1%	❌
PeytonManning	MAE_val	0.34955	0.35129	0.5%	✅
PeytonManning	RMSE_val	0.50049	0.50154	0.21%	✅
PeytonManning	Loss_val	0.01771	0.0178	0.53%	✅
PeytonManning	MAE	0.34653	0.34697	0.13%	✅
PeytonManning	RMSE	0.49312	0.49406	0.19%	✅
PeytonManning	Loss	0.01464	0.01468	0.28%	✅
PeytonManning	time	11.8971	269.68	2166.76%	❌
AirPassengers	MAE_val	30.6285	30.9382	1.01%	✅
AirPassengers	RMSE_val	31.5254	32.1915	2.11%	✅
AirPassengers	Loss_val	0.01277	0.01332	4.27%	⚠️
AirPassengers	MAE	6.16861	6.91875	12.16%	❌
AirPassengers	RMSE	7.85225	8.9841	14.41%	❌
AirPassengers	Loss	0.00065	0.00078	20.92%	❌
AirPassengers	time	7.75698	30.72	296.03%	❌

Model training plots

Model Training

PeytonManning

YosemiteTemps

AirPassengers

EnergyPriceDaily