Very large memory usage of ERM reconciler with method = 'reg' or 'reg_bu'

[dclaz]

Hello, is there some extra step being taken with the ERM regularization methods (reg and reg_bu) that require a very large memory overhead? Relative to ERM with method=closed ?

I've got a large-scale time series forecast task and there is a hierarchical nature that I hope can be used to improve both the aggregate and granular forecasts.

The most granular series are sales office / material combinations, which can be nested under either a sales office or material
The unique counts of each of these are given below:

sales office unique:		 28
material unique:		 3213
sales office X material unique:	 11449

So in total, there are 11449+3213+28=14690 series that are to be forecasted. The S matrix looks like this:

I've produced test forecasts and have been able to successfully compare the BottomUp and ERM(Closed) reconcilers with the base forecasts, but unfortunately the bottom-up reconciler cannot improve the granular sales office X material forecasts and the ERM-Closed method results in very poor MSE at the sales office X material level (but vastly improves the others).

I was hoping that utilizing regularisation with the ERM reconciler with method='reg' or method='reg_bu' would be helpful here, but I run into a memory issue:

MemoryError: Unable to allocate 80.9 TiB for an array with shape (14690, 9, 11449, 14690) and data type float32

My understanding is that these methods use a LASSO to obtain the optimal weights. I'm not sure about what the reconciler is doing in the background, but I do believe there should be very memory-efficient implementations of the LASSO that could be leveraged here.

For very wide S matrices, I think that:

$$\mathbf{P}^{*} = \text{argmin}{\mathbf{P}} ||\mathbf{Y}-\mathbf{S} \mathbf{P} \hat{Y} ||^{2}{2} + \lambda ||\mathbf{P}-\mathbf{P}{\text{BU}}||{1}
$$

Should be much easier to compute than

$$\mathbf{P}^{*} = \left(\mathbf{S}^{\intercal}\mathbf{S}\right)^{-1}\mathbf{Y}^{\intercal}\hat{\mathbf{Y}}\left(\hat{\mathbf{Y}}\hat{\mathbf{Y}}\right)^{-1} $$

Additionally, I am surprised that the closed-form solution for ERM works at all and only takes ~13 seconds, given it has to invert a 14690x14690 matrix in this scenario.

[kdgutier]

Thanks for the comment @dclaz, and sorry for the inconvenience.

Currently, the lasso ERM is creating matrices using Kronecker products which are very inefficient in memory and computation.

We plan to work on the efficiency of both MinTrace and ERM to make them usable.
In the meantime, a hacky solution to obtain ERM results is to get a giant machine.

[dclaz]

Thanks for your rapid reply @kdgutier. Sadly I don't think I can afford a multi-petabyte machine in this instance.

[kdgutier]

Yep sorry,

An efficient ERM-lasso is among the most important improvements I have to make to the library. I will ping you when it is ready :)

Meanwhile ERM-closed or the MinTrace variants could be an alternative.

Thanks for reporting the memory issue.

[dclaz]

Yep sorry,

An efficient ERM-lasso is among the most important improvements I have to make to the library. I will ping you when it is ready :)

Meanwhile ERM-closed or the MinTrace variants could be an alternative.

Thanks for reporting the memory issue.

That would be amazing, thank you so much.