Possible [BUG] - High memory usage causing colab notebooks to crash and local system swap usage.

[samLozier]

Caveat that I'm new to using most of these libraries, so this could just be operator error. My expectation was that given that these examples were based on a Kaggle competition, they should generally be able to run on readily available free/cheap systems.

Describe the bug
I attempted to follow the M5 examples in the "examples" repo. I could not get the docker container to work properly locally so I decided to follow the more manual process. On my local machine it's taking hours to run and using all available memory and significant swap, in colab the notebook crashes in a matter of seconds due to using all available memory

To Reproduce
Steps to reproduce the behavior:

On my local machine (2013 macbook pro, conda python 3.7) - I can get the code to run, by it took significantly more time that I expected:

With the following settings, this cell took 6 hours to run (copied from the M5 example notebook):
from hts import HTSRegressor clf = HTSRegressor(model='prophet', revision_method='OLS') model = clf.fit(df, hierarchy)

Switching to the "low memory" mode - it actually ran faster.. only 2 hours. however the very next cell, the forecast cell, has been running for over 12 hours and is only at 60%. Both training and prediction have taken over my system resources, I'm seeing several python3 processes with the worst one using 20gb of memory (only 8gb actually on board), so this is clearly using a LOT of swap space, which I expect is responsible for slowing it down..
from hts import HTSRegressor clf = HTSRegressor(model='prophet', revision_method='OLS', low_memory=True) model = clf.fit(df, hierarchy)

Finally, attempting to run it in a colab notebook caused the notebook to crash due to using up all available ram in <30s.
To be clear i'm referring to this same code:
from hts import HTSRegressor clf = HTSRegressor(model='prophet', revision_method='OLS', low_memory=True) model = clf.fit(df, hierarchy)

Expected behavior
A clear and concise description of what you expected to happen.

• relatively quick run time, not crashing.

I'm trying to understand if this is expected behavior, user error, or a possible bug related to memory management or starting too many processes at once. Based on the fact that the example notebook was designed with the goal of demonstrating the efficacy of the package for solving a kaggle competition, and the fact that several leading notebooks run in a minute or two, I though it was unusual to experience multi-hour run times and crashes when using Colab. Did I just need different settings, or is this a bug?

Desktop (please complete the following information):

• OS: [e.g. OS X Catalina] OSX Catalina,
• scikit-hts version: [e.g. 0.2.1] packages from the "requirements.txt"
• Python version: [e.g. 3.7.4] python 3.7 (anaconda),

Colab - default colab settings. Out of curiosity, I tried it with both TPU and GPU, but as expected, they made no difference.

[aakashparsi]

Hey @samLozier, I'm facing a similar issue like taking too long time, high memory usage. Have you got any solution for that?