Here is the output of a run of Higgs boson with memory_profiler:

mprof run benchmarks/bench_higgs_boson.py --no-lightgbm --n-trees 100 --n-leaf-nodes 255
mprof plot

• on master:

• on this branch:

So basically there is no significant difference. I think it means that the memory used by the reused splitting context instance is very little compared to the memory used by the histograms stored on each node instance. Those linked nodes in the grower tree it-self cannot be reused very easily without significant refactoring as far as I know.

Also note that according to the above chart, the memory usage of pygbm is not that bad even if it's fluctuating because of the garbage collections of the temporary histogram arrays.

Thanks for the plots!

Indeed it doesn't seem to help much, at least with this number of samples.

I don't think histograms use much memory compared to the splitter though, according to my calculations:

histograms for one iteration:

• n_bins * n_features * sizeof(HIST_DTYPE) * n_nodes = 255 * 28 * 12 * 255 = 22MB

SplittingContext for one iteration (re-used):

• there are 5 arrays of size uint32 or float32 (gradients, hessians, partition, left_indices_buffer, right_indices_buffer) so total = 5 * 4 * n_samples = 220MB

That being said, LightGBM uses an LRU cache for the histograms.

Maybe it would be helpful if @maartenbreddels could try this branch on the big dataset that used to cause a memory stress? Until then I'm in favor for closing the PR, and maybe resuscitate it eventually if needed.

Indeed you are right about the relative sizes of the datastructures. But then I don't understand what is causing the fluctuations when running the benchmark on this PR (with the reset-able splitting context variant).

Well maybe is just like you said, the TreeNode objects (that contain the histograms) accumulate, and are freed regularly?

Happy to test this branch out (maybe this week? can't guarantee), note that I was using OS X, maybe that mattters.

Here is a possible fix for the accumulated memory usage of the stored histograms that proved very efficient in scikit-learn: scikit-learn/scikit-learn#18242

Also note that Higgs boson is not the best benchmark for memory efficiency: it has many samples and few features and a single binary target. So compared to the amount of computation, it allocates few histograms.