Memory build-up on various locations

[hasan7n]

When using the FeTS tool (script), we noticed two sources of memory build-up:

1. Memory build-up due to figure_generator (lines), around 200MB per call.
2. Memory build-up due to GaNDLF's main_run function (lines), around 400MB per call.

Further profiling of GaNDLF'S main_run showed the information below (note: the profiling was done on GaNDLF's commit 616b37bafad8f89d5c816a88f44fa30470601311)

• Around 100MB build-up caused by this line. Note that clearing the model variable at the end of the inference loop will resolve this.
• Around 280MB build-up caused by this line. Note that clearing the compiled_model variable will not resolve this, which suggests that the build-up happens inside the compiled ov library.
• Other multiple lines of code in different places cause few MBs build-ups

[sarthakpati]

Thanks for the detailed report!

I am guessing the main_run buildup could be related to #780, but this needs to be investigated.

[hasan7n]

FWIW, this is the memory profiler used:
https://pypi.org/project/memory-profiler/

[sarthakpati]

FYI, 1 is not in GaNDLF, and is a different library (https://github.com/CBICA/FigureGenerator). Though the underlying dependencies are very similar (SimpleITK and numpy).

[github-actions]

Stale issue message