Incorrect reported estimate/used memory with GPU solver

[craig-warren]

I think this is an error with how the memory usage is being reported when run with the solver on GPU, i.e. it is not the estimated memory that is wrong, rather the reported actual usage. At the moment we use the psutil module with the memory info function (https://psutil.readthedocs.io/en/latest/#psutil.Process.memory_info). If anyone can recommend a better method please let me know, especially for returning GPU used memory.

[craig-warren]

I've implemented some reporting of the memory that is used on the GPU after the simulation has run -
a1b5355#diff-0c31f357656dc5640678b6a63583dee9

For smaller models the reported memory usage (using plutil 'Resident Set Size') does vary from run to run and depending (I think) on the machine also.

I'd be grateful for some feedback.

[spaceVStab]

Many articles suggest using 'Unique Set Size' as a measure for memory usage unique to the current process. I was wondering how you have decided the value of stdoverhead while estimating Memory usage.

Also, gputil provides the GPU memory usage by nvidia-smi. It can be used to report the GPU memory usage after the completion of simulation.

[craig-warren]

@spaceVStab thanks, I see http://grodola.blogspot.com/2016/02/psutil-4-real-process-memory-and-environ.html, I will change psutil to use 'Unique Set Size'. If you have any thoughts on better ways to estimate the memory usage beforehand that would be helpful. The value of stdoverhead is just based on running memory profiler and noting that ~50MB is always required before any of the main arrays are initialised.

I will look into gputil as it can probably do a lot of the things we are currently trying to code ourselves. We currently use a call to PyCUDA to get the memory used on the last iteration of the simulation using https://documen.tician.de/pycuda/driver.html#pycuda.driver.mem_get_info