espresso with CUDA unexpected behaviour

[stekajack]

Hi,

I find it curious that espresso with CUDA enabled will spawn a GPU process regardless if the simulation actually uses GPUs. basically import espressomd will spawn a process that reserves some amount of VRAM. Is this intended behaviour? is there a particular reason this should happen?

[jngrad]

Hi,

I don't know for sure, but this is probably a relic of how the CUDA interface had to be initialized at the very beginning (during import espressomd), so that GPU algorithms like Stokesian, MMM1D, P3M and electrokinetics could access the device properties and device arrays on-demand, i.e. without having to initialize anything on the device other than the particle data. The GPU code a changed somewhat since 4.1 Perhaps the code is now in a state where GPU process initialization can be deferred until the C++ object of a GPU algorithm is initialized.

I could see two obstacles to such a refactoring: 1) CUDA state initialization can only happen on the head node, but the MPI logic cannot be included in .cu files, so several CUDA algorithms would require a hook to call the initialization code only on the head node; and 2) the electrokinetics code doesn't have a constructor and therefore its lifetime isn't well-defined, one would have to make sure whether the ek_initialized global flag can be a reliable proxy to signal that CUDA initialization is required.

That would be a lot of work, though. If this is important to you and you would like to investigate the issue, it's probably better to wait until ESPResSo 4.3 is released, as it will be much easier to read through the CUDA code. In particular, the old GPU electrokinetics and GPU LB code will be replaced by waLBerla code.

[stekajack]

Hi,
thanks for the insight. I cannot tell whether it is important or not really. On face value i can see that having espresso jobs that so not use GPU at all on a version of espresso compiled with CUDA support, will not only take up VRAM but will also ramp up the power draw of the GPU a bit which means it is doing something. It is an issue with a particular workflow where i have a bunch of single core jobs running in the background and a massive GPU job in parallel. Here the issue is that side jobs eat up VRAM and similarly to running xorg window, introduce ever so slight overhead on the GPU. But clearly this is neither a particularly relevant workflow, nor a performance issue work the effort xD id be happy to play around with this once waLBerla replaces the current implementation. If anything so i learn the tools better while i fail miserably. Cheers

[jngrad]

You may consider compiling two versions of ESPResSo: one with GPU support, and one without. It is then just a matter of setting the $PATH environment variable at the top of the job script to point to either /usr/local/espresso414cuda or /usr/local/espresso414, depending on whether that job needs a GPU or not. On a Fedora or CentOS workstation, you could write module files so that you can do module load espresso414cuda or module load espresso414, which is more convenient than manually editing the $PATH.