CuPy loading error when trying to use custom force object on GPU cluster

[Viinnert]

Hello!

I am trying to simulate active polymers using HOOMD-blue. To apply the active forces i am using the hoomd.md.force.Custom() object. when running the code locally on CPU everything works fine.

To run the code remotely on a supercomputer I am using the singularity container I recently downloaded from "https://glotzerlab-software.readthedocs.io/en/latest/container.html". When I use this image to run my simulation on the supercomputer's CPU, everything is working fine too. However, problems arise when I try to use the supercomputer's GPU. When I want to use this GPU I keep getting the following error:

Traceback (most recent call last):

File "cupy_backends/cuda/_softlink.pyx", line 25, in cupy_backends.cuda._softlink.SoftLink.init
File "/usr/lib/python3.9/ctypes/init.py", line 374, in init
self._handle = _dlopen(self._name, mode)
OSError: libnvrtc.so.11.2: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/gpfs/home1/vkrabbenborg/MscThesisAmsterdam/Msc-Thesis-Amsterdam/experiments/../src/test.py", line 76, in
simulation.run(T_sim)
File "/opt/glotzerlab/lib/python3.9/site-packages/hoomd/simulation.py", line 562, in run
self._cpp_sys.run(steps_int, write_at_start)
File "/gpfs/home1/vkrabbenborg/MscThesisAmsterdam/Msc-Thesis-Amsterdam/experiments/../src/test.py", line 16, in set_forces
dev_indices = snapshot.particles.rtag[self.motor_indices]
File "/opt/glotzerlab/lib/python3.9/site-packages/hoomd/data/array.py", line 655, in getitem
arr = self._coerce_to_ndarray()[index]
File "cupy/_core/core.pyx", line 1521, in cupy._core.core._ndarray_base.getitem
File "cupy/_core/_routines_indexing.pyx", line 42, in cupy._core._routines_indexing._ndarray_getitem
File "cupy/_core/core.pyx", line 829, in cupy._core.core._ndarray_base.take
File "cupy/_core/_routines_indexing.pyx", line 131, in cupy._core._routines_indexing._ndarray_take
File "cupy/_core/_routines_indexing.pyx", line 825, in cupy._core._routines_indexing._take
File "cupy/_core/_kernel.pyx", line 920, in cupy._core._kernel.ElementwiseKernel.call
File "cupy/_core/_kernel.pyx", line 945, in cupy._core._kernel.ElementwiseKernel._get_elementwise_kernel
File "cupy/_util.pyx", line 64, in cupy._util.memoize.decorator.ret
File "cupy/_core/_kernel.pyx", line 728, in cupy._core._kernel._get_elementwise_kernel
File "cupy/_core/_kernel.pyx", line 82, in cupy._core._kernel._get_simple_elementwise_kernel_from_code
File "cupy/_core/core.pyx", line 2254, in cupy._core.core.compile_with_cache
File "/opt/glotzerlab/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 484, in _compile_module_with_cache
return _compile_with_cache_cuda(
File "/opt/glotzerlab/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 499, in _compile_with_cache_cuda
arch = _get_arch()
File "cupy/_util.pyx", line 64, in cupy._util.memoize.decorator.ret
File "/opt/glotzerlab/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 148, in _get_arch
nvrtc_max_compute_capability = _get_max_compute_capability()
File "cupy/_util.pyx", line 64, in cupy._util.memoize.decorator.ret
File "/opt/glotzerlab/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 126, in _get_max_compute_capability
major, minor = _get_nvrtc_version()
File "/opt/glotzerlab/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 115, in _get_nvrtc_version
_nvrtc_version = nvrtc.getVersion()
File "cupy_backends/cuda/libs/nvrtc.pyx", line 56, in cupy_backends.cuda.libs.nvrtc.getVersion
File "cupy_backends/cuda/libs/nvrtc.pyx", line 57, in cupy_backends.cuda.libs.nvrtc.getVersion
File "cupy_backends/cuda/libs/_cnvrtc.pxi", line 72, in cupy_backends.cuda.libs.nvrtc.initialize
File "cupy_backends/cuda/libs/_cnvrtc.pxi", line 76, in cupy_backends.cuda.libs.nvrtc._initialize
File "cupy_backends/cuda/libs/_cnvrtc.pxi", line 143, in cupy_backends.cuda.libs.nvrtc._get_softlink
File "cupy_backends/cuda/_softlink.pyx", line 32, in cupy_backends.cuda._softlink.SoftLink.init
RuntimeError: CuPy failed to load libnvrtc.so.11.2: OSError: libnvrtc.so.11.2: cannot open shared object file: No such file or directory

The last line of code of my own files in this error is the "dev_indices = snapshot.particles.rtag[motor_indices]" in the MolecularMotors.py file. Do you know what is causing this problem? I'm getting the idea that it might be an error in the singularity container. Thank you in advance!

A reduced script that reproduces the error:
test.txt

NOTE:
While creating the reduced script to reproduce the error i noticed the following thing: If i slightly change the code, such that the "motor_indices" and the "dev_indices" are an integer instead of a numpy array, i get the following error:

_Traceback (most recent call last):
File "/gpfs/home1/vkrabbenborg/MscThesisAmsterdam/Msc-Thesis-Amsterdam/experiments/../src/test.py", line 73, in
simulation.run(T_sim)
File "/opt/glotzerlab/lib/python3.9/site-packages/hoomd/simulation.py", line 562, in run
self._cpp_sys.run(steps_int, write_at_start)
File "/gpfs/home1/vkrabbenborg/MscThesisAmsterdam/Msc-Thesis-Amsterdam/experiments/../src/test.py", line 20, in set_forces
arrays.force[dev_indices] += np.array([0, 1, 0])
File "/opt/glotzerlab/lib/python3.9/site-packages/hoomd/data/array.py", line 653, in getitem
arr = self._coerce_to_ndarray()[index._coerce_to_ndarray()]
File "/opt/glotzerlab/lib/python3.9/site-packages/hoomd/data/array.py", line 678, in coerce_to_ndarray
raise HOOMDArrayError(
hoomd.data.array.HOOMDArrayError: Cannot access HOOMDGPUArray outside context manager. Use cupy.array(obj, copy=True) inside context manager instead.

With the script
test2.txt

It says that it cannot access the GPU force arrays outside the context managers. But i am not accessing it outside the context manager.

I hope i cleary presented the problem. Thanks in advance!

[joaander]

The glotzerlab/software:nompi containers were built against CUDA 11.2 which does not contain libnvrtc.so.11.1: https://github.com/glotzerlab/software/blob/5b191b849c0ef2c543d6711fbe0a94c4ac33da0c/make_dockerfiles.py#L41

I will rebuild the image against CUDA 11.8 later today. However, I would suggest that you use the conda package instead. Problems like the one you report are less likely to occur with a package that runs directly on the host. The hoomd package on conda-forge offers the same capabilities as the nompi container, is easier to install, manage custom software environments, and update.

Due to numerous issues like this, I have deprecated the glotzerlab-software container builds and will no longer update them after June 1, 2024. I have replaced these with conda recipes that allow you to locally build a hoomd conda package with MPI (and optionally CUDA) support linked to your cluster's native libraries.

[joaander]

The CUDA 11.8 latest build is now available: https://hub.docker.com/r/glotzerlab/software/tags