APPTAINERENV_CUDA_VISIBLE_DEVICES env not working

[joehays]

All,

I have a system with multiple GPUs. I want my apptainer to only have access to one of the GPUs (#2). According to,

https://apptainer.org/docs/user/1.2/gpu.html#

I should be able to use,

APPTAINERENV_CUDA_VISIBLE_DEVICES=2 apptainer exec --nv <my_container_image>.sif bash -c "nvidia-smi"

but when I run this I see all my GPUs listed.

How can I isolate the container to only use specified GPUs? The documented method is not working for me.

Thanks.

~$ apptainer --version
apptainer version 1.2.5

Expected behavior

I expected APPTAINERENV_CUDA_VISIBLE_DEVICES=2 apptainer exec --nv <my_container_image>.sif bash -c "nvidia-smi" to only include a single GPU in the list.

Actual behavior

All GPUs on my system were included.

Steps to reproduce this behavior

APPTAINERENV_CUDA_VISIBLE_DEVICES=2 apptainer exec --nv <my_container_image>.sif bash -c "nvidia-smi"

What OS/distro are you running

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

How did you install Apptainer

sudo apt install -y singularity

[DrDaveD]

That's because nvidia-smi ignores the CUDA variable. Instead, run the tensorflow image and python command shown in the documentation. The setting of CUDA_VISIBLE_DEVICES made a difference in the recognized GPUs when I tried it just now.

Apptainer isn't doing anything special for this feature, it's all handled by the GPU software.