could not attach image file to loop device: no loop devices available

[MariusCausemann]

Version of Singularity
singularity-ce version 3.11.4

Describe the bug
When running singularity container in parallel (hybrid mode), the image creation fails with "no loop device available". This used to work before but fails since the machine got updated to a newer kernel and OS.
Error message:

mpirun -n 4 singularity -v run shub://vsoch/hello-world
INFO:    Use cached image
VERBOSE: Setting HOME=/home/mariusca
VERBOSE: Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
INFO:    Use cached image
INFO:    Use cached image
VERBOSE: Setting HOME=/home/mariusca
VERBOSE: Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
VERBOSE: Setting HOME=/home/mariusca
VERBOSE: Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
INFO:    Use cached image
VERBOSE: Setting HOME=/home/mariusca
VERBOSE: Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
VERBOSE: Set messagelevel to: 4
VERBOSE: Starter initialization
VERBOSE: Check if we are running as setuid
VERBOSE: Drop root privileges
VERBOSE: Drop root privileges permanently
VERBOSE: Spawn stage 1
VERBOSE: Set messagelevel to: 4
VERBOSE: Starter initialization
VERBOSE: Check if we are running as setuid
VERBOSE: Drop root privileges
VERBOSE: Set messagelevel to: 4
VERBOSE: Starter initialization
VERBOSE: Set messagelevel to: 4
VERBOSE: Starter initialization
VERBOSE: Drop root privileges permanently
VERBOSE: Spawn stage 1
VERBOSE: Check if we are running as setuid
VERBOSE: Drop root privileges
VERBOSE: Check if we are running as setuid
VERBOSE: Drop root privileges
VERBOSE: Drop root privileges permanently
VERBOSE: Spawn stage 1
VERBOSE: Drop root privileges permanently
VERBOSE: Spawn stage 1
VERBOSE: Execute stage 1
VERBOSE: Execute stage 1
VERBOSE: stage 1 exited with status 0
VERBOSE: Get root privileges
VERBOSE: Change filesystem uid to 6494
VERBOSE: Spawn master process
VERBOSE: Create mount namespace
VERBOSE: Entering in mount namespace
VERBOSE: Create mount namespace
VERBOSE: Execute stage 1
VERBOSE: Execute stage 1
VERBOSE: Spawn RPC server
VERBOSE: stage 1 exited with status 0
VERBOSE: Get root privileges
VERBOSE: Change filesystem uid to 6494
VERBOSE: Spawn master process
VERBOSE: Create mount namespace
VERBOSE: Entering in mount namespace
VERBOSE: Create mount namespace
VERBOSE: Spawn RPC server
VERBOSE: stage 1 exited with status 0
VERBOSE: stage 1 exited with status 0
VERBOSE: Get root privileges
VERBOSE: Get root privileges
VERBOSE: Change filesystem uid to 6494
VERBOSE: Change filesystem uid to 6494
VERBOSE: Spawn master process
VERBOSE: Spawn master process
VERBOSE: Create mount namespace
VERBOSE: Create mount namespace
VERBOSE: Entering in mount namespace
VERBOSE: Entering in mount namespace
VERBOSE: Create mount namespace
VERBOSE: Create mount namespace
VERBOSE: Execute master process
VERBOSE: Spawn RPC server
VERBOSE: Spawn RPC server
VERBOSE: Default mount: /proc:/proc
VERBOSE: Default mount: /sys:/sys
VERBOSE: Default mount: /dev:/dev
VERBOSE: Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE: Found 'bind path' = /etc/hosts, /etc/hosts
VERBOSE: Default mount: /tmp:/tmp
VERBOSE: Default mount: /var/tmp:/var/tmp
VERBOSE: Default mount: /etc/resolv.conf:/etc/resolv.conf
VERBOSE: Serve RPC requests
VERBOSE: Serve RPC requests
VERBOSE: Execute master process
VERBOSE: Execute master process
VERBOSE: Execute master process
VERBOSE: Default mount: /proc:/proc
VERBOSE: Default mount: /sys:/sys
VERBOSE: Default mount: /dev:/dev
VERBOSE: Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE: Found 'bind path' = /etc/hosts, /etc/hosts
VERBOSE: Serve RPC requests
VERBOSE: Default mount: /tmp:/tmp
VERBOSE: Default mount: /var/tmp:/var/tmp
VERBOSE: Default mount: /etc/resolv.conf:/etc/resolv.conf
VERBOSE: Serve RPC requests
VERBOSE: Default mount: /proc:/proc
VERBOSE: Default mount: /sys:/sys
VERBOSE: Default mount: /dev:/dev
VERBOSE: Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE: Found 'bind path' = /etc/hosts, /etc/hosts
VERBOSE: Default mount: /proc:/proc
VERBOSE: Default mount: /sys:/sys
VERBOSE: Default mount: /dev:/dev
VERBOSE: Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE: Found 'bind path' = /etc/hosts, /etc/hosts
VERBOSE: Default mount: /tmp:/tmp
VERBOSE: Default mount: /var/tmp:/var/tmp
VERBOSE: Default mount: /etc/resolv.conf:/etc/resolv.conf
VERBOSE: Default mount: /tmp:/tmp
VERBOSE: Default mount: /var/tmp:/var/tmp
VERBOSE: Default mount: /etc/resolv.conf:/etc/resolv.conf
VERBOSE: Checking for template passwd file: /cm/shared/apps/singularity-ce/3.11.4/var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE: Creating passwd content
VERBOSE: Creating template passwd file and appending user data: /cm/shared/apps/singularity-ce/3.11.4/var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE: Default mount: /etc/passwd:/etc/passwd
VERBOSE: Checking for template group file: /cm/shared/apps/singularity-ce/3.11.4/var/singularity/mnt/session/rootfs/etc/group
VERBOSE: Creating group content
VERBOSE: Default mount: /etc/group:/etc/group
VERBOSE: /home/mariusca found within container
VERBOSE: rpc server exited with status 0
VERBOSE: Execute stage 2
FATAL:   container creation failed: mount /proc/self/fd/3->/cm/shared/apps/singularity-ce/3.11.4/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to find loop device: could not attach image file to loop device: no loop devices available
VERBOSE: Checking for template passwd file: /cm/shared/apps/singularity-ce/3.11.4/var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE: Creating passwd content
FATAL:   container creation failed: mount /proc/self/fd/3->/cm/shared/apps/singularity-ce/3.11.4/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to find loop device: could not attach image file to loop device: no loop devices available
VERBOSE: Creating template passwd file and appending user data: /cm/shared/apps/singularity-ce/3.11.4/var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE: Default mount: /etc/passwd:/etc/passwd
VERBOSE: Checking for template group file: /cm/shared/apps/singularity-ce/3.11.4/var/singularity/mnt/session/rootfs/etc/group
VERBOSE: Creating group content
VERBOSE: Default mount: /etc/group:/etc/group
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
RaawwWWWWWRRRR!! Avocado!
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[63632,1],1]
  Exit code:    255
--------------------------------------------------------------------------

To Reproduce
Steps to reproduce the behavior: Run e.g.:

mpirun -n 4 singularity -v run shub://vsoch/hello-world

Expected behavior
I expect the container to run.

OS / Linux Distribution
Which Linux distribution are you using?

PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Additional context

Maybe related to the kernel version: 5.15.0-79-generic

[dtrudg]

Workaround here: #1499 (comment)

See also an Ubuntu bug related to this: https://bugs.launchpad.net/ubuntu/+source/linux-hwe-5.15/+bug/2013086

[dtrudg]

We should document this formally, and perhaps show an error that mentions max_loop as a pointer.

Note that the kernel patch that changes the behaviour only appears to affect Ubuntu, as it sets CONFIG_BLK_DEV_LOOP_MIN_COUNT=8 in its kernel config, while others set CONFIG_BLK_DEV_LOOP_MIN_COUNT=0.