Nesting apptainer instance start fails within podman-hpc

[asnaylor]

Version of Apptainer

apptainer version 1.3.0-1.el7

Expected behavior

Should be able to run apptainer instance within podman-hpc (a wrapper script around podman).

Actual behavior

Returns error and fails to start:

FATAL:   while executing starter: failed to create instance log files: user: unknown userid 75235

Steps to reproduce this behavior

user $ podman-hpc run --rm --cvmfs -it docker.io/cmssw/el7:x86_64
(root in container) $ export PATH=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin:$PATH
(root in container) $ apptainer -v instance start /cvmfs/unpacked.cern.ch/registry.hub.docker.com/fastml/triton-torchgeo:22.07-py3-geometric triton
VERBOSE: Effective capability CAP_SYS_ADMIN is missing, fallback to user namespace
VERBOSE: Not forwarding APPTAINER_VERBOSE environment variable
VERBOSE: Setting HOME=/root
VERBOSE: Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
FATAL:   while executing starter: failed to create instance log files: user: unknown userid 75235

Note

The image works when using apptainer shell within podman-hpc:

user $ podman-hpc run --rm --cvmfs -it docker.io/cmssw/el7:x86_64
(root in container) $ apptainer shell /cvmfs/unpacked.cern.ch/registry.hub.docker.com/fastml/triton-torchgeo:22.07-py3 geometric
WARNING: No current working directory set: skipping mount
WARNING: user: unknown userid 0
WARNING: user: unknown userid 0
Singularity> 

What OS/distro are you running

Outside container SUSE Linux Enterprise Server 15 SP4. The container is CentOS 7.

How did you install Apptainer

From cvmfs.

[DrDaveD]

This is related to #1066, perhaps it is even a duplicate.

Maybe with podman you need to bind in /etc/passwd or give some other option to have it define the running user id in /etc/passwd.

Interestingly #2164 was also first reported to me using the same containers and host operating system as this issue, but was reproducible in the easier to reproduce case I presented it as. It did instance start inside of another apptainer however, not inside of podman.