buildah unshare cannot create user namespace but unshare can

[Syquel]

I am trying to run buildah within a podman container within WSL to simulate a capability-restricted environment.
In my scenario the command buildah unshare is not able to create a new user namespace to mount a container, but unshare is.

WSL Environment

# uname -a
Linux f6fc426fc67a 5.15.90.4-microsoft-standard-WSL2 #1 SMP Tue Jul 18 21:28:32 UTC 2023 x86_64 GNU/Linux

# podman --version
podman version 4.6.2

Container Environment

The podman container is started via:

# podman run --rm -it --cap-drop ALL --cap-add CAP_SETFCAP,CAP_SETUID,CAP_SETGID,CAP_DAC_OVERRIDE --security-opt no-new-privileges --device /dev/fuse registry.fedoraproject.org/fedora-minimal:39 /bin/bash

After installing buildah and fuse-overlayfs in the container:

# buildah --version
buildah version 1.32.0 (image-spec 1.1.0-rc.4, runtime-spec 1.1.0)

# fuse-overlayfs --version
fuse-overlayfs: version 1.12
FUSE library version 3.16.1
using FUSE kernel interface version 7.38
fusermount3 version: 3.16.1

Running Buildah

The new container is created via:

# buildah from scratch
working-container

Without User Namespace

Trying to mount the container fails as expected without a new user namespace:

# buildah mount --log-level debug --storage-opt overlay.mount_program=/usr/bin/fuse-overlayfs working-container
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
WARN[0000] failed to shutdown storage: "mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted"

Buildah Unshare

Trying to create a new user namespace via buildah unshare fails unexpectedly:

# buildah --log-level debug unshare --mount working-container
DEBU[0000] effective capabilities: [audit_control=false audit_read=false audit_write=false block_suspend=false bpf=false checkpoint_restore=false chown=false dac_override=false dac_read_search=false fowner=false fsetid=false ipc_lock=false ipc_owner=false kill=false lease=false linux_immutable=false mac_admin=false mac_override=false mknod=false net_admin=false net_bind_service=false net_broadcast=false net_raw=false perfmon=false setfcap=true setgid=true setpcap=false setuid=true sys_admin=false sys_boot=false sys_chroot=true sys_module=false sys_nice=false sys_pacct=false sys_ptrace=false sys_rawio=false sys_resource=false sys_time=false sys_tty_config=false syslog=false wake_alarm=false]
DEBU[0000] Running [buildah-in-a-user-namespace --log-level debug unshare --mount working-container] with environment [HOSTNAME=f6fc426fc67a DISTTAG=f39container PWD=/build container=oci HOME=/root FGC=f39 TERM=xterm SHLVL=1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin _=/usr/bin/buildah TMPDIR=/var/tmp _CONTAINERS_USERNS_CONFIGURED=1 BUILDAH_ISOLATION=rootless], UID map [{ContainerID:0 HostID:0 Size:4294967295}], and GID map [{ContainerID:0 HostID:0 Size:4294967295}]
DEBU[0000] effective capabilities: [audit_control=true audit_read=true audit_write=true block_suspend=true bpf=true checkpoint_restore=true chown=true dac_override=true dac_read_search=true fowner=true fsetid=true ipc_lock=true ipc_owner=true kill=true lease=true linux_immutable=true mac_admin=true mac_override=true mknod=true net_admin=true net_bind_service=true net_broadcast=true net_raw=true perfmon=true setfcap=true setgid=true setpcap=true setuid=true sys_admin=true sys_boot=true sys_chroot=true sys_module=true sys_nice=true sys_pacct=true sys_ptrace=true sys_rawio=true sys_resource=true sys_time=true sys_tty_config=true syslog=true wake_alarm=true]
DEBU[0000] Running [buildah-in-a-user-namespace-in-a-user-namespace --log-level debug unshare --mount working-container] with environment [HOSTNAME=f6fc426fc67a DISTTAG=f39container PWD=/build container=oci HOME=/root FGC=f39 TERM=xterm SHLVL=1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin _=/usr/bin/buildah TMPDIR=/var/tmp BUILDAH_ISOLATION=rootless _CONTAINERS_USERNS_CONFIGURED=1 _CONTAINERS_ROOTLESS_UID=0 _CONTAINERS_ROOTLESS_GID=0], UID map [{ContainerID:0 HostID:0 Size:1} {ContainerID:1 HostID:100000 Size:655361}], and GID map [{ContainerID:0 HostID:0 Size:1} {ContainerID:1 HostID:100000 Size:655361}]
DEBU[0000] effective capabilities: [audit_control=true audit_read=true audit_write=true block_suspend=true bpf=true checkpoint_restore=true chown=true dac_override=true dac_read_search=true fowner=true fsetid=true ipc_lock=true ipc_owner=true kill=true lease=true linux_immutable=true mac_admin=true mac_override=true mknod=true net_admin=true net_bind_service=true net_broadcast=true net_raw=true perfmon=true setfcap=true setgid=true setpcap=true setuid=true sys_admin=true sys_boot=true sys_chroot=true sys_module=true sys_nice=true sys_pacct=true sys_ptrace=true sys_rawio=true sys_resource=true sys_time=true sys_tty_config=true syslog=true wake_alarm=true]
DEBU[0000] Running [buildah-in-a-user-namespace-in-a-user-namespace-in-a-user-namespace --log-level debug unshare --mount working-container] with environment [HOSTNAME=f6fc426fc67a DISTTAG=f39container PWD=/build container=oci HOME=/root FGC=f39 TERM=xterm SHLVL=1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin _=/usr/bin/buildah TMPDIR=/var/tmp BUILDAH_ISOLATION=rootless _CONTAINERS_USERNS_CONFIGURED=1 _CONTAINERS_ROOTLESS_UID=0 _CONTAINERS_ROOTLESS_GID=0], UID map [{ContainerID:0 HostID:0 Size:1} {ContainerID:1 HostID:100000 Size:655361}], and GID map [{ContainerID:0 HostID:0 Size:1} {ContainerID:1 HostID:100000 Size:655361}]
Error: writing "0 0 1\n1 100000 655361\n" to /proc/165/gid_map: write /proc/165/gid_map: operation not permitted
ERRO[0000] writing "0 0 1\n1 100000 655361\n" to /proc/165/gid_map: write /proc/165/gid_map: operation not permitted
ERRO[0000] (Unable to determine exit status)
DEBU[0000] exit status 1
DEBU[0000] exit status 1

Unshare

But creating a new user namespace via unshare works:

# unshare --user --map-auto --map-root-user --mount buildah mount --log-level debug --storage-opt overlay.mount_program=/usr/bin/fuse-overlayfs working-container
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] backingFs=overlayfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/b8fe927025d03a91347333e2229a980e549a5666081acaf21f70017b9122a53f/empty,upperdir=/var/lib/containers/storage/overlay/b8fe927025d03a91347333e2229a980e549a5666081acaf21f70017b9122a53f/diff,workdir=/var/lib/containers/storage/overlay/b8fe927025d03a91347333e2229a980e549a5666081acaf21f70017b9122a53f/work,,volatile
/var/lib/containers/storage/overlay/b8fe927025d03a91347333e2229a980e549a5666081acaf21f70017b9122a53f/merged
DEBU[0000] shutting down the store

GID / UID Mappings

The following subuids / subgids are defined:

# cat /etc/subuid /etc/subgid
root:100000:655361
root:100000:655361

unshare creates the following uid / gid mappings:

# unshare --user --map-auto --map-root-user --mount cat /proc/self/uid_map /proc/self/gid_map
         0          0          1
         1     100000     655360
         0          0          1
         1     100000     655360

And according to the buildah unshare error message it tries to create the gid mappings 0 0 1 and 1 100000 655361:

# buildah unshare --mount working-container
Error: writing "0 0 1\n1 100000 655361\n" to /proc/263/gid_map: write /proc/263/gid_map: operation not permitted

If I interpret that correctly buildah unshare has an off-by-one error and the writing to the gid_map is rejected because it tries to map more gids than there are subgids?

Question

My question is if I misunderstand / misuse buildah unshare, if this is intended behavior or if this is a bug.
My expectation has been that the unshare command above would have the same result as buildah unshare.

[nalind]

A binary being exec'd with an argv[0] of "buildah-in-a-user-namespace-in-a-user-namespace-in-a-user-namespace" suggests that buildah tried to run a copy of itself in a new user namespace with mappings applied, but the child process didn't have the privileges that its parent expected it to be given, so it tried again (and... again). Of course for the first child process, a number of the IDs that it would map into the new user namespace for its child aren't valid, because anything that wasn't mapped into the child's namespace is invalid.
The first process was started without cap_sys_admin, but as UID 0, so it attempted to re-exec itself in a new namespace with the same set of UIDs and GIDs, which requires cap_setfcap, which it had.
The child process had the CAP_SYS_ADMIN capability, and its parent ensured that it was started as UID 0 in its namespace, so it shouldn't be trying to set things up again. Was the buildah binary built without cgo? That could cause a malfunction in this area.

[Syquel]

It's the official Fedora 39 / 38 buildah package and according to the build spec it has been built with cgo.

Still not sure if this is a bug.

[nalind]

Wait, are you already running as root? If so, how did the various capabilities get dropped?