kinit segfaults with apptainer 1.3.0

[matyasselmeci]

Version of Apptainer

What version of Apptainer (or Singularity) are you using? Run

apptainer --version (or singularity --version).

1.3.0-1.el9

Expected behavior

I have the following def file to create a minimal reproducer, named sf.def:

Bootstrap: docker
From: almalinux:9

%post
    dnf -y install krb5-workstation

%environment
    export LANG=C
    export LC_ALL=C
    export KRB5CCNAME=FILE:/dev/shm/krb5cc_file_$(id -u)

%runscript
    kinit username@CS.WISC.EDU

build with apptainer build sf.sif sf.def
run with apptainer run sf.sif

expected: a password prompt from CS.WISC.EDU

Actual behavior

/.singularity.d/runscript: line 3: 11713 Segmentation fault      (core dumped) kinit username@CS.WISC.EDU

I downgraded to apptainer 1.2.5 and the problem went away. Note that I also don't see a problem if I use almalinux:8 instead of almalinux:9 as the base image.

What OS/distro are you running

$ cat /etc/os-release
NAME="AlmaLinux"
VERSION="9.3 (Shamrock Pampas Cat)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.3 (Shamrock Pampas Cat)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.3"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"

How did you install Apptainer

I installed the RPMs apptainer and apptainer-suid from EPEL. Here is my /etc/apptainer/apptainer.conf (without comments):

allow setuid = yes
max loop devices = 256
allow pid ns = yes
config passwd = yes
config group = yes
config resolv_conf = yes
mount proc = yes
mount sys = yes
mount dev = yes
mount devpts = yes
mount home = yes
mount tmp = yes
mount hostfs = no
bind path = /etc/localtime
bind path = /etc/hosts
user bind control = yes
enable fusemount = yes
enable overlay = yes
enable underlay = yes
mount slave = yes
sessiondir max size = 64
allow container sif = yes
allow container encrypted = yes
allow container squashfs = yes
allow container extfs = yes
allow container dir = yes
always use nv = no
use nvidia-container-cli = no
always use rocm = no
root default capabilities = full
memory fs type = tmpfs
mksquashfs procs = 0
shared loop devices = no
image driver =
download concurrency = 3
download part size = 5242880
download buffer size = 32768
systemd cgroups = yes
apptheus socket path = /run/apptheus/gateway.sock
allow monitoring = no

[DrDaveD]

I can reproduce the problem both in suid mode and user namespace mode on an al9 machine.

Interestingly, I tested it first with a configuration that had allow setuid-mount squashfs = yes and it did not happen then. Thank you for including your apptainer.conf settings; even though they are the defaults, it enabled me to quickly identify my non-default setting.

[DrDaveD]

The problem also happens with apptainer-1.2.5 in user namespace mode. The default in suid mode then was to use the kernel squashfs so that's what's different in 1.3.0.

[DrDaveD]

I found with gdb that it is crashing after attempting to list files in /usr/lib64/krb5/plugins/libkrb5 and finding it empty. So a workaround is to install the sssd-client package which installs a file there.

It crashes at this line of code because fnames is null. The k5_dir_filenames function behaves that way when a directory is empty, so the code should also be checking for fnames != NULL but it doesn't. I don't know why it doesn't crash in other scenarios. They are just lucky, I guess, unless there are some scenarios that explicitly place a zero at address zero.

So this is a bug in krb5, not apptainer. Please submit a bug report to Alma, Red Hat, or krb5.

[DrDaveD]

No it wasn't just luck. With the kernel squashfs mount, k5_dir_filenames returns 2 fnames . and .., and with squashfuse it returns none. I'm still looking.

[DrDaveD]

Bug reported in vasi/squashfuse#133.