v1.3.1

v1.3.1 - [2024-04-24]

• Make 'apptainer build' work with signed Docker containers.
• Fixed regression introduced in 1.3.0 that prevented closing cryptsetup and the corresponding loop device after running an encrypted sif container file in suid mode.
• Stopped binding over the default timezone in the container with the host's timezone, which led to unexpected behavior if the application changed timezones.
• Added progress bars for oras:// push and pull.
• Hide Instance stats will not be available message under --sharens mode.
• Fix problem where credentials locally stored with registry login command were not usable in some execution flows. Run registry login again with latest version to ensure credentials are stored correctly.
• Make runscript timeout configurable.
• Return invalid bind path mount options during bind path parsing.
• Make the INFO message more helpful when a running background process at exit time causes a FUSE mount to not shut down cleanly.
• Fixed the wrong mediaType in the oras push manifest.

Apptainer 1.3.0

v1.3.0 - [2024-03-12]

Changes since v1.2.5

Security

This release fixes two moderate severity denial of service vulnerabilities by upgrading a dependent library: CVE-2024-28176 and CVE-2024-28180.

Changed defaults / behaviours

• FUSE mounts are now supported in setuid mode, enabling full functionality even when kernel filesystem mounts are insecure due to unprivileged users having write access to raw filesystems in containers.

  When allow setuid-mount extfs = no (the default) in apptainer.conf, then the fuse2fs image driver will be used to mount ext3 images in setuid mode instead of the kernel driver (ext3 images are primarily used for the --overlay feature), restoring functionality that was removed by default in Apptainer 1.1.8 because of the security risk.

  The allow setuid-mount squashfs configuration option in apptainer.conf now has a new default called iflimited which allows kernel squashfs mounts only if there is at least one limit container option set or if Execution Control Lists are activated in ecl.toml. If kernel squashfs mounts are are not allowed, then the squashfuse image driver will be used instead.
  iflimited is the default because if one of those limits are used the system administrator ensures that unprivileged users do not have write access to the containers, but on the other hand using FUSE would enable a user to theoretically bypass the limits via ptrace() because the FUSE process runs as that user.

  The fuse-overlayfs image driver will also now be tried in setuid mode if the kernel overlayfs driver does not work (for example if one of the layers is a FUSE filesystem).

  In addition, if allow setuid-mount encrypted = no then the unprivileged gocryptfs format will be used for encrypting SIF files instead of the kernel device-mapper. If a SIF file was encrypted using the gocryptfs format, it can now be mounted in setuid mode in addition to non-setuid mode.

• The four dependent FUSE programs for various reasons all now need to be compiled from source and included in Apptainer installations and packages.
  Scripts are provided to make this easy; see the updated instructions in INSTALL.md. The bundled squashfuse_ll is updated to version 0.5.1.

• Change the default in user namespace mode to use either kernel overlayfs or fuse-overlayfs instead of the underlay feature for the purpose of adding bind mount points. That was already the default in setuid mode; this change makes it consistent. The underlay feature can still be used with the --underlay option, but it is deprecated because the implementation is complicated and measurements have shown that the performance of underlay is similar to overlayfs and fuse-overlayfs.
  For now the underlay feature can be made the default again with a new preferred value on the enable underlay configuration option.
  Also the --underlay option can be used in setuid mode or as the root user, although it was ignored previously.

• Prefer again to use kernel overlayfs over fuse-overlayfs when a lower layer is FUSE and there's no writable upper layer, undoing the change from 1.2.0. Another workaround was found for the problem that change addressed. This applies in both setuid mode and in user namespace mode (except the latter not on CentOS7 where it isn't supported).

• --cwd is now the preferred form of the flag for setting the container's working directory, though --pwd is still supported for compatibility.

• When building RPM, we will now use /var/lib/apptainer (rather than /var/apptainer) to store local state files.

• The way --home is handled when running as root (e.g. sudo apptainer) or with --fakeroot has changed. Previously, we were only modifying the HOME environment variable in these cases, while leaving the container's /etc/passwd file unchanged (with its homedir field pointing to /root, regardless of the value passed to --home). With this change, both value of HOME and the contents of /etc/passwd in the container will reflect the value passed to --home if the container is readonly. If the container is writable, the /etc/passwd file is left alone because it can interfere with commands that want to modify it.

• The --vm and related flags to start apptainer inside a VM have been removed. This functionality was related to the retired Singularity Desktop / SyOS projects.

• The keyserver-related commands that were under remote have been moved to their own, dedicated keyserver command. Run apptainer help keyserver for more information.

• The commands related to OCI/Docker registries that were under remote have been moved to their own, dedicated registry command. Run apptainer help registry for more information.

• The the remote list subcommand now outputs only remote endpoints (with keyservers and OCI/Docker registries having been moved to separate commands), and the output has been streamlined.

• Adding a new remote endpoint using the apptainer remote add command will now set the new endpoint as default. This behavior can be suppressed by supplying the --no-default (or -n) flag to remote add.

• Skip parsing build definition file template variables after comments beginning with a hash symbol.

• Improved the clarity of apptainer key list output.

• The global /tmp directory is no longer used for gocryptfs mountpoints.

• Updated minimum go version to 1.20

New Features & Functionality

• The remote status command will now print the username, realname, and email of the logged-in user, if available.
• Add monitoring feature support, which requires the usage of an additional tool named apptheus, this tool will put apptainer starter into a newly created cgroup and collect system metrics.
• A new --no-pid flag for apptainer run/shell/exec disables the PID namespace inferred by --containall and --compat.
• Added --config option tokeyserver commands.
• Honor an optional remoteName argument to the keyserver list command.
• Added the APPTAINER_ENCRYPTION_PEM_DATA env var to allow for encrypting and running encrypted containers without a PEM file.
• Adding --sharens mode for apptainer exec/run/shell, which enables to run multiple apptainer instances created by the same parent using the same image in the same user namespace.

Developer / API

• Changes in pkg/build/types.Definition struct. New .FullRaw field introduced, which always contains the raw data for the entire definition file. Behavior of .Raw field has changed: for multi-stage builds parsed with pkg/build/types/parser.All(), .Raw contains the raw content of a single build stage. Otherwise, it is equal to .FullRaw.

Bug fixes

• Don't bind /var/tmp on top of /tmp in the container, where /var/tmp resolves to same location as /tmp.
• Support parentheses in test / [ commands in container startup scripts, via dependency update of mvdan.cc/sh.
• Fix regression introduced in v1.2.0 that led to an empty user's shell field in the /etc/passwd file.
• Prevent container builds from failing when $HOME points to a non-readable directory.
• Fix the use of nvidia-container-cli on Ubuntu 22.04 where an ldconfig wrapper script gets in the way. Instead, we use ldconfig.real directly.
• Run image drivers with CAP_DAC_OVERRIDE in user namespace mode. This fixes --nvccli with NVIDIA_DRIVER_CAPABILITIES=graphics, which previously failed when using fuse-overlayfs.

Release change

• Releases will generate apptainer Docker images for the Linux amd64 and arm64 architectures at ghcr.io/apptainer/apptainer.

Apptainer 1.3.0 Release Candidate 2

v1.3.0-rc.2 - [2024-02-15]

Changes since v1.3.0-rc.1

• Change the default in user namespace mode to use either kernel overlayfs or fuse-overlayfs instead of the underlay feature for the purpose of adding bind mount points. That was already the default in setuid mode; this change makes it consistent. The underlay feature can still be used with the --underlay option, but it is deprecated because the implementation is complicated and measurements have shown that the performance of underlay is similar to overlayfs and fuse-overlayfs. For now the underlay feature can be made the default again with a new preferred value on the enable underlay configuration option. Also the --underlay option can be used in setuid mode or as the root user, although it was ignored previously.
• Prefer again to use kernel overlayfs over fuse-overlayfs when a lower layer is FUSE and there's no writable upper layer, undoing the change from 1.2.0. Another workaround was found for the problem that change addressed. This applies in both setuid mode and in user namespace mode (except the latter not on CentOS7 where it isn't supported).
• Fix the use of an overlay ext3 filesystem in SIF files.
• Fix --sharens failure on EL8.
• Fix Harbor registry login failure.
• Prevent container builds from failing when $HOME points to a non-readable directory.

Apptainer 1.3.0 Release Candidate 1

v1.3.0-rc.1 - [2024-01-10]

Changes since v1.2.5

Changed defaults / behaviours

• FUSE mounts are now supported in setuid mode, enabling full functionality even when kernel filesystem mounts are insecure due to unprivileged users having write access to raw filesystems in containers.

  When allow setuid-mount extfs = no (the default) in apptainer.conf, then the fuse2fs image driver will be used to mount ext3 images in setuid mode instead of the kernel driver (ext3 images are primarily used for the --overlay feature), restoring functionality that was removed by default in Apptainer 1.1.8 because of the security risk.

  The allow setuid-mount squashfs configuration option in apptainer.conf now has a new default called iflimited which allows kernel squashfs mounts only if there is at least one limit container option set or if Execution Control Lists are activated in ecl.toml. If kernel squashfs mounts are are not allowed, then the squashfuse image driver will be used instead. iflimited is the default because if one of those limits are used the system administrator ensures that unprivileged users do not have write access to the containers, but on the other hand using FUSE would enable a user to theoretically bypass the limits via ptrace() because the FUSE process runs as that user.

  The fuse-overlayfs image driver will also now be tried in setuid mode if the kernel overlayfs driver does not work (for example if one of the layers is a FUSE filesystem).

  In addition, if allow setuid-mount encrypted = no then the unprivileged gocryptfs format will be used for encrypting SIF files instead of the kernel device-mapper. If a SIF file was encrypted using the gocryptfs format, it can now be mounted in setuid mode in addition to non-setuid mode.

• The four dependent FUSE programs for various reasons all now need to be compiled from source and included in Apptainer installations and packages. Scripts are provided to make this easy; see the updated instructions in INSTALL.md.

• --cwd is now the preferred form of the flag for setting the container's working directory, though --pwd is still supported for compatibility.

• When building RPM, we will now use /var/lib/apptainer (rather than /var/apptainer) to store local state files.

• The way --home is handled when running as root (e.g. sudo apptainer) or with --fakeroot has changed. Previously, we were only modifying the HOME environment variable in these cases, while leaving the container's /etc/passwd file unchanged (with its homedir field pointing to /root, regardless of the value passed to --home). With this change, both value of HOME and the contents of /etc/passwd in the container will reflect the value passed to --home if the container is readonly. If the container is writable, the /etc/passwd file is left alone because it can interfere with commands that want to modify it.

• The --vm and related flags to start apptainer inside a VM have been removed. This functionality was related to the retired Singularity Desktop / SyOS projects.

• The keyserver-related commands that were under remote have been moved to their own, dedicated keyserver command. Run apptainer help keyserver for more information.

• The commands related to OCI/Docker registries that were under remote have been moved to their own, dedicated registry command. Run apptainer help registry for more information.

• The the remote list subcommand now outputs only remote endpoints (with keyservers and OCI/Docker registries having been moved to separate commands), and the output has been streamlined.

• Adding a new remote endpoint using the apptainer remote add command will now set the new endpoint as default. This behavior can be suppressed by supplying the --no-default (or -n) flag to remote add.

• Skip parsing build definition file template variables after comments beginning with a hash symbol.

• Improved the clarity of apptainer key list output.

• The global /tmp directory is no longer used for gocryptfs mountpoints.

• Updated minimum go version to 1.20

New Features & Functionality

• The remote status command will now print the username, realname, and email of the logged-in user, if available.
• Add monitoring feature support, which requires the usage of an additional tool named apptheus, this tool will put apptainer starter into a newly created cgroup and collect system metrics.
• A new --no-pid flag for apptainer run/shell/exec disables the PID namespace inferred by --containall and --compat.
• Added --config option tokeyserver commands.
• Honor an optional remoteName argument to the keyserver list command.
• Added the APPTAINER_ENCRYPTION_PEM_DATA env var to allow for encrypting and running encrypted containers without a PEM file.
• Adding --sharens mode for apptainer exec/run/shell, which enables to run multiple apptainer instances created by the same parent using the same image in the same user namespace.

Developer / API

• Changes in pkg/build/types.Definition struct. New .FullRaw field introduced, which always contains the raw data for the entire definition file. Behavior of .Raw field has changed: for multi-stage builds parsed with pkg/build/types/parser.All(), .Raw contains the raw content of a single build stage. Otherwise, it is equal to .FullRaw.

Bug fixes

• Don't bind /var/tmp on top of /tmp in the container, where /var/tmp resolves to same location as /tmp.
• Support parentheses in test / [ commands in container startup scripts, via dependency update of mvdan.cc/sh.
• Fix regression introduced in v1.2.0 that led to an empty user's shell field in the /etc/passwd file.

Release change

• Releases will generate apptainer Docker images for the Linux amd64 and arm64 architectures.

Apptainer 1.2.5

v1.2.5 - [2023-11-21]

• Added libnvidia-nvvm to nvliblist.conf. Newer NVIDIA Drivers (known with >= 525.85.05) require this lib to compile OpenCL programs against NVIDIA GPUs, i.e. libnvidia-opencl depends on libnvidia-nvvm.
• Disable the usage of cgroup in instance creation when --fakeroot is passed.
• Disable the usage of cgroup in instance creation when hidepid mount option on /proc is set.

Apptainer 1.2.4

v1.2.4 - [2023-10-10]

• Fixed a problem with relocating an unprivileged installation of apptainer on el8 and a mounted remote filesystem when using the --fakeroot option without /etc/subuid mapping. The fix was to change the switch to an unprivileged root-mapped namespace to be the equivalent of unshare -r instead of unshare -rm on action commands, to work around a bug in the el8 kernel.
• Fixed a regression introduced in 1.2.0 where the user's password file information was not copied in to the container when there was a parent root-mapped user namespace (as is the case for example in cvmfsexec).
• Added the upcoming NVIDIA driver library libnvidia-gpucomp.so to the list of libraries to add to NVIDIA GPU-enabled containers.
• Fixed missing error handling during the creation of an encrypted image that lead to the generation of corrupted images.
• Use APPTAINER_TMPDIR for temporary files during privileged image encryption.
• If rootless unified cgroups v2 is available when starting an image but XDG_RUNTIME_DIR or DBUS_SESSION_BUS_ADDRESS is not set, print an info message that stats will not be available instead of exiting with a fatal error.
• Allow templated build arguments to definition files to have empty values.

Apptainer 1.2.3

v1.2.3 - [2023-09-14]

• The apptainer push/pull commands now show a progress bar for the oras protocol like there was for docker and library protocols.
• The --nv and --rocm flags can now be used simultaneously.
• Fix the use of APPTAINER_CONFIGDIR with apptainer instance start and action commands that refer to instance://.
• Ignore undefined macros, to fix yum bootstrap agent on el7.
• Fix the issue that apptainer would not read credentials from the Docker fallback path ~/.docker/config.json if missing in the apptainer credentials.

Apptainer 1.2.2

v1.2.2 - [2023-07-27]

• Fix $APPTAINER_MESSAGELEVEL to correctly set the logging level.
• Fix build failures when in setuid mode and unprivileged user namespaces are unavailable and the --fakeroot option is not selected.

Apptainer 1.2.1

v1.2.1 - [2023-07-24]

Security fix

• Included a fix for moderate severity security advisory GHSA-mmx5-32m4-wxvx which describes an ineffective privilege drop when requesting a container network with a setuid installation of Apptainer. The vulnerability allows an attacker to delete any directory on the host filesystems with a crafted starter config. Only affects v1.2.0-rc.2 and v1.2.0.

Apptainer 1.2.0

v1.2.0 - [2023-07-18]

Changed defaults / behaviours

• Create the current working directory in a container when it doesn't exist. This restores behavior as it was before singularity 3.6.0. As a result, using --no-mount home won't have any effect when running apptainer from a home directory and will require --no-mount home,cwd to avoid mounting that directory.
• Handle current working directory paths containing symlinks both on the host and in a container but pointing to different destinations. If detected, the current working directory is not mounted when the destination directory in the container exists.
• Destination mount points are now sorted by shortest path first to ensure that a user bind doesn't override a previous bind path when set in arbitrary order on the CLI. This is also applied to image binds.
• When the kernel supports unprivileged overlayfs mounts in a user namespace, the container will be constructed by default using an overlay instead of an underlay layout for bind mounts. A new --underlay action option can be used to prefer underlay instead of overlay.
• Use fuse-overlayfs instead of the kernel overlayfs when a lower dir is a FUSE filesystem, even when the overlay layer is not writable. That always used to be done when the overlay layer was writable, but this fixes a problem seen when squashfuse (which is read-only) was used for the overlay layer.
• Fix the enable overlay = driver configuration option to always use the overlay image driver (that is, fuse-overlayfs) even when the kernel overlayfs is usable.
• Overlay is blocked on the panfs filesystem, allowing sandbox directories to be run from panfs without error.
• sessiondir maxsize in apptainer.conf now defaults to 64 MiB for new installations. This is an increase from 16 MiB in prior versions.
• The apptainer cache is now architecture aware, so the same home directory cache can be shared by machines with different architectures.
• Show standard output of yum bootstrap if log level is verbose or higher while building a container.
• Lookup and store user/group information in stage one prior to entering any namespaces, to fix an issue with winbind not correctly looking up user/group information when using user namespaces.
• A new --reproducible flag for ./mconfig will configure Apptainer so that its binaries do not contain non-reproducible paths. This disables plugin functionality.

New features / functionalities

• Support for unprivileged encryption of SIF files using gocryptfs. The gocryptfs command is included in rpm and debian packaging. This is not compatible with privileged encryption, so containers encrypted by root need to be rebuilt by an unprivileged user.
• Templating support for definition files. Users can now define variables in definition files via a matching pair of double curly brackets. Variables of the form {{ variable }} will be replaced by a value defined either by a variable=value entry in the %arguments section of the definition file or through new build options --build-arg or --build-arg-file. By default any unused variables given in --build-arg or --build-arg-file result in a fatal error but the option --warn-unused-build-args changes that to a warning rather than a fatal error.
• Add a new instance run command that will execute the runscript when an instance is initiated instead of executing the startscript.
• The sign and verify commands now support signing and verification with non-PGP key material by specifying the path to a private key via the --key flag.
• The verify command now supports verification with X.509 certificates by specifying the path to a certificate via the --certificate flag. By default, the system root certificate pool is used as trust anchors unless overridden via the --certificate-roots flag. A pool of intermediate certificates that are not trust anchors, but can be used to form a certificate chain, can also be specified via the --certificate-intermediates flag.
• Support for online verification checks of X.509 certificates using OCSP protocol via the new verify --ocsp-verify option.
• The instance stats command displays the resource usage every second. The --no-stream option disables this interactive mode and shows the point-in-time usage.
• Instances are now started in a cgroup by default, when run as root or when unified cgroups v2 with systemd as manager is configured. This allows apptainer instance stats to be supported by default when possible.
• The instance start command now accepts an optional --app <name> argument which invokes a start script within the %appstart <name> section in the definition file. The instance stop command still only requires the instance name.
• The instance name is now available inside an instance via the new APPTAINER_INSTANCE environment variable.
• Add ability to set a custom config directory via the new APPTAINER_CONFIGDIR environment variable.
• Add ability to change log level through environment variables, APPTAINER_SILENT, APPTAINER_QUIET, and APPTAINER_VERBOSE. Also add APPTAINER_NOCOLOR for the --nocolor option.
• Add discussion of using TMPDIR or APPTAINER_TMPDIR in the build help.
• The --no-mount flag now accepts the value bind-paths to disable mounting of all bind path entries in apptainer.conf.
• Support for DOCKER_HOST parsing when using docker-daemon://
• DOCKER_USERNAME and DOCKER_PASSWORD supported without APPTAINER_ prefix.
• Add new Linux capabilities CAP_PERFMON, CAP_BPF, and CAP_CHECKPOINT_RESTORE.
• Add setopt definition file header for the yum bootstrap agent. The setopt value is passed to yum / dnf using the --setopt flag. This permits setting e.g. install_weak_deps=False to bootstrap recent versions of Fedora, where systemd (a weak dependency) cannot install correctly in the container. See examples/Fedora for an example definition file.
• Warn user that a yum bootstrap of an older distro may fail if the host rpm _db_backend is not bdb.
• The remote get-login-password command allows users to retrieve a remote's token. This enables piping the secret directly into docker login while preventing it from showing up in a shell's history.
• Define EUID in %environment alongside UID.
• In --rocm mode, the whole of /dev/dri is now bound into the container when --contain is in use. This makes /dev/dri/render devices available, required for later ROCm versions.

Other changes

• Update minimum go version to 1.19.
• Upgrade squashfuse_ll to version 0.2.0, removing the need for applying patches during compilation. The new version includes a fix to prevent it from triggering 'No data available errors' on overlays of SIF files that were built on machines with SELinux enabled.
• Fix non-root instance join with unprivileged systemd-managed cgroups v2, when join is from outside a user-owned cgroup.
• Fix joining cgroup of instance started as root, with cgroups v1, non-default cgroupfs manager, and no device rules.
• Avoid UID / GID / EUID readonly var warnings with --env-file.
• Ensure consistent binding of libraries under --nv/--rocm when duplicate <library>.so[.version] files are listed by ldconfig -p.
• Ensure DOCKER_HOST is honored in non-build flows.
• Corrected apptainer.conf comment, to refer to correct file as source of default capabilities when root default capabilities = file.
• Fix memory usage calculation during apptainer compilation on RaspberryPi.
• Fix misleading error when an overlay is requested by the root user while the overlay kernel module is not loaded.
• Fix interaction between --workdir and --scratch options when the former is given a relative path.
• Remove the warning about a missing signature when building an image based on a local unsigned SIF file.
• Set real UID to zero when escalating privileges for CNI plugins, to fix issue appeared with RHEL 9.X.
• Fix seccomp filters to allow mknod/mknodat syscalls to create pipe/socket and character devices with device number 0 for fakeroot builds.
• Add 32-bit compatibility mode for 64-bit architectures in the fakeroot seccomp filter.