Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to initialize container #687

Closed
ptomato opened this issue Jan 30, 2021 · 31 comments
Closed

Failed to initialize container #687

ptomato opened this issue Jan 30, 2021 · 31 comments
Labels
1. Bug Something isn't working 2. Container Realm The issue is related to what happens inside of a toolbox container

Comments

@ptomato
Copy link

ptomato commented Jan 30, 2021

Describe the bug
Toolbox fails to start after an upgrade and a reboot. This seems to be a different error than #669, and I did not do any unusual updates as described there, so I'm creating a different issue.

Steps how to reproduce the behaviour

  1. toolbox enter

Expected behaviour
Toolbox environment should start as normal.

Actual behaviour

$ toolbox enter -v --log-level debug
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user ptomato 
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Toolbox config directory is /var/home/ptomato/.config/toolbox 
DEBU Current Podman version is 2.2.1              
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Old Podman version is 2.2.1                  
DEBU Migration not needed: Podman version 2.2.1 is unchanged 
DEBU Resolving container and image names          
DEBU Container: ''                                
DEBU Distribution: ''                             
DEBU Image: ''                                    
DEBU Release: ''                                  
DEBU Resolved container and image names           
DEBU Container: 'fedora-toolbox-33'               
DEBU Image: 'fedora-toolbox:33'                   
DEBU Release: '33'                                
DEBU Checking if container fedora-toolbox-33 exists 
DEBU Inspecting mounts of container fedora-toolbox-33 
DEBU Requires org.freedesktop.Flatpak.SessionHelper 
DEBU Calling org.freedesktop.Flatpak.SessionHelper.RequestSession 
DEBU Starting container fedora-toolbox-33         
DEBU Inspecting entry point of container fedora-toolbox-33 
DEBU Entry point PID is a float64                 
DEBU Entry point of container fedora-toolbox-33 is toolbox (PID=12307) 
DEBU Waiting for container fedora-toolbox-33 to finish initializing 
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Checking if initialization stamp /run/user/1000/toolbox/container-initialized-12307 exists 
Error: failed to initialize container fedora-toolbox-33

Screenshots
N/A

Output of toolbox --version (v0.0.90+)
toolbox version 0.0.99

Toolbox package info (rpm -q toolbox)
toolbox-0.0.99-1.fc33.x86_64

Output of podman version

Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.5
Built:        Tue Dec  8 06:37:50 2020
OS/Arch:      linux/amd64

Podman package info (rpm -q podman)
podman-2.2.1-1.fc33.x86_64

Info about your OS
Fedora Silverblue 33, freshly updated as of the time of writing

Additional context
This happened after a updating the Silverblue image to the latest, and rebooting. The toolbox image is Fedora 33.

$ toolbox list -i
IMAGE ID      IMAGE NAME                                        CREATED
fe7b8c2393f9  registry.fedoraproject.org/f32/fedora-toolbox:32  9 months ago
95dc9d4b0f27  registry.fedoraproject.org/f33/fedora-toolbox:33  4 months ago

Here's a list of packages upgraded (note, it's the list I got from rpm-ostree rollback, so you have to read it in reverse):

Downgraded:
  ModemManager 1.14.10-1.fc33 -> 1.14.6-1.fc33
  ModemManager-glib 1.14.10-1.fc33 -> 1.14.6-1.fc33
  PackageKit-glib 1.2.2-2.fc33 -> 1.2.1-1.fc33
  abattis-cantarell-fonts 0.301-1.fc33 -> 0.201-4.fc33
  adwaita-qt5 1.2.0-1.fc33 -> 1.1.3-4.fc33
  alsa-plugins-pulseaudio 1.2.2-4.fc33 -> 1.2.2-3.fc33
  alsa-sof-firmware 1.6.1-1.fc33 -> 1.6-1.fc33
  atkmm 2.28.1-1.fc33 -> 2.24.3-6.fc33
  audit 3.0-1.fc33 -> 3.0-0.21.20191104git1c2f876.fc33
  audit-libs 3.0-1.fc33 -> 3.0-0.21.20191104git1c2f876.fc33
  authselect 1.2.2-1.fc33 -> 1.2.1-4.fc33
  authselect-libs 1.2.2-1.fc33 -> 1.2.1-4.fc33
  bind-libs 32:9.11.27-1.fc33 -> 32:9.11.24-2.fc33
  bind-libs-lite 32:9.11.27-1.fc33 -> 32:9.11.24-2.fc33
  bind-license 32:9.11.27-1.fc33 -> 32:9.11.24-2.fc33
  bind-utils 32:9.11.27-1.fc33 -> 32:9.11.24-2.fc33
  binutils 2.35-18.fc33 -> 2.35-14.fc33
  binutils-gold 2.35-18.fc33 -> 2.35-14.fc33
  bolt 0.9.1-1.fc33 -> 0.9-3.fc33
  btrfs-progs 5.10-1.fc33 -> 5.9-1.fc33
  buildah 1.18.0-1.fc33 -> 1.16.2-1.fc33
  c-ares 1.17.0-1.fc33 -> 1.16.1-3.fc33
  chromium 88.0.4324.96-1.fc33 -> 86.0.4240.183-1.fc33
  chromium-common 88.0.4324.96-1.fc33 -> 86.0.4240.183-1.fc33
  cifs-utils 6.11-2.fc33 -> 6.11-1.fc33
  cldr-emoji-annotation 1:38-1.1.fc33 -> 1:38-1.fc33
  cldr-emoji-annotation-dtd 1:38-1.1.fc33 -> 1:38-1.fc33
  containernetworking-plugins 0.9.0-1.fc33 -> 0.8.7-1.fc33
  containers-common 1:1.2.0-13.fc33 -> 1:1.2.0-3.fc33
  coreutils 8.32-15.fc33 -> 8.32-12.fc33
  coreutils-common 8.32-15.fc33 -> 8.32-12.fc33
  crun 0.17-1.fc33 -> 0.15.1-1.fc33
  cups 1:2.3.3op1-1.fc33 -> 1:2.3.3-18.fc33
  cups-client 1:2.3.3op1-1.fc33 -> 1:2.3.3-18.fc33
  cups-filesystem 1:2.3.3op1-1.fc33 -> 1:2.3.3-18.fc33
  cups-filters 1.28.6-1.fc33 -> 1.28.5-1.fc33
  cups-filters-libs 1.28.6-1.fc33 -> 1.28.5-1.fc33
  cups-ipptool 1:2.3.3op1-1.fc33 -> 1:2.3.3-18.fc33
  cups-libs 1:2.3.3op1-1.fc33 -> 1:2.3.3-18.fc33
  curl 7.71.1-8.fc33 -> 7.71.1-7.fc33
  dbus-broker 26-1.fc33 -> 24-1.fc33
  dnsmasq 2.83-1.fc33 -> 2.82-3.fc33
  enchant2 2.2.14-1.fc33 -> 2.2.13-1.fc33
  ethtool 2:5.10-1.fc33 -> 2:5.9-1.fc33
  evolution-data-server 3.38.3-1.fc33 -> 3.38.1-1.fc33
  evolution-data-server-langpacks 3.38.3-1.fc33 -> 3.38.1-1.fc33
  f33-backgrounds-base 33.0.8-1.fc33 -> 33.0.7-1.fc33
  f33-backgrounds-gnome 33.0.8-1.fc33 -> 33.0.7-1.fc33
  fedora-chromium-config 1.1-7.fc33 -> 1.1-6.fc33
  fedora-release-common 33-3 -> 33-2
  fedora-release-identity-silverblue 33-3 -> 33-2
  fedora-release-silverblue 33-3 -> 33-2
  ffmpeg-libs 4.3.1-16.fc33 -> 4.3.1-11.fc33
  firefox 85.0-5.fc33 -> 83.0-3.fc33
  firewalld 0.8.6-1.fc33 -> 0.8.4-1.fc33
  firewalld-filesystem 0.8.6-1.fc33 -> 0.8.4-1.fc33
  flac-libs 1.3.3-4.fc33 -> 1.3.3-3.fc33
  flashrom 1.2-5.fc33 -> 1.2-3.fc33
  flatpak 1.10.1-1.fc33 -> 1.8.3-1.fc33
  flatpak-libs 1.10.1-1.fc33 -> 1.8.3-1.fc33
  flatpak-selinux 1.10.1-1.fc33 -> 1.8.3-1.fc33
  flatpak-session-helper 1.10.1-1.fc33 -> 1.8.3-1.fc33
  fuse-overlayfs 1.4.0-1.fc33 -> 1.2.0-1.fc33
  fwupd 1.5.5-1.fc33 -> 1.5.1-1.fc33
  fwupd-plugin-flashrom 1.5.5-1.fc33 -> 1.5.1-1.fc33
  fwupd-plugin-modem-manager 1.5.5-1.fc33 -> 1.5.1-1.fc33
  gcr 3.38.1-1.fc33 -> 3.38.0-2.fc33
  gcr-base 3.38.1-1.fc33 -> 3.38.0-2.fc33
  gdbm-libs 1:1.19-1.fc33 -> 1:1.18.1-5.fc33
  gdisk 1.0.6-1.fc33 -> 1.0.5-2.fc33
  gdm 1:3.38.2.1-1.fc33 -> 1:3.38.2-1.fc33
  ghostscript 9.53.3-4.fc33 -> 9.53.3-2.fc33
  git-core 2.29.2-3.fc33 -> 2.28.0-1.fc33
  gjs 1.66.2-2.fc33 -> 1.66.1-2.fc33
  glib2 2.66.4-1.fc33 -> 2.66.3-1.fc33
  glibc 2.32-3.fc33 -> 2.32-2.fc33
  glibc-all-langpacks 2.32-3.fc33 -> 2.32-2.fc33
  glibc-common 2.32-3.fc33 -> 2.32-2.fc33
  glibc-devel 2.32-3.fc33 -> 2.32-2.fc33
  glibc-headers-x86 2.32-3.fc33 -> 2.32-2.fc33
  glibmm24 2.64.5-3.fc33 -> 2.64.2-4.fc33
  gnome-classic-session 3.38.2-1.fc33 -> 3.38.1-1.fc33
  gnome-control-center 3.38.3-1.fc33 -> 3.38.1-2.fc33
  gnome-control-center-filesystem 3.38.3-1.fc33 -> 3.38.1-2.fc33
  gnome-desktop3 3.38.3-1.fc33 -> 3.38.1-1.fc33
  gnome-disk-utility 3.38.1-1.fc33 -> 3.38.0-1.fc33
  gnome-getting-started-docs 3.38.0-1.fc33 -> 3.36.2-2.fc33
  gnome-initial-setup 3.38.3-1.fc33 -> 3.38.1-1.fc33
  gnome-shell 3.38.3-1.fc33 -> 3.38.1-3.fc33
  gnome-shell-extension-apps-menu 3.38.2-1.fc33 -> 3.38.1-1.fc33
  gnome-shell-extension-common 3.38.2-1.fc33 -> 3.38.1-1.fc33
  gnome-shell-extension-horizontal-workspaces 3.38.2-1.fc33 -> 3.38.1-1.fc33
  gnome-shell-extension-launch-new-instance 3.38.2-1.fc33 -> 3.38.1-1.fc33
  gnome-shell-extension-places-menu 3.38.2-1.fc33 -> 3.38.1-1.fc33
  gnome-shell-extension-user-theme 3.38.2-1.fc33 -> 3.38.1-1.fc33
  gnome-shell-extension-window-list 3.38.2-1.fc33 -> 3.38.1-1.fc33
  gnome-terminal 3.38.1-2.fc33 -> 3.38.1-1.fc33
  gnome-terminal-nautilus 3.38.1-2.fc33 -> 3.38.1-1.fc33
  gnome-tweaks 3.34.1-1.fc33 -> 3.34.0-6.fc33
  gnome-user-docs 3.38.2-1.fc33 -> 3.38.1-1.fc33
  gnupg2 2.2.25-2.fc33 -> 2.2.23-1.fc33
  gnupg2-smime 2.2.25-2.fc33 -> 2.2.23-1.fc33
  google-noto-emoji-color-fonts 20200916-1.fc33 -> 20200723-2.fc33
  gst-editing-services 1.18.2-1.fc33 -> 1.18.0-1.fc33
  gstreamer1 1.18.2-1.fc33 -> 1.18.1-1.fc33
  gstreamer1-libav 1.18.2-1.fc33 -> 1.18.1-1.fc33
  gstreamer1-plugins-bad-free 1.18.2-1.fc33 -> 1.18.1-1.fc33
  gstreamer1-plugins-bad-freeworld 1.18.2-1.fc33 -> 1.18.1-1.fc33
  gstreamer1-plugins-base 1.18.2-1.fc33 -> 1.18.1-1.fc33
  gstreamer1-plugins-good 1.18.2-1.fc33 -> 1.18.1-1.fc33
  gstreamer1-plugins-good-gtk 1.18.2-1.fc33 -> 1.18.1-1.fc33
  gstreamer1-plugins-good-qt 1.18.2-1.fc33 -> 1.18.1-1.fc33
  gstreamer1-plugins-ugly 1.18.2-1.fc33 -> 1.18.1-1.fc33
  gstreamer1-plugins-ugly-free 1.18.2-1.fc33 -> 1.18.1-1.fc33
  gtk-update-icon-cache 3.24.24-1.fc33 -> 3.24.23-1.fc33
  gtk2 2.24.33-1.fc33 -> 2.24.32-8.fc33
  gtk3 3.24.24-1.fc33 -> 3.24.23-1.fc33
  gtkmm30 3.24.3-1.fc33 -> 3.24.2-3.fc33
  gutenprint 5.3.4-1.fc33 -> 5.3.3-7.fc33
  gutenprint-cups 5.3.4-1.fc33 -> 5.3.3-7.fc33
  gutenprint-libs 5.3.4-1.fc33 -> 5.3.3-7.fc33
  gvfs 1.46.2-1.fc33 -> 1.46.1-1.fc33
  gvfs-afc 1.46.2-1.fc33 -> 1.46.1-1.fc33
  gvfs-afp 1.46.2-1.fc33 -> 1.46.1-1.fc33
  gvfs-archive 1.46.2-1.fc33 -> 1.46.1-1.fc33
  gvfs-client 1.46.2-1.fc33 -> 1.46.1-1.fc33
  gvfs-fuse 1.46.2-1.fc33 -> 1.46.1-1.fc33
  gvfs-goa 1.46.2-1.fc33 -> 1.46.1-1.fc33
  gvfs-gphoto2 1.46.2-1.fc33 -> 1.46.1-1.fc33
  gvfs-mtp 1.46.2-1.fc33 -> 1.46.1-1.fc33
  gvfs-smb 1.46.2-1.fc33 -> 1.46.1-1.fc33
  hplip 3.20.11-1.fc33 -> 3.20.9-2.fc33
  hplip-common 3.20.11-1.fc33 -> 3.20.9-2.fc33
  hplip-libs 3.20.11-1.fc33 -> 3.20.9-2.fc33
  hwdata 0.343-1.fc33 -> 0.340-1.fc33
  ibus 1.5.23-2.fc33 -> 1.5.23-1.fc33
  ibus-gtk2 1.5.23-2.fc33 -> 1.5.23-1.fc33
  ibus-gtk3 1.5.23-2.fc33 -> 1.5.23-1.fc33
  ibus-libs 1.5.23-2.fc33 -> 1.5.23-1.fc33
  ibus-setup 1.5.23-2.fc33 -> 1.5.23-1.fc33
  ibus-typing-booster 2.10.2-1.fc33 -> 2.10.0-1.fc33
  iproute 5.9.0-1.fc33 -> 5.8.0-1.fc33
  iproute-tc 5.9.0-1.fc33 -> 5.8.0-1.fc33
  iptables 1.8.5-4.fc33 -> 1.8.5-3.fc33
  iptables-libs 1.8.5-4.fc33 -> 1.8.5-3.fc33
  iptables-nft 1.8.5-4.fc33 -> 1.8.5-3.fc33
  iwl100-firmware 39.31.5.1-116.fc33 -> 39.31.5.1-113.fc33
  iwl1000-firmware 1:39.31.5.1-116.fc33 -> 1:39.31.5.1-113.fc33
  iwl105-firmware 18.168.6.1-116.fc33 -> 18.168.6.1-113.fc33
  iwl135-firmware 18.168.6.1-116.fc33 -> 18.168.6.1-113.fc33
  iwl2000-firmware 18.168.6.1-116.fc33 -> 18.168.6.1-113.fc33
  iwl2030-firmware 18.168.6.1-116.fc33 -> 18.168.6.1-113.fc33
  iwl3160-firmware 1:25.30.13.0-116.fc33 -> 1:25.30.13.0-113.fc33
  iwl3945-firmware 15.32.2.9-116.fc33 -> 15.32.2.9-113.fc33
  iwl4965-firmware 228.61.2.24-116.fc33 -> 228.61.2.24-113.fc33
  iwl5000-firmware 8.83.5.1_1-116.fc33 -> 8.83.5.1_1-113.fc33
  iwl5150-firmware 8.24.2.2-116.fc33 -> 8.24.2.2-113.fc33
  iwl6000-firmware 9.221.4.1-116.fc33 -> 9.221.4.1-113.fc33
  iwl6000g2a-firmware 18.168.6.1-116.fc33 -> 18.168.6.1-113.fc33
  iwl6000g2b-firmware 18.168.6.1-116.fc33 -> 18.168.6.1-113.fc33
  iwl6050-firmware 41.28.5.1-116.fc33 -> 41.28.5.1-113.fc33
  iwl7260-firmware 1:25.30.13.0-116.fc33 -> 1:25.30.13.0-113.fc33
  kernel 5.10.10-200.fc33 -> 5.9.8-200.fc33
  kernel-core 5.10.10-200.fc33 -> 5.9.8-200.fc33
  kernel-devel 5.10.10-200.fc33 -> 5.9.8-200.fc33
  kernel-headers 5.10.9-200.fc33 -> 5.9.7-200.fc33
  kernel-modules 5.10.10-200.fc33 -> 5.9.8-200.fc33
  kernel-modules-extra 5.10.10-200.fc33 -> 5.9.8-200.fc33
  keyutils 1.6.1-1.fc33 -> 1.6-5.fc33
  keyutils-libs 1.6.1-1.fc33 -> 1.6-5.fc33
  libaom 2.0.1-3.fc33 -> 2.0.0-3.fc33
  libarchive 3.5.1-1.fc33 -> 3.4.3-3.fc33
  libblkid 2.36.1-1.fc33 -> 2.36-3.fc33
  libbluray 1.2.1-2.fc33 -> 1.2.0-3.fc33
  libcurl 7.71.1-8.fc33 -> 7.71.1-7.fc33
  libdb 5.3.28-45.fc33 -> 5.3.28-44.fc33
  libdnf 0.55.2-1.fc33 -> 0.54.2-3.fc33
  libepoxy 1.5.5-1.fc33 -> 1.5.4-3.fc33
  libertas-sd8686-firmware 20201218-116.fc33 -> 20201022-113.fc33
  libertas-sd8787-firmware 20201218-116.fc33 -> 20201022-113.fc33
  libertas-usb8388-firmware 2:20201218-116.fc33 -> 2:20201022-113.fc33
  libfdisk 2.36.1-1.fc33 -> 2.36-3.fc33
  libgcc 10.2.1-9.fc33 -> 10.2.1-6.fc33
  libgcrypt 1.8.7-1.fc33 -> 1.8.6-4.fc33
  libgomp 10.2.1-9.fc33 -> 10.2.1-6.fc33
  libgpg-error 1.41-1.fc33 -> 1.37-2.fc33
  libgs 9.53.3-4.fc33 -> 9.53.3-2.fc33
  libhandy1 1.0.3-1.fc33 -> 1.0.2-1.fc33
  libibverbs 33.0-2.fc33 -> 32.0-1.fc33
  libical 3.0.9-1.fc33 -> 3.0.8-5.fc33
  libical-glib 3.0.9-1.fc33 -> 3.0.8-5.fc33
  libinput 1.16.4-1.fc33 -> 1.16.3-1.fc33
  libipa_hbac 2.4.0-4.fc33 -> 2.4.0-2.fc33
  libjcat 0.1.5-1.fc33 -> 0.1.4-1.fc33
  libmaxminddb 1.4.3-1.fc33 -> 1.4.2-3.fc33
  libmodulemd 2.12.0-1.fc33 -> 2.9.4-3.fc33
  libmount 2.36.1-1.fc33 -> 2.36-3.fc33
  libmtp 1.1.18-1.fc33 -> 1.1.16-5.fc33
  libnet 1.2-1.fc33 -> 1.1.6-20.fc33
  libnfsidmap 1:2.5.2-1.rc4.fc33 -> 1:2.5.2-1.rc1.fc33
  libopenmpt 0.4.16-1.fc33 -> 0.4.15-1.fc33
  libphonenumber 8.12.11-1.fc33 -> 8.12.7-1.fc33
  libpskc 2.6.6-1.fc33 -> 2.6.2-6.fc33
  libqmi 1.26.8-1.fc33 -> 1.26.6-1.fc33
  libqmi-utils 1.26.8-1.fc33 -> 1.26.6-1.fc33
  libreport-filesystem 2.14.0-15.fc33 -> 2.14.0-13.fc33
  librsvg2 2.50.2-1.fc33 -> 2.50.1-1.fc33
  libsane-airscan 0.99.23-1.fc33 -> 0.99.18-1.fc33
  libsane-hpaio 3.20.11-1.fc33 -> 3.20.9-2.fc33
  libsigc++20 2.10.6-1.fc33 -> 2.10.4-1.fc33
  libslirp 4.3.1-3.fc33 -> 4.3.1-2.fc33
  libsmartcols 2.36.1-1.fc33 -> 2.36-3.fc33
  libsmbclient 2:4.13.4-0.fc33 -> 2:4.13.2-1.fc33
  libsmbios 2.4.3-1.fc33 -> 2.4.2-10.fc33
  libsss_autofs 2.4.0-4.fc33 -> 2.4.0-2.fc33
  libsss_certmap 2.4.0-4.fc33 -> 2.4.0-2.fc33
  libsss_idmap 2.4.0-4.fc33 -> 2.4.0-2.fc33
  libsss_nss_idmap 2.4.0-4.fc33 -> 2.4.0-2.fc33
  libsss_sudo 2.4.0-4.fc33 -> 2.4.0-2.fc33
  libstdc++ 10.2.1-9.fc33 -> 10.2.1-6.fc33
  libtiff 4.1.0-6.fc33 -> 4.1.0-4.fc33
  libtracker-control 2.3.6-2.fc33 -> 2.3.6-1.fc33
  libtracker-miner 2.3.6-2.fc33 -> 2.3.6-1.fc33
  libtracker-sparql 2.3.6-2.fc33 -> 2.3.6-1.fc33
  libtracker-sparql3 3.0.2-3.fc33 -> 3.0.1-1.fc33
  libuuid 2.36.1-1.fc33 -> 2.36-3.fc33
  libwacom 1.7-2.fc33 -> 1.5-1.fc33
  libwacom-data 1.7-2.fc33 -> 1.5-1.fc33
  libwbclient 2:4.13.4-0.fc33 -> 2:4.13.2-1.fc33
  libzstd 1.4.7-1.fc33 -> 1.4.5-5.fc33
  linux-firmware 20201218-116.fc33 -> 20201022-113.fc33
  linux-firmware-whence 20201218-116.fc33 -> 20201022-113.fc33
  low-memory-monitor 2.1-1.fc33 -> 2.0-6.fc33
  lua-libs 5.4.2-1.fc33 -> 5.4.1-1.fc33
  mesa-dri-drivers 20.3.3-3.fc33 -> 20.2.2-1.fc33
  mesa-filesystem 20.3.3-3.fc33 -> 20.2.2-1.fc33
  mesa-libEGL 20.3.3-3.fc33 -> 20.2.2-1.fc33
  mesa-libGL 20.3.3-3.fc33 -> 20.2.2-1.fc33
  mesa-libgbm 20.3.3-3.fc33 -> 20.2.2-1.fc33
  mesa-libglapi 20.3.3-3.fc33 -> 20.2.2-1.fc33
  mesa-libxatracker 20.3.3-3.fc33 -> 20.2.2-1.fc33
  mesa-vulkan-drivers 20.3.3-3.fc33 -> 20.2.2-1.fc33
  microcode_ctl 2:2.1-43.fc33 -> 2:2.1-40.fc33
  minizip-compat 1.2.11-23.fc33 -> 1.2.11-22.fc33
  mlocate 0.26-27.fc33 -> 0.26-26.fc33
  mozjs78 78.7.0-1.fc33 -> 78.4.0-1.fc33
  mtr 2:0.94-1.fc33 -> 2:0.92-7.fc33
  mutter 3.38.3-1.fc33 -> 3.38.1-2.fc33
  nautilus 3.38.2-1.fc33 -> 3.38.1-1.fc33
  nautilus-extensions 3.38.2-1.fc33 -> 3.38.1-1.fc33
  net-snmp-libs 1:5.9-4.fc33 -> 1:5.9-2.fc33
  nfs-utils 1:2.5.2-1.rc4.fc33 -> 1:2.5.2-1.rc1.fc33
  nftables 1:0.9.3-7.fc33 -> 1:0.9.3-6.fc33
  nss 3.60.1-1.fc33 -> 3.58.0-3.fc33
  nss-softokn 3.60.1-1.fc33 -> 3.58.0-3.fc33
  nss-softokn-freebl 3.60.1-1.fc33 -> 3.58.0-3.fc33
  nss-sysinit 3.60.1-1.fc33 -> 3.58.0-3.fc33
  nss-util 3.60.1-1.fc33 -> 3.58.0-3.fc33
  open-vm-tools 11.2.5-1.fc33 -> 11.2.0-1.fc33
  open-vm-tools-desktop 11.2.5-1.fc33 -> 11.2.0-1.fc33
  openjpeg2 2.3.1-10.fc33 -> 2.3.1-7.fc33
  opensc 0.21.0-1.fc33 -> 0.20.0-7.fc33
  openssh 8.4p1-4.fc33 -> 8.4p1-2.fc33
  openssh-clients 8.4p1-4.fc33 -> 8.4p1-2.fc33
  openssh-server 8.4p1-4.fc33 -> 8.4p1-2.fc33
  openssl-libs 1:1.1.1i-1.fc33 -> 1:1.1.1h-1.fc33
  openvpn 2.4.10-1.fc33 -> 2.4.9-2.fc33
  orca 3.38.2-1.fc33 -> 3.38.0-1.fc33
  osinfo-db 20201218-1.fc33 -> 20201015-1.fc33
  ostree 2020.8-1.fc33 -> 2020.7-3.fc33
  ostree-grub2 2020.8-1.fc33 -> 2020.7-3.fc33
  ostree-libs 2020.8-1.fc33 -> 2020.7-3.fc33
  p11-kit 0.23.22-2.fc33 -> 0.23.21-2.fc33
  p11-kit-server 0.23.22-2.fc33 -> 0.23.21-2.fc33
  p11-kit-trust 0.23.22-2.fc33 -> 0.23.21-2.fc33
  pam 1.4.0-10.fc33 -> 1.4.0-8.fc33
  pango 1.48.1-1.fc33 -> 1.48.0-1.fc33
  pangomm 2.42.2-1.fc33 -> 2.42.1-2.fc33
  pcre2 10.36-1.fc33 -> 10.35-8.fc33
  pcre2-syntax 10.36-1.fc33 -> 10.35-8.fc33
  pcre2-utf16 10.36-1.fc33 -> 10.35-8.fc33
  pcre2-utf32 10.36-1.fc33 -> 10.35-8.fc33
  perl-AutoLoader 5.74-466.fc33 -> 5.74-464.fc33
  perl-B 1.80-466.fc33 -> 1.80-464.fc33
  perl-Class-Struct 0.66-466.fc33 -> 0.66-464.fc33
  perl-Encode 4:3.08-458.fc33 -> 4:3.07-457.fc33
  perl-Errno 1.30-466.fc33 -> 1.30-464.fc33
  perl-Fcntl 1.13-466.fc33 -> 1.13-464.fc33
  perl-File-Basename 2.85-466.fc33 -> 2.85-464.fc33
  perl-File-stat 1.09-466.fc33 -> 1.09-464.fc33
  perl-FileHandle 2.03-466.fc33 -> 2.03-464.fc33
  perl-Getopt-Std 1.12-466.fc33 -> 1.12-464.fc33
  perl-IO 1.43-466.fc33 -> 1.43-464.fc33
  perl-IPC-Open3 1.21-466.fc33 -> 1.21-464.fc33
  perl-NDBM_File 1.15-466.fc33 -> 1.15-464.fc33
  perl-POSIX 1.94-466.fc33 -> 1.94-464.fc33
  perl-SelectSaver 1.02-466.fc33 -> 1.02-464.fc33
  perl-Socket 4:2.031-1.fc33 -> 4:2.030-2.fc33
  perl-Symbol 1.08-466.fc33 -> 1.08-464.fc33
  perl-base 2.27-466.fc33 -> 2.27-464.fc33
  perl-if 0.60.800-466.fc33 -> 0.60.800-464.fc33
  perl-interpreter 4:5.32.1-466.fc33 -> 4:5.32.0-464.fc33
  perl-libnet 3.13-1.fc33 -> 3.11-458.fc33
  perl-libs 4:5.32.1-466.fc33 -> 4:5.32.0-464.fc33
  perl-macros 4:5.32.1-466.fc33 -> 4:5.32.0-464.fc33
  perl-mro 1.23-466.fc33 -> 1.23-464.fc33
  perl-overload 1.31-466.fc33 -> 1.31-464.fc33
  perl-overloading 0.02-466.fc33 -> 0.02-464.fc33
  perl-subs 1.03-466.fc33 -> 1.03-464.fc33
  perl-vars 1.05-466.fc33 -> 1.05-464.fc33
  pinentry 1.1.0-9.fc33 -> 1.1.0-8.fc33
  pinentry-gnome3 1.1.0-9.fc33 -> 1.1.0-8.fc33
  pinentry-gtk 1.1.0-9.fc33 -> 1.1.0-8.fc33
  pipewire 0.3.20-1.fc33 -> 0.3.15-2.fc33
  pipewire-gstreamer 0.3.20-1.fc33 -> 0.3.15-2.fc33
  pipewire-libs 0.3.20-1.fc33 -> 0.3.15-2.fc33
  podman 2:2.2.1-1.fc33 -> 2:2.1.1-12.fc33
  podman-plugins 2:2.2.1-1.fc33 -> 2:2.1.1-12.fc33
  procps-ng 3.3.16-2.fc33 -> 3.3.16-1.fc33
  pugixml 1.11.4-1.fc33 -> 1.10-4.fc33
  pulseaudio 14.0-2.fc33 -> 13.99.2-1.fc33
  pulseaudio-libs 14.0-2.fc33 -> 13.99.2-1.fc33
  pulseaudio-libs-glib2 14.0-2.fc33 -> 13.99.2-1.fc33
  pulseaudio-module-bluetooth 14.0-2.fc33 -> 13.99.2-1.fc33
  pulseaudio-module-x11 14.0-2.fc33 -> 13.99.2-1.fc33
  pulseaudio-utils 14.0-2.fc33 -> 13.99.2-1.fc33
  python-unversioned-command 3.9.1-2.fc33 -> 3.9.0-1.fc33
  python3 3.9.1-2.fc33 -> 3.9.0-1.fc33
  python3-audit 3.0-1.fc33 -> 3.0-0.21.20191104git1c2f876.fc33
  python3-cups 2.0.1-7.fc33 -> 2.0.1-5.fc33
  python3-firewall 0.8.6-1.fc33 -> 0.8.4-1.fc33
  python3-libs 3.9.1-2.fc33 -> 3.9.0-1.fc33
  python3-nftables 1:0.9.3-7.fc33 -> 1:0.9.3-6.fc33
  python3-pillow 7.2.0-3.fc33 -> 7.2.0-2.fc33
  python3-rpm 4.16.1.2-1.fc33 -> 4.16.0-1.fc33
  python3-speechd 0.10.2-1.fc33 -> 0.10.1-2.fc33
  qemu-guest-agent 2:5.1.0-9.fc33 -> 2:5.1.0-5.fc33
  qgnomeplatform 0.7.0-2.fc33 -> 0.6.1-3.fc33
  qpdf-libs 10.1.0-1.fc33 -> 10.0.3-1.fc33
  qt-settings 33.0-3.fc33 -> 33.0-2.fc33
  qt5-qtbase 5.15.2-2.fc33 -> 5.15.1-7.fc33
  qt5-qtbase-common 5.15.2-2.fc33 -> 5.15.1-7.fc33
  qt5-qtbase-gui 5.15.2-2.fc33 -> 5.15.1-7.fc33
  qt5-qtdeclarative 5.15.2-2.fc33 -> 5.15.1-1.fc33
  qt5-qtwayland 5.15.2-3.fc33 -> 5.15.1-1.fc33
  qt5-qtx11extras 5.15.2-2.fc33 -> 5.15.1-1.fc33
  qt5-qtxmlpatterns 5.15.2-2.fc33 -> 5.15.1-1.fc33
  raptor2 2.0.15-27.fc33 -> 2.0.15-26.fc33
  rav1e-libs 0.4.0~alpha-0.1.fc33 -> 0.3.3-2.fc33.1
  rdma-core 33.0-2.fc33 -> 32.0-1.fc33
  rpm 4.16.1.2-1.fc33 -> 4.16.0-1.fc33
  rpm-build-libs 4.16.1.2-1.fc33 -> 4.16.0-1.fc33
  rpm-libs 4.16.1.2-1.fc33 -> 4.16.0-1.fc33
  rpm-ostree 2021.1-2.fc33 -> 2020.8-1.fc33
  rpm-ostree-libs 2021.1-2.fc33 -> 2020.8-1.fc33
  rpm-plugin-selinux 4.16.1.2-1.fc33 -> 4.16.0-1.fc33
  rpm-sign-libs 4.16.1.2-1.fc33 -> 4.16.0-1.fc33
  rsync 3.2.3-3.fc33 -> 3.2.3-1.fc33
  samba-client 2:4.13.4-0.fc33 -> 2:4.13.2-1.fc33
  samba-client-libs 2:4.13.4-0.fc33 -> 2:4.13.2-1.fc33
  samba-common 2:4.13.4-0.fc33 -> 2:4.13.2-1.fc33
  samba-common-libs 2:4.13.4-0.fc33 -> 2:4.13.2-1.fc33
  samba-libs 2:4.13.4-0.fc33 -> 2:4.13.2-1.fc33
  sane-airscan 0.99.23-1.fc33 -> 0.99.18-1.fc33
  selinux-policy 3.14.6-34.fc33 -> 3.14.6-30.fc33
  selinux-policy-targeted 3.14.6-34.fc33 -> 3.14.6-30.fc33
  shadow-utils 2:4.8.1-5.fc33 -> 2:4.8.1-4.fc33
  skopeo 1:1.2.0-13.fc33 -> 1:1.2.0-3.fc33
  slirp4netns 1.1.8-1.fc33 -> 1.1.4-4.dev.giteecccdb.fc33
  speech-dispatcher 0.10.2-1.fc33 -> 0.10.1-2.fc33
  speech-dispatcher-espeak-ng 0.10.2-1.fc33 -> 0.10.1-2.fc33
  sqlite-libs 3.34.1-1.fc33 -> 3.33.0-2.fc33
  sssd 2.4.0-4.fc33 -> 2.4.0-2.fc33
  sssd-ad 2.4.0-4.fc33 -> 2.4.0-2.fc33
  sssd-client 2.4.0-4.fc33 -> 2.4.0-2.fc33
  sssd-common 2.4.0-4.fc33 -> 2.4.0-2.fc33
  sssd-common-pac 2.4.0-4.fc33 -> 2.4.0-2.fc33
  sssd-ipa 2.4.0-4.fc33 -> 2.4.0-2.fc33
  sssd-kcm 2.4.0-4.fc33 -> 2.4.0-2.fc33
  sssd-krb5 2.4.0-4.fc33 -> 2.4.0-2.fc33
  sssd-krb5-common 2.4.0-4.fc33 -> 2.4.0-2.fc33
  sssd-ldap 2.4.0-4.fc33 -> 2.4.0-2.fc33
  sssd-nfs-idmap 2.4.0-4.fc33 -> 2.4.0-2.fc33
  sssd-proxy 2.4.0-4.fc33 -> 2.4.0-2.fc33
  sudo 1.9.5p2-1.fc33 -> 1.9.2-1.fc33
  switcheroo-control 2.4-1.fc33 -> 2.2-2.fc33
  system-config-printer-libs 1.5.15-1.fc33 -> 1.5.13-1.fc33
  system-config-printer-udev 1.5.15-1.fc33 -> 1.5.13-1.fc33
  systemd 246.7-2.fc33 -> 246.6-3.fc33
  systemd-libs 246.7-2.fc33 -> 246.6-3.fc33
  systemd-networkd 246.7-2.fc33 -> 246.6-3.fc33
  systemd-pam 246.7-2.fc33 -> 246.6-3.fc33
  systemd-rpm-macros 246.7-2.fc33 -> 246.6-3.fc33
  systemd-udev 246.7-2.fc33 -> 246.6-3.fc33
  texlive-lib 7:20200327-19.fc33 -> 7:20200327-18.fc33
  thermald 2.4.1-1.fc33 -> 2.3-2.fc33
  time 1.9-12.fc33 -> 1.9-11.fc33
  toolbox 0.0.99-1.fc33 -> 0.0.97-1.fc33
  tpm2-tss 3.0.3-1.fc33 -> 3.0.1-1.fc33
  tracker 2.3.6-2.fc33 -> 2.3.6-1.fc33
  tracker3 3.0.2-3.fc33 -> 3.0.1-1.fc33
  tracker3-miners 3.0.3-1.fc33 -> 3.0.1-1.fc33
  tzdata 2021a-1.fc33 -> 2020d-1.fc33
  urw-base35-bookman-fonts 20200910-2.fc33 -> 20200910-1.fc33
  urw-base35-c059-fonts 20200910-2.fc33 -> 20200910-1.fc33
  urw-base35-d050000l-fonts 20200910-2.fc33 -> 20200910-1.fc33
  urw-base35-fonts 20200910-2.fc33 -> 20200910-1.fc33
  urw-base35-fonts-common 20200910-2.fc33 -> 20200910-1.fc33
  urw-base35-gothic-fonts 20200910-2.fc33 -> 20200910-1.fc33
  urw-base35-nimbus-mono-ps-fonts 20200910-2.fc33 -> 20200910-1.fc33
  urw-base35-nimbus-roman-fonts 20200910-2.fc33 -> 20200910-1.fc33
  urw-base35-nimbus-sans-fonts 20200910-2.fc33 -> 20200910-1.fc33
  urw-base35-p052-fonts 20200910-2.fc33 -> 20200910-1.fc33
  urw-base35-standard-symbols-ps-fonts 20200910-2.fc33 -> 20200910-1.fc33
  urw-base35-z003-fonts 20200910-2.fc33 -> 20200910-1.fc33
  usb_modeswitch 2.6.1-1.fc33 -> 2.6.0-2.fc33
  util-linux 2.36.1-1.fc33 -> 2.36-3.fc33
  vim-minimal 2:8.2.2146-2.fc33 -> 2:8.2.1961-1.fc33
  virtualbox-guest-additions 6.1.18-1.fc33 -> 6.1.16-1.fc33
  vte-profile 0.62.1-2.fc33 -> 0.62.1-1.fc33
  vte291 0.62.1-2.fc33 -> 0.62.1-1.fc33
  wavpack 5.4.0-1.fc33 -> 5.3.0-2.fc33
  webkit2gtk3 2.30.4-1.fc33 -> 2.30.2-1.fc33
  webkit2gtk3-jsc 2.30.4-1.fc33 -> 2.30.2-1.fc33
  wget 1.21.1-1.fc33 -> 1.20.3-8.fc33
  xorg-x11-server-Xorg 1.20.10-1.fc33 -> 1.20.9-1.fc33
  xorg-x11-server-Xwayland 1.20.10-1.fc33 -> 1.20.9-1.fc33
  xorg-x11-server-common 1.20.10-1.fc33 -> 1.20.9-1.fc33
  xz 5.2.5-4.fc33 -> 5.2.5-3.fc33
  xz-libs 5.2.5-4.fc33 -> 5.2.5-3.fc33
  yelp 2:3.38.2-1.fc33 -> 2:3.38.1-1.fc33
  yelp-libs 2:3.38.2-1.fc33 -> 2:3.38.1-1.fc33
  yelp-xsl 3.38.2-1.fc33 -> 3.38.1-1.fc33
  zchunk-libs 1.1.9-1.fc33 -> 1.1.5-3.fc33
  zimg 3.0.1-2.fc33 -> 2.9.3-3.fc33
  zlib 1.2.11-23.fc33 -> 1.2.11-22.fc33
  zlib-devel 1.2.11-23.fc33 -> 1.2.11-22.fc33
Removed:
  cifs-utils-info-6.11-2.fc33.x86_64
  criu-libs-3.15-1.fc33.x86_64
  evince-thumbnailer-3.38.0-2.fc33.x86_64
  fedora-repos-archive-33-1.noarch
  libadwaita-qt5-1.2.0-1.fc33.x86_64
  libjaylink-0.2.0-1.fc33.x86_64
  libldac-2.0.2.3-7.fc33.x86_64
  sudo-python-plugin-1.9.5p2-1.fc33.x86_64
Added:
  xorg-x11-font-utils-1:7.5-45.fc33.x86_64
@ptomato ptomato added the 1. Bug Something isn't working label Jan 30, 2021
@ptomato
Copy link
Author

ptomato commented Jan 30, 2021

At least, a workaround is rolling back the upgrade.

@ralfkaa
Copy link

ralfkaa commented Feb 2, 2021

same issue, @ptomato, except my rollback didn't provide a workaround.

not sure where to go from here, short of giving up on Silverblue and going back to Workstation...

@debarshiray
Copy link
Member

These lines confirm that the entry point of the container got started (as PID 12307):

DEBU Entry point of container fedora-toolbox-33 is toolbox (PID=12307)
DEBU Waiting for container fedora-toolbox-33 to finish initializing
DEBU Creating runtime directory /run/user/1000/toolbox
DEBU Checking if initialization stamp /run/user/1000/toolbox/container-initialized-12307 exists
Error: failed to initialize container fedora-toolbox-33

Hence, this is different from #669 where the entry point didn't even start because the basic runtime environment is busted.

What's happening here is that the toolbox enter process is waiting for the container's entry point to signal that it's done initializing the container, so that it can proceed and present the user with the interactive shell prompt. However, that signal never arrives, and toolbox enter times out. I suspect that the container's entry point either crashed or errored out.

What does this say:

$ podman start --attach fedora-toolbox-33

@Shadow53
Copy link

Shadow53 commented Feb 8, 2021

I am getting a similar issue, and podman start --attach fedora-toolbox-33 gives the following:

Error: failed to connect to container's attach socket: /run/user/1000/libpod/tmp/socket/f33018c3d7069ffc05e6c8f97bf905b598cf01e9aa7b4f5f09367bb2e7ca6172/attach: dial unixpacket /run/user/1000/libpod/tmp/socket/f33018c3d7069ffc05e6c8f97bf905b598cf01e9aa7b4f5f09367bb2e7ca6172/attach: connect: permission denied

@debarshiray
Copy link
Member

Error: failed to connect to container's attach socket

What's the output from podman info?

@vrothberg
Copy link
Member

vrothberg commented Feb 8, 2021 •
edited

Can you try running with the latest version of conmon?

@ptomato
Copy link
Author

ptomato commented Feb 10, 2021

How urgently do you need this info? I am happy to provide it but it may take me quite some time until I find a moment when I can intentionally upgrade my system to a broken state and then downgrade it again, since I need to use it for work in the meantime 😄

@debarshiray
Copy link
Member

How urgently do you need this info? I am happy to provide it but it may take
me quite some time until I find a moment when I can intentionally upgrade
my system to a broken state and then downgrade it again, since I need to use
it for work in the meantime

It's ok; no stress. All of us have plenty on our plates to keep us occupied. :)

@djselbeck
Copy link

djselbeck commented Feb 13, 2021 •
edited

I have the same issue since the last silverblue update which updated conman.

Error: command /bin/bash not found in container fedora-toolbox-33
Using /bin/bash instead.
[hendrik@ghost ~]$ podman info
host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.26-1.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.26, commit: 777074ecdb5e883b9bec233f3630c5e7fa37d521'
  cpus: 4
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: ghost.djselbeck.de
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1732600001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1732600001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.14-200.fc33.x86_64
  linkmode: dynamic
  memFree: 12743008256
  memTotal: 16645468160
  ociRuntime:
    name: crun
    package: crun-0.17-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1732600001/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 2m 30.11s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /var/home/hendrik/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.4.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.4
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/hendrik/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1732600001/containers
  volumePath: /home/hendrik/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1607438270
  BuiltTime: Tue Dec  8 15:37:50 2020
  GitCommit: ""
  GoVersion: go1.15.5
  OsArch: linux/amd64
  Version: 2.2.1
[hendrik@ghost ~]$ podman ps
CONTAINER ID  IMAGE                                             COMMAND               CREATED       STATUS            PORTS   NAMES
bba25afb94b4  registry.fedoraproject.org/f33/fedora-toolbox:33  toolbox --verbose...  15 hours ago  Up 5 minutes ago          fedora-toolbox-33
[hendrik@ghost ~]$ podman exec -it bba25afb94b4 /usr/bin/bash
ERRO[0005] Container bba25afb94b4c496f11e098a53dfedaf6c292ee30eb7eddf675c7df4a9f04f7c exec session 69e51c66052128bdc89ef95fec184a01c1ebb9e43b25164ca5d0590892f2ddb7 error: failed to connect to container's attach socket: /run/user/1732600001/libpod/tmp/socket/69e51c66052128bdc89ef95fec184a01c1ebb9e43b25164ca5d0590892f2ddb7/att: dial unixpacket /run/user/1732600001/libpod/tmp/socket/69e51c66052128bdc89ef95fec184a01c1ebb9e43b25164ca5d0590892f2ddb7/att: connect: no such file or directory 
Error: timed out waiting for file /home/hendrik/.local/share/containers/storage/overlay-containers/bba25afb94b4c496f11e098a53dfedaf6c292ee30eb7eddf675c7df4a9f04f7c/userdata/69e51c66052128bdc89ef95fec184a01c1ebb9e43b25164ca5d0590892f2ddb7/exit/bba25afb94b4c496f11e098a53dfedaf6c292ee30eb7eddf675c7df4a9f04f7c: internal libpod error

Selecting previous ostree in Grub with working toolbox:

[hendrik@ghost ~]$ podman info
host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-3.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'
  cpus: 4
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: ghost.djselbeck.de
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1732600001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1732600001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.14-200.fc33.x86_64
  linkmode: dynamic
  memFree: 14064754688
  memTotal: 16645459968
  ociRuntime:
    name: crun
    package: crun-0.17-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1732600001/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 49.26s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /var/home/hendrik/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.4.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.4
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/hendrik/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1732600001/containers
  volumePath: /home/hendrik/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1607438270
  BuiltTime: Tue Dec  8 15:37:50 2020
  GitCommit: ""
  GoVersion: go1.15.5
  OsArch: linux/amd64
  Version: 2.2.1

@andrewgdunn
Copy link

andrewgdunn commented Feb 13, 2021 •
edited

Wanted to point out the post I made here a little bit back, also attempting to follow what is in this thread:

[agd@enoch ~]$ lsblk -f
NAME                             FSTYPE      FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINT
zram0                                                                                                        [SWAP]
nvme0n1                                                                                                      
├─nvme0n1p1                      vfat        FAT32       62C2-FE2F                            1013.5M     1% /boot/efi
├─nvme0n1p2                      ext4        1.0         0acadce3-1f45-459b-a873-bbce11c6548b  818.1M     9% /boot
└─nvme0n1p3                      crypto_LUKS 2           2f50a945-cd8c-4ee8-abe3-ceebe6ea0f37                
  └─luks-2f50a945-cd8c-4ee8-abe3-ceebe6ea0f37
                                 btrfs             enoch f580fb00-596b-4546-8212-f536d292a487    1.4T    27% /var

root is on btrfs, as that may have something to do with this. Attempting to follow the golden path silverblue installation. (looked at what the automatic partitioning did and copied it with custom but with slightly different sizes).

[agd@enoch ~]$ du -sh .local/share/containers
du: cannot access '.local/share/containers': No such file or directory

[agd@enoch ~]$ toolbox create test
Image required to create toolbox container.
Download registry.fedoraproject.org/f33/fedora-toolbox:33 (500MB)? [y/N]: y
Created container: test
Enter with: toolbox enter test

[agd@enoch ~]$ du -sh .local/share/containers
du: cannot read directory '.local/share/containers/storage/overlay-containers/3b8f5f6945438b788f7f331d788793ad5ee7f14e786e565301c308a1a6d308fb/userdata': Permission denied
364M	.local/share/containers

[agd@enoch ~]$ toolbox enter -c test
Error: failed to initialize container test

[agd@enoch ~]$ podman info
host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.26-1.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.26, commit: 777074ecdb5e883b9bec233f3630c5e7fa37d521'
  cpus: 8
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: enoch
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.14-200.fc33.x86_64
  linkmode: dynamic
  memFree: 23844376576
  memTotal: 33360297984
  ociRuntime:
    name: crun
    package: crun-0.17-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 16m 42.84s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /var/home/agd/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.4.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.4
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /var/home/agd/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /var/home/agd/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1607438270
  BuiltTime: Tue Dec  8 09:37:50 2020
  GitCommit: ""
  GoVersion: go1.15.5
  OsArch: linux/amd64
  Version: 2.2.1

[agd@enoch ~]$ podman images
REPOSITORY                                     TAG     IMAGE ID      CREATED     SIZE
registry.fedoraproject.org/f33/fedora-toolbox  33      e6fddaf5336e  2 days ago  344 MB

[agd@enoch ~]$ podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

[agd@enoch ~]$ podman ps -a
CONTAINER ID  IMAGE                                             COMMAND               CREATED        STATUS                    PORTS   NAMES
3b8f5f694543  registry.fedoraproject.org/f33/fedora-toolbox:33  toolbox --verbose...  3 minutes ago  Exited (1) 2 minutes ago          test

[agd@enoch ~]$ podman start --attach test 
level=debug msg="Running as real user ID 0"
level=debug msg="Resolved absolute path to the executable as /usr/bin/toolbox"
level=debug msg="TOOLBOX_PATH is /usr/bin/toolbox"
level=debug msg="Creating /run/.toolboxenv"
level=debug msg="Monitoring host"
level=debug msg="Path /run/host/etc exists"
level=debug msg="Resolved /etc/localtime to /run/host/usr/share/zoneinfo/America/Detroit"
level=debug msg="Binding /etc/machine-id to /run/host/etc/machine-id"
level=debug msg="Creating /run/libvirt"
level=debug msg="Binding /run/libvirt to /run/host/run/libvirt"
level=debug msg="Creating /run/systemd/journal"
level=debug msg="Binding /run/systemd/journal to /run/host/run/systemd/journal"
level=debug msg="Creating /run/udev/data"
level=debug msg="Binding /run/udev/data to /run/host/run/udev/data"
level=debug msg="Creating /tmp"
level=debug msg="Binding /tmp to /run/host/tmp"
level=debug msg="Creating /var/lib/flatpak"
level=debug msg="Binding /var/lib/flatpak to /run/host/var/lib/flatpak"
level=debug msg="Creating /var/lib/libvirt"
level=debug msg="Binding /var/lib/libvirt to /run/host/var/lib/libvirt"
level=debug msg="Creating /var/lib/systemd/coredump"
level=debug msg="Binding /var/lib/systemd/coredump to /run/host/var/lib/systemd/coredump"
level=debug msg="Creating /var/log/journal"
level=debug msg="Binding /var/log/journal to /run/host/var/log/journal"
level=debug msg="Creating /var/mnt"
level=debug msg="Binding /var/mnt to /run/host/var/mnt"
level=debug msg="Creating /sys/fs/selinux"
level=debug msg="Binding /sys/fs/selinux to /usr/share/empty"
level=debug msg="Preparing to redirect /home to /var/home"
level=debug msg="/var/home isn't a symbolic link"
level=debug msg="Redirecting /home to /var/home"
level=debug msg="Looking up group for sudo"
level=debug msg="Group for sudo is wheel"
level=debug msg="Modifying user agd with UID 1000:"
level=debug msg=usermod
level=debug msg=--append
level=debug msg=--groups
level=debug msg=wheel
level=debug msg=--home
level=debug msg=/var/home/agd
level=debug msg=--shell
level=debug msg=/bin/bash
level=debug msg=--uid
level=debug msg=1000
level=debug msg=agd
level=debug msg="Removing password for user agd"
level=debug msg="Removing password for user root"
level=debug msg="Setting up daily ticker"
level=debug msg="Setting up watches for file system events"
Error: no space left on device

@debarshiray
Copy link
Member

Looks like both @djselbeck and @storrgie are having problems with conmon-2.0.26.

At least in @djselbeck 's case, conmon-2.0.21 works. Could you try that, @storrgie? If you don't have an older OSTree image, then you could download the RPM and use rpm-ostree override replace ... to create a new image locally with the older build.

@bam80
Copy link

bam80 commented Feb 16, 2021 •
edited

Could we have at least task somewhere to make unit tests for it?
This kind of problems seems recurring and it's not good to leave it on users back..

@debarshiray
Copy link
Member

So far, it looks like a conmon bug. In that case the tests would have to go there.

@bam80
Copy link

bam80 commented Feb 16, 2021

Even if the reason is in conmon, it's toolbox who fails to start, so maybe it's worth to have at least some basic automated tests here?
Otherwise, we will continue to get these problems in the future..

@debarshiray
Copy link
Member

Obviously there are basic automated tests. I am not sure why those aren't failing, though. At least 103-run.bats should be failing.

Sometimes, writing good tests isn't quite so easy. :)

@bam80
Copy link

bam80 commented Feb 16, 2021

Obviously there are basic automated tests.

Thanks, I didn't know that! That's really important.

At least 103-run.bats should be failing.

Yeah, probably we should make it so :)
Any ideas?

@lebel
Copy link

lebel commented Feb 16, 2021

toolbox keeps stopping working. I had the issue where my uid and gid were not the same, so I change it so they're the same, and get access back to toolbox, but now, I'm hit with this issue and even downgrading conmon as suggested above don't fix the issue.

I get the same behaviour as @storrgie.

@ralfkaa
Copy link

ralfkaa commented Feb 16, 2021

@storrgie mentioned btrfs may have something to do with it.
for whatever it's worth (i'm not a software engineer, don't know if this makes a difference): i gave up on Silverblue (deadlines...) and installed Workstation instead, with root on btrfs. i cannot replicate the problem i was having on Silverblue. toolbox works fine on F33 Workstation with root on btrfs.

@lebel
Copy link

lebel commented Feb 16, 2021 •
edited

I do use btrfs over luks, and my layout is as follow:

 I  1  lebel@gallifrey  ~  
↳ df -h
Sys. de fichiers    Taille Utilisé Dispo Uti% Monté sur
devtmpfs              7,7G       0  7,7G   0% /dev
tmpfs                 7,7G    168K  7,7G   1% /dev/shm
tmpfs                 3,1G     11M  3,1G   1% /run
/dev/dm-1             459G    104G  355G  23% /sysroot
tmpfs                 7,7G     56K  7,7G   1% /tmp
/dev/dm-1             459G    104G  355G  23% /var/home
/dev/dm-1             459G    104G  355G  23% /var/srv
/dev/dm-1             459G    104G  355G  23% /var/usrlocal
/dev/dm-1             459G    104G  355G  23% /var/srv/podman
/dev/nvme0n1p2       1014M    113M  902M  12% /boot
/dev/nvme0n1p1        499M    8,6M  491M   2% /boot/efi
/dev/mapper/cr_tank   932G    582G  326G  65% /var/mnt/tank
tmpfs                 1,6G    292K  1,6G   1% /run/user/1026
 I  lebel@gallifrey  ~  
↳

@andrewgdunn
Copy link

@debarshiray sorry it took me a moment:

[agd@enoch ~]$ date
Tue Feb 16 08:27:21 PM EST 2021

[agd@enoch ~]$ rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/33/x86_64/silverblue
                   Version: 33.20210216.0 (2021-02-16T01:47:24Z)
                BaseCommit: 54c3dada26965b67c1172a8d2c5067a58f33314db512fe98b97a8afd6b46f305
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
      ReplacedBasePackages: conmon 2:2.0.26-1.fc33 -> 2:2.0.21-3.fc33
           LayeredPackages: ansible baobab beets beets-plugins cheese darktable eog evince evolution fcct ffmpeg
                            file-roller firewall-config gimp git-lfs gmpc gnome-boxes gnome-firmware gnome-screenshot
                            gnome-sound-recorder gnome-tweaks htop hugo iotop iperf ipmitool keepassxc libreoffice make
                            mpc mpd mpdscribble mumble nautilus-image-converter nmap numix-gtk-theme
                            numix-icon-theme-circle oathtool openssl p7zip p7zip-plugins pass pavucontrol pavumeter
                            peek powertop rawtherapee screen seahorse seahorse-nautilus simple-scan sshuttle
                            system-config-printer vim virt-manager vlc wireguard-tools youtube-dl
             LocalPackages: rpmfusion-nonfree-release-33-1.noarch rpmfusion-free-release-33-1.noarch

Starting fresh (I think) by killing off the ~/.local/share/containers:

[agd@enoch ~]$ sudo rm -rf containers/

[agd@enoch ~]$ du -sh .local/share/containers/
du: cannot read directory '.local/share/containers/storage/overlay-containers/af0c4489a9d264310d5498ab112da3fffc23f5a2fc0a964047500f37935d4e42/userdata': Permission denied
364M	.local/share/containers/

[agd@enoch ~]$ toolbox enter -c test 
Error: failed to initialize container test

😢

[agd@enoch ~]$ podman info
host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-3.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'
  cpus: 8
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: enoch
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.15-200.fc33.x86_64
  linkmode: dynamic
  memFree: 25114050560
  memTotal: 33360302080
  ociRuntime:
    name: crun
    package: crun-0.17-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 8m 9.8s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /var/home/agd/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.4.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.4
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /var/home/agd/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /var/home/agd/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1607438270
  BuiltTime: Tue Dec  8 09:37:50 2020
  GitCommit: ""
  GoVersion: go1.15.5
  OsArch: linux/amd64
  Version: 2.2.1

[agd@enoch ~]$ podman start --attach test
level=debug msg="Running as real user ID 0"
level=debug msg="Resolved absolute path to the executable as /usr/bin/toolbox"
level=debug msg="TOOLBOX_PATH is /usr/bin/toolbox"
level=debug msg="Creating /run/.toolboxenv"
level=debug msg="Monitoring host"
level=debug msg="Path /run/host/etc exists"
level=debug msg="Resolved /etc/localtime to /run/host/usr/share/zoneinfo/America/Detroit"
level=debug msg="Binding /etc/machine-id to /run/host/etc/machine-id"
level=debug msg="Creating /run/libvirt"
level=debug msg="Binding /run/libvirt to /run/host/run/libvirt"
level=debug msg="Creating /run/systemd/journal"
level=debug msg="Binding /run/systemd/journal to /run/host/run/systemd/journal"
level=debug msg="Creating /run/udev/data"
level=debug msg="Binding /run/udev/data to /run/host/run/udev/data"
level=debug msg="Creating /tmp"
level=debug msg="Binding /tmp to /run/host/tmp"
level=debug msg="Creating /var/lib/flatpak"
level=debug msg="Binding /var/lib/flatpak to /run/host/var/lib/flatpak"
level=debug msg="Creating /var/lib/libvirt"
level=debug msg="Binding /var/lib/libvirt to /run/host/var/lib/libvirt"
level=debug msg="Creating /var/lib/systemd/coredump"
level=debug msg="Binding /var/lib/systemd/coredump to /run/host/var/lib/systemd/coredump"
level=debug msg="Creating /var/log/journal"
level=debug msg="Binding /var/log/journal to /run/host/var/log/journal"
level=debug msg="Creating /var/mnt"
level=debug msg="Binding /var/mnt to /run/host/var/mnt"
level=debug msg="Creating /sys/fs/selinux"
level=debug msg="Binding /sys/fs/selinux to /usr/share/empty"
level=debug msg="Preparing to redirect /home to /var/home"
level=debug msg="/var/home isn't a symbolic link"
level=debug msg="Redirecting /home to /var/home"
level=debug msg="Looking up group for sudo"
level=debug msg="Group for sudo is wheel"
level=debug msg="Modifying user agd with UID 1000:"
level=debug msg=usermod
level=debug msg=--append
level=debug msg=--groups
level=debug msg=wheel
level=debug msg=--home
level=debug msg=/var/home/agd
level=debug msg=--shell
level=debug msg=/bin/bash
level=debug msg=--uid
level=debug msg=1000
level=debug msg=agd
level=debug msg="Removing password for user agd"
level=debug msg="Removing password for user root"
level=debug msg="Setting up daily ticker"
level=debug msg="Setting up watches for file system events"
Error: no space left on device

[agd@enoch ~]$ sudo df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G   84K   16G   1% /dev/shm
tmpfs           6.3G   11M  6.3G   1% /run
/dev/dm-0       1.9T  519G  1.4T  28% /sysroot
/dev/nvme0n1p2  976M   91M  819M  10% /boot
/dev/nvme0n1p1 1022M  8.6M 1014M   1% /boot/efi
tmpfs            16G   40K   16G   1% /tmp
tmpfs           3.2G  180K  3.2G   1% /run/user/1000

Next test?

@HarryMichal HarryMichal added the 2. Container Realm The issue is related to what happens inside of a toolbox container label Feb 17, 2021
@debarshiray debarshiray mentioned this issue Feb 18, 2021
Closed
@debarshiray
Copy link
Member

Ok, so clearly @djselbeck and @storrgie are facing different bugs. Not sure about @ptomato yet.

@storrgie could you please file a separate issue about Btrfs and Error: no space left on device? That we can avoid having too many unrelated bugs in the same issue.

@lebel
Copy link

lebel commented Feb 18, 2021 •
edited

Additionnal informations, as I said, I run a fully uptodate Silverblue 33:

Deployments:
● ostree://fedora:fedora/33/x86_64/silverblue
                   Version: 33.20210217.0 (2021-02-17T04:32:09Z)
                BaseCommit: 34a158b0bf31bba4555b50e8455d971d8ec4f4f409240d7e1a95c5f4826e9010
                            ├─ repo-0 (2020-10-19T23:27:19Z)
                            ├─ repo-1 (2021-02-17T04:02:16Z)
                            └─ repo-2 (2021-02-17T04:08:43Z)
                    Commit: ee49ea1595c21eadcae979d4054b2fce12134e11a4c4c403305c40728818622b
                    Staged: no
                 StateRoot: fedora
              GPGSignature: 1 signature
                            Signature made mar 16 fév 2021 23:32:14 using RSA key ID 49FD77499570FF31
                            Good signature from "Fedora <fedora-33-primary@fedoraproject.org>"
           LayeredPackages: btrbk cargo ctags edk2-ovmf evolution-ews exfat-utils
                            fedora-workstation-repositories ffmpeg fish foo2xqx foo2zjs fuse-exfat go htop
                            kitty neovim postfix python3-neovim qemu-kvm rust scap-workbench snapper
                            syncthing teams tlp tmux-powerline virt-install virt-manager virt-viewer
                            waypipe

Here's when I try to attach with full debug:

↳ podman start --attach fedora-toolbox-33 --log-level debug                   712ms  jeu 2021-02-18 11h23
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called start.PersistentPreRunE(podman start --attach fedora-toolbox-33 --log-level debug) 
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.29.0 Annotations:[] CgroupNS:private Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:true Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:false Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private SeccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{ImageBuildFormat:oci CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/run/user/1026/libpod/tmp/events/events.log EventsLogger:journald HooksDir:[/usr/share/containers/oci/hooks.d] ImageDefaultTransport:docker:// InfraCommand: InfraImage:k8s.gcr.io/pause:3.2 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 StaticDir:/var/home/lebel/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/run/user/1026/libpod/tmp VolumePath:/var/home/lebel/.local/share/containers/storage/volumes} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/var/home/lebel/.config/cni/net.d}} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/lebel/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/lebel/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1026/containers     
DEBU[0000] Using static dir /var/home/lebel/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1026/libpod/tmp      
DEBU[0000] Using volume path /var/home/lebel/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/crun"                
WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] using runtime "/usr/bin/runc"                
INFO[0000] Setting parallel job count to 25             
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called start.PersistentPreRunE(podman start --attach fedora-toolbox-33 --log-level debug) 
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.29.0 Annotations:[] CgroupNS:private Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:true Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:false Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private SeccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{ImageBuildFormat:oci CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/run/user/1026/libpod/tmp/events/events.log EventsLogger:journald HooksDir:[/usr/share/containers/oci/hooks.d] ImageDefaultTransport:docker:// InfraCommand: InfraImage:k8s.gcr.io/pause:3.2 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 StaticDir:/var/home/lebel/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/run/user/1026/libpod/tmp VolumePath:/var/home/lebel/.local/share/containers/storage/volumes} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/var/home/lebel/.config/cni/net.d}} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/lebel/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/lebel/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1026/containers     
DEBU[0000] Using static dir /var/home/lebel/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1026/libpod/tmp      
DEBU[0000] Using volume path /var/home/lebel/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument 
INFO[0000] Setting parallel job count to 25             
DEBU[0000] overlay: mount_data=lowerdir=/var/home/lebel/.local/share/containers/storage/overlay/l/DBAK4O7WTZGYIJVSNZWFI2OD6U:/var/home/lebel/.local/share/containers/storage/overlay/l/VLVTGZBB5T4MPR7MCYUSQCKCOP,upperdir=/var/home/lebel/.local/share/containers/storage/overlay/ff3eced8a796f008391571db39e36515a4c5d4f94b962bf9db3be19a78013032/diff,workdir=/var/home/lebel/.local/share/containers/storage/overlay/ff3eced8a796f008391571db39e36515a4c5d4f94b962bf9db3be19a78013032/work,context="system_u:object_r:container_file_t:s0:c181,c885" 
DEBU[0000] mounted container "d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f" at "/var/home/lebel/.local/share/containers/storage/overlay/ff3eced8a796f008391571db39e36515a4c5d4f94b962bf9db3be19a78013032/merged" 
DEBU[0000] Created root filesystem for container d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f at /var/home/lebel/.local/share/containers/storage/overlay/ff3eced8a796f008391571db39e36515a4c5d4f94b962bf9db3be19a78013032/merged 
DEBU[0000] Not modifying container d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f /etc/passwd 
DEBU[0000] Not modifying container d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f /etc/group 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f to user.slice:libpod:d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] Created OCI spec for container d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f at /var/home/lebel/.local/share/containers/storage/overlay-containers/d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f -u d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f -r /usr/bin/crun -b /var/home/lebel/.local/share/containers/storage/overlay-containers/d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f/userdata -p /run/user/1026/containers/overlay-containers/d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f/userdata/pidfile -n fedora-toolbox-33 --exit-dir /run/user/1026/libpod/tmp/exits --socket-dir-path /run/user/1026/libpod/tmp/socket -s -l k8s-file:/var/home/lebel/.local/share/containers/storage/overlay-containers/d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f/userdata/ctr.log --log-level debug --syslog --conmon-pidfile /run/user/1026/containers/overlay-containers/d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/lebel/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1026/containers --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1026/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f" 
Error: unable to start container d26714d4e3adcdc49d9992fff59488e215e3b8e1dbfba0cb53cd56703083736f: make `/var/home/lebel/.local/share/containers/storage/overlay/ff3eced8a796f008391571db39e36515a4c5d4f94b962bf9db3be19a78013032/merged` private: Permission denied: OCI permission denied

The regular toolbox enter goes like this:

↳ toolbox enter -v --log-level=debug                                          393ms  jeu 2021-02-18 11h20
DEBU Running as real user ID 1026                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user lebel 
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Toolbox config directory is /var/home/lebel/.config/toolbox 
DEBU Current Podman version is 2.2.1              
DEBU Creating runtime directory /run/user/1026/toolbox 
DEBU Old Podman version is 3.0.0-dev              
DEBU Migration not needed: Podman version 2.2.1 is old 
DEBU Resolving container and image names          
DEBU Container: ''                                
DEBU Distribution: ''                             
DEBU Image: ''                                    
DEBU Release: ''                                  
DEBU Resolved container and image names           
DEBU Container: 'fedora-toolbox-33'               
DEBU Image: 'fedora-toolbox:33'                   
DEBU Release: '33'                                
DEBU Checking if container fedora-toolbox-33 exists 
DEBU Inspecting mounts of container fedora-toolbox-33 
DEBU Starting container fedora-toolbox-33         
Error: failed to start container fedora-toolbox-33

My id output is:

uid=1026(lebel) gid=1026(lebel) groupes=1026(lebel),10(wheel),100(users) contexte=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

@omichalek
Copy link

I too cannot use my containers
I am not on btrfs (it wasn't available back when I installed unfortunately).

Deployments:
● ostree://fedora:fedora/33/x86_64/silverblue
                   Version: 33.20210220.0 (2021-02-20T00:47:42Z)
                BaseCommit: b9bdf7d43d388d4159b1be0b0dbfb0fe6ed0b4003ae8862569610c0ecdabb3e8
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
       RemovedBasePackages: firefox 85.0.1-1.fc33
           LayeredPackages: dkms fuse-exfat hdparm langpacks-en lxc lxc-templates ncdu openssl recapp rhash snapd testdisk unrar wine-core
             LocalPackages: rpmfusion-nonfree-release-33-1.noarch rpmfusion-free-release-33-1.noarch touchegg-2.0.6-1.x86_64

  ostree://fedora:fedora/33/x86_64/silverblue
                   Version: 33.20210207.0 (2021-02-07T00:56:44Z)
                BaseCommit: 14706469057c430ff7484aee7014bc44ceed975c2367c60da5b4927918aaaeab
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
           LayeredPackages: dkms fuse-exfat hdparm langpacks-en lxc lxc-templates ncdu openssl recapp rhash snapd testdisk unrar wine-core
             LocalPackages: rpmfusion-nonfree-release-33-1.noarch rpmfusion-free-release-33-1.noarch touchegg-2.0.6-1.x86_64

$ toolbox enter
Error: failed to inspect entry point of container fedora-toolbox-33

$ toolbox rm fedora-toolbox-33
Error: failed to inspect container fedora-toolbox-33

$ toolbox enter --container fedora-toolbox-32
Error: failed to start container fedora-toolbox-32

$ toolbox enter --container fedora-toolbox-31
Error: failed to start container fedora-toolbox-31

$ toolbox enter --container fedora-toolbox-30
Error: failed to start container fedora-toolbox-30

only the fedora 33 container showed different error:

$ toolbox enter --container fedora-toolbox-33
Error: **failed to inspect entry point of container** fedora-toolbox-33

$ podman start --attach fedora-toolbox-33
level=debug msg="Running as real user ID 0"
level=debug msg="Resolved absolute path to the executable as /usr/bin/toolbox"
level=debug msg="TOOLBOX_PATH is /usr/bin/toolbox"
level=debug msg="XDG_RUNTIME_DIR is unset"
level=debug msg="XDG_RUNTIME_DIR set to /run/user/1000"
level=debug msg="Creating /run/.toolboxenv"
level=debug msg="Monitoring host"
level=debug msg="Path /run/host/etc exists"
level=debug msg="Resolved /etc/localtime to /run/host/usr/share/zoneinfo/Europe/Prague"
level=debug msg="Binding /etc/machine-id to /run/host/etc/machine-id"
level=debug msg="Creating /run/systemd/journal"
level=debug msg="Binding /run/systemd/journal to /run/host/run/systemd/journal"
level=debug msg="Creating /run/udev/data"
level=debug msg="Binding /run/udev/data to /run/host/run/udev/data"
level=debug msg="Creating /tmp"
level=debug msg="Binding /tmp to /run/host/tmp"
level=debug msg="Creating /var/lib/flatpak"
level=debug msg="Binding /var/lib/flatpak to /run/host/var/lib/flatpak"
level=debug msg="Creating /var/lib/systemd/coredump"
level=debug msg="Binding /var/lib/systemd/coredump to /run/host/var/lib/systemd/coredump"
level=debug msg="Creating /var/log/journal"
level=debug msg="Binding /var/log/journal to /run/host/var/log/journal"
level=debug msg="Creating /var/mnt"
level=debug msg="Binding /var/mnt to /run/host/var/mnt"
level=debug msg="Creating /sys/fs/selinux"
level=debug msg="Binding /sys/fs/selinux to /usr/share/empty"
level=debug msg="Preparing to redirect /home to /var/home"
level=debug msg="/var/home isn't a symbolic link"
level=debug msg="Redirecting /home to /var/home"
level=debug msg="Looking up group for sudo"
level=debug msg="Group for sudo is wheel"
level=debug msg="Modifying user user with UID 1000:"
level=debug msg=usermod
level=debug msg=--append
level=debug msg=--groups
level=debug msg=wheel
level=debug msg=--home
level=debug msg=/home/user
level=debug msg=--shell
level=debug msg=/bin/bash
level=debug msg=--uid
level=debug msg=1000
level=debug msg=user
level=debug msg="Removing password for user user"
level=debug msg="Removing password for user root"
level=debug msg="Setting up daily ticker"
level=debug msg="Setting up watches for file system events"
level=debug msg="Finished initializing container"
level=debug msg="Creating runtime directory /run/user/1000/toolbox"
level=debug msg="Creating initialization stamp /run/user/1000/toolbox/container-initialized-12258"
level=debug msg="Listening to file system and ticker events"

and then it changed to the same error the other containers show:

$ toolbox enter --container fedora-toolbox-33
Error: failed to start container fedora-toolbox-33

$ podman info
host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.26-1.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.26, commit: 777074ecdb5e883b9bec233f3630c5e7fa37d521'
  cpus: 8
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: PreAg
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.16-200.fc33.x86_64
  linkmode: dynamic
  memFree: 39933894656
  memTotal: 50411151360
  ociRuntime:
    name: crun
    package: crun-0.17-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 12712927232
  swapTotal: 12712927232
  uptime: 2h 41m 53.11s (Approximately 0.08 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /var/home/cecko/.config/containers/storage.conf
  containerStore:
    number: 12
    paused: 0
    running: 0
    stopped: 12
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.4.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.4
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /var/home/cecko/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 21
  runRoot: /run/user/1000
  volumePath: /var/home/cecko/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1607438270
  BuiltTime: Tue Dec  8 15:37:50 2020
  GitCommit: ""
  GoVersion: go1.15.5
  OsArch: linux/amd64
  Version: 2.2.1

@biji
Copy link

biji commented Mar 4, 2021

This works for me. Edit ~/.config/containers/containers.conf

[engine]
cgroup_manager = "cgroupfs"

reboot and try toolbox enter container again

@biji biji mentioned this issue Mar 4, 2021
Closed
@djselbeck
Copy link

The issue for me is fixed with a recent update in F33 silverblue.

Fixed podman info:

$  podman info
host:
  arch: amd64
  buildahVersion: 1.19.4
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.26-1.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.26, commit: 777074ecdb5e883b9bec233f3630c5e7fa37d521'
  cpus: 4
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: ghost.djselbeck.de
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1732600001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1732600001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.19-200.fc33.x86_64
  linkmode: dynamic
  memFree: 10808819712
  memTotal: 16645472256
  ociRuntime:
    name: crun
    package: crun-0.18-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.18
      commit: 808420efe3dc2b44d6db9f1a3fac8361dde42a95
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1732600001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 7m 27.88s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /var/home/hendrik/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.4.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.4
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/hendrik/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1732600001/containers
  volumePath: /home/hendrik/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1613753777
  BuiltTime: Fri Feb 19 17:56:17 2021
  GitCommit: ""
  GoVersion: go1.15.8
  OsArch: linux/amd64
  Version: 3.0.1

@returntrip
Copy link

I am having this issue on f34, as a workaround I followed @biji advice. Happy to porovide more info to trooubleshoot.

@ptomato
Copy link
Author

ptomato commented Mar 13, 2021

As in @djselbeck's case, the issue is resolved for me on the latest Fedora 33 Silverblue.

@notklaatu
Copy link

notklaatu commented May 5, 2021 •
edited

I'm also having this issue on Fedora Silverblue 34.

$ toolbox enter -v --log-level=debug 
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user klaatu 
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Toolbox config directory is /var/home/klaatu/.config/toolbox 
DEBU Current Podman version is 3.1.0              
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Old Podman version is 3.1.0                  
DEBU Migration not needed: Podman version 3.1.0 is unchanged 
DEBU Resolving container and image names          
DEBU Container: ''                                
DEBU Distribution: ''                             
DEBU Image: ''                                    
DEBU Release: ''                                  
DEBU Resolved container and image names           
DEBU Container: 'fedora-toolbox-34'               
DEBU Image: 'fedora-toolbox:34'                   
DEBU Release: '34'                                
DEBU Checking if container fedora-toolbox-34 exists 
DEBU Inspecting mounts of container fedora-toolbox-34 
DEBU Starting container fedora-toolbox-34         
DEBU Inspecting entry point of container fedora-toolbox-34 
DEBU Entry point PID is a float64                 
DEBU Entry point of container fedora-toolbox-34 is toolbox (PID=5797) 
DEBU Waiting for container fedora-toolbox-34 to finish initializing 
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Checking if initialization stamp /run/user/1000/toolbox/container-initialized-5797 exists 
Error: failed to initialize container fedora-toolbox-34

I may have been the cause, though, because I created a group called users (100) in /etc/group and promoted it as my primary group.

$ id
uid=1000(klaatu) gid=100(users) groups=100(users),10(wheel),1000(klaatu) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

Using newgrp to change my primary group and then running toolbox enter has no effect, and podman info seems to be aware of the correct id mappings:

  idMappings:
    gidmap:
    - container_id: 0
      host_id: 100
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536

@ptomato ptomato mentioned this issue May 15, 2021
Closed
@debarshiray
Copy link
Member

@ptomato and @djselbeck are reporting that their problems got fixed. Thanks for confirming that!

Closing.

@heyakyra
Copy link

heyakyra commented May 11, 2023 •
edited

Has this bug returned? I'm hitting it on a F38 container on Silverblue F38 host

#1260

@biji
Copy link

biji commented May 12, 2023

lol, breaks again?

@heyakyra heyakyra mentioned this issue Jun 15, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
1. Bug Something isn't working 2. Container Realm The issue is related to what happens inside of a toolbox container
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests