home not in container

[FlorianLudwig]

toolbox stopped working for me completely:

$ toolbox create f31 --image fedora-toolbox:31
$ toolbox enter --container f31
Error: directory /home/f.ludwig not found in container f31

For any directory.

System:

fedora 32

$ toolbox --version
toolbox version 0.0.92

$ podman --version
podman version 2.0.2

[HarryMichal]

Hi @FlorianLudwig! Is it possible for you to reproduce the problem again with all steps?? I'd like to to see the verbose output (--log-level debug) of commands toolbox create and toolbox enter. Also, I'd like to see the output of podman logs <name-of-container> after you try to enter the container.

[FlorianLudwig]

Sure, here you go:

$ toolbox create f31 --log-level debug --image fedora-toolbox:31
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v1 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user f.ludwig 
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Toolbox config directory is /home/f.ludwig/.config/toolbox 
DEBU Current Podman version is 2.0.2              
DEBU Old Podman version is 2.0.2                  
DEBU Migration not needed: Podman version 2.0.2 is unchanged 
DEBU Resolving container and image names          
DEBU Container: 'f31'                             
DEBU Image: 'fedora-toolbox:31'                   
DEBU Release: ''                                  
DEBU Resolved container and image names           
DEBU Container: 'f31'                             
DEBU Image: 'fedora-toolbox:31'                   
DEBU Release: '31'                                
DEBU Checking if container f31 already exists     
DEBU Looking for image fedora-toolbox:31          
DEBU Resolving fully qualified name for image fedora-toolbox:31 
DEBU Resolved image fedora-toolbox:31 to registry.fedoraproject.org/f31/fedora-toolbox:31 
DEBU Looking up group for sudo                    
DEBU Group for sudo is wheel                      
DEBU Checking if 'podman create' supports '--ulimit host' 
DEBU 'podman create' supports '--ulimit host'     
DEBU Resolving path to the D-Bus system socket    
DEBU Calling org.freedesktop.Flatpak.SessionHelper.RequestSession 
DEBU /home/f.ludwig canonicalized to /home/f.ludwig 
DEBU Checking if /usr is mounted read-only or read-write 
DEBU Mount-point of /usr is /                     
DEBU Mount flags of /usr on the host are rw,noatime,seclabel 
DEBU Resolving path to the KCM socket             
DEBU Checking if /media is a symbolic link to /run/media 
DEBU Checking if /mnt is a symbolic link to /var/mnt 
DEBU Looking for toolbox.sh                       
DEBU Found /etc/profile.d/toolbox.sh              
DEBU Checking if /home is a symbolic link to /var/home 
DEBU Creating container f31:                      
DEBU podman                                       
DEBU --log-level                                  
DEBU error                                        
DEBU create                                       
DEBU --dns                                        
DEBU none                                         
DEBU --env                                        
DEBU TOOLBOX_PATH=/usr/bin/toolbox                
DEBU --group-add                                  
DEBU wheel                                        
DEBU --hostname                                   
DEBU toolbox                                      
DEBU --ipc                                        
DEBU host                                         
DEBU --label                                      
DEBU com.github.containers.toolbox=true           
DEBU --label                                      
DEBU com.github.debarshiray.toolbox=true          
DEBU --name                                       
DEBU f31                                          
DEBU --network                                    
DEBU host                                         
DEBU --no-hosts                                   
DEBU --pid                                        
DEBU host                                         
DEBU --privileged                                 
DEBU --security-opt                               
DEBU label=disable                                
DEBU --ulimit                                     
DEBU host                                         
DEBU --userns=keep-id                             
DEBU --user                                       
DEBU root:root                                    
DEBU --volume                                     
DEBU /etc:/run/host/etc                           
DEBU --volume                                     
DEBU /dev:/dev:rslave                             
DEBU --volume                                     
DEBU /run:/run/host/run:rslave                    
DEBU --volume                                     
DEBU /tmp:/run/host/tmp:rslave                    
DEBU --volume                                     
DEBU /var:/run/host/var:rslave                    
DEBU --volume                                     
DEBU /run/dbus/system_bus_socket:/run/dbus/system_bus_socket 
DEBU --volume                                     
DEBU /run/user/1000/.flatpak-helper/monitor:/run/host/monitor 
DEBU --volume                                     
DEBU /home/f.ludwig:/home/f.ludwig:rslave         
DEBU --volume                                     
DEBU /usr/bin/toolbox:/usr/bin/toolbox:ro         
DEBU --volume                                     
DEBU /usr:/run/host/usr:rw,rslave                 
DEBU --volume                                     
DEBU /run/user/1000:/run/user/1000                
DEBU --volume                                     
DEBU /run/.heim_org.h5l.kcm-socket:/run/.heim_org.h5l.kcm-socket 
DEBU --volume                                     
DEBU /media:/media:rslave                         
DEBU --volume                                     
DEBU /mnt:/mnt:rslave                             
DEBU --volume                                     
DEBU /run/media:/run/media:rslave                 
DEBU --volume                                     
DEBU /etc/profile.d/toolbox.sh:/etc/profile.d/toolbox.sh:ro 
DEBU registry.fedoraproject.org/f31/fedora-toolbox:31 
DEBU toolbox                                      
DEBU --verbose                                    
DEBU init-container                               
DEBU --home                                       
DEBU /home/f.ludwig                               
DEBU --monitor-host                               
DEBU --shell                                      
DEBU /bin/bash                                    
DEBU --uid                                        
DEBU 1000                                         
DEBU --user                                       
DEBU f.ludwig                                     
Created container: f31
Enter with: toolbox enter --container f31

$ toolbox enter --container f31 --log-level debug
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v1 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user f.ludwig 
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Toolbox config directory is /home/f.ludwig/.config/toolbox 
DEBU Current Podman version is 2.0.2              
DEBU Old Podman version is 2.0.2                  
DEBU Migration not needed: Podman version 2.0.2 is unchanged 
DEBU Resolving container and image names          
DEBU Container: 'f31'                             
DEBU Image: ''                                    
DEBU Release: ''                                  
DEBU Resolved container and image names           
DEBU Container: 'f31'                             
DEBU Image: 'fedora-toolbox:32'                   
DEBU Release: '32'                                
DEBU Checking if container f31 exists             
DEBU Calling org.freedesktop.Flatpak.SessionHelper.RequestSession 
DEBU Starting container f31                       
DEBU Inspecting entry point of container f31      
DEBU Entry point PID is a float64                 
DEBU Entry point of container f31 is toolbox (PID=456924) 
DEBU Waiting for container f31 to finish initializing 
DEBU Checking if initialization stamp /run/user/1000/toolbox/container-initialized-456924 exists 
DEBU Container f31 is initialized                 
DEBU Looking for command /bin/bash in container f31 
Error: exec failed: container_linux.go:349: starting container process caused "exec: \"sh\": executable file not found in $PATH": OCI runtime command not found error
DEBU command /bin/bash not found in container f31; using /bin/bash instead 
DEBU Creating list of environment variables to forward 
DEBU COLORTERM=truecolor                          
DEBU DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus 
DEBU DBUS_SYSTEM_BUS_ADDRESS is unset             
DEBU DESKTOP_SESSION=gnome                        
DEBU DISPLAY=:0                                   
DEBU LANG=en_US.UTF-8                             
DEBU SHELL=/bin/bash                              
DEBU SSH_AUTH_SOCK=/run/user/1000/keyring/ssh     
DEBU TERM=xterm-256color                          
DEBU TOOLBOX_PATH=/usr/bin/toolbox                
DEBU VTE_VERSION=6003                             
DEBU WAYLAND_DISPLAY=wayland-0                    
DEBU XDG_CURRENT_DESKTOP=GNOME:Unity              
DEBU XDG_DATA_DIRS=/home/f.ludwig/.local/share/flatpak/exports/share/:/var/lib/flatpak/exports/share/:/usr/local/share/:/usr/share/ 
DEBU XDG_MENU_PREFIX=gnome-                       
DEBU XDG_RUNTIME_DIR=/run/user/1000               
DEBU XDG_SEAT is unset                            
DEBU XDG_SESSION_DESKTOP=gnome                    
DEBU XDG_SESSION_ID is unset                      
DEBU XDG_SESSION_TYPE=wayland                     
DEBU XDG_VTNR is unset                            
DEBU Running in container f31:                    
DEBU podman                                       
DEBU --log-level                                  
DEBU error                                        
DEBU exec                                         
DEBU --interactive                                
DEBU --tty                                        
DEBU --user                                       
DEBU f.ludwig                                     
DEBU --workdir                                    
DEBU /home/f.ludwig                               
DEBU --env=COLORTERM=truecolor                    
DEBU --env=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus 
DEBU --env=DESKTOP_SESSION=gnome                  
DEBU --env=DISPLAY=:0                             
DEBU --env=LANG=en_US.UTF-8                       
DEBU --env=SHELL=/bin/bash                        
DEBU --env=SSH_AUTH_SOCK=/run/user/1000/keyring/ssh 
DEBU --env=TERM=xterm-256color                    
DEBU --env=TOOLBOX_PATH=/usr/bin/toolbox          
DEBU --env=VTE_VERSION=6003                       
DEBU --env=WAYLAND_DISPLAY=wayland-0              
DEBU --env=XDG_CURRENT_DESKTOP=GNOME:Unity        
DEBU --env=XDG_DATA_DIRS=/home/f.ludwig/.local/share/flatpak/exports/share/:/var/lib/flatpak/exports/share/:/usr/local/share/:/usr/share/ 
DEBU --env=XDG_MENU_PREFIX=gnome-                 
DEBU --env=XDG_RUNTIME_DIR=/run/user/1000         
DEBU --env=XDG_SESSION_DESKTOP=gnome              
DEBU --env=XDG_SESSION_TYPE=wayland               
DEBU f31                                          
DEBU capsh                                        
DEBU --caps=                                      
DEBU --                                           
DEBU -c                                           
DEBU exec "$@"                                    
DEBU /bin/sh                                      
DEBU /bin/bash                                    
Error: exec failed: container_linux.go:349: starting container process caused "exec: \"capsh\": executable file not found in $PATH": OCI runtime command not found error
DEBU Looking for path /home/f.ludwig in container f31 
Error: exec failed: container_linux.go:349: starting container process caused "exec: \"sh\": executable file not found in $PATH": OCI runtime command not found error
Error: directory /home/f.ludwig not found in container f31

$ podman logs f31
level=debug msg="Running as real user ID 0"
level=debug msg="Resolved absolute path to the executable as /usr/bin/toolbox"
level=debug msg="TOOLBOX_PATH is /usr/bin/toolbox"
level=debug msg="XDG_RUNTIME_DIR is unset"
level=debug msg="XDG_RUNTIME_DIR set to /run/user/1000"
level=debug msg="Creating /run/.toolboxenv"
level=debug msg="Monitoring host"
level=debug msg="Path /run/host/etc exists"
level=debug msg="Redirecting /etc/host.conf to /run/host/etc/host.conf"
level=debug msg="Redirecting /etc/hosts to /run/host/etc/hosts"
level=debug msg="Redirecting /etc/resolv.conf to /run/host/etc/resolv.conf"
level=debug msg="Binding /etc/machine-id to /run/host/etc/machine-id"
level=debug msg="Creating /run/libvirt"
level=debug msg="Binding /run/libvirt to /run/host/run/libvirt"
level=debug msg="Creating /run/systemd/journal"
level=debug msg="Binding /run/systemd/journal to /run/host/run/systemd/journal"
level=debug msg="Creating /var/lib/flatpak"
level=debug msg="Binding /var/lib/flatpak to /run/host/var/lib/flatpak"
level=debug msg="Creating /var/log/journal"
level=debug msg="Binding /var/log/journal to /run/host/var/log/journal"
level=debug msg="Creating /sys/fs/selinux"
level=debug msg="Binding /sys/fs/selinux to /usr/share/empty"
level=debug msg="Path /run/host/monitor exists"
level=debug msg="Redirecting /etc/localtime to /run/host/monitor/localtime"
level=debug msg="Redirecting /etc/timezone to /run/host/monitor/timezone"
level=debug msg="Looking up group for sudo"
level=debug msg="Group for sudo is wheel"
level=debug msg="Adding user f.ludwig with UID 1000:"
level=debug msg=useradd
level=debug msg=--home-dir
level=debug msg=/home/f.ludwig
level=debug msg=--no-create-home
level=debug msg=--shell
level=debug msg=/bin/bash
level=debug msg=--uid
level=debug msg=1000
level=debug msg=--groups
level=debug msg=wheel
level=debug msg=f.ludwig
level=debug msg="Removing password for user f.ludwig"
passwd: Note: deleting a password also unlocks the password.
level=debug msg="Removing password for user root"
passwd: Note: deleting a password also unlocks the password.
level=debug msg="Setting KCM as the default Kerberos credential cache"
level=debug msg="Finished initializing container"
level=debug msg="Creating runtime directory /run/user/1000/toolbox"
level=debug msg="Creating initialization stamp /run/user/1000/toolbox/container-initialized-456924"
level=debug msg="Going to sleep"

[HarryMichal]

Thank you for the logs, @FlorianLudwig! I just managed to reproduce this issue without even trying but my scenario is a bit different because I use Silverblue where a home folder (/home/<insert-name> is a link to /var/home/<insert-name>). I managed to bypass it by entering the /var/home directory first and then trying to enter the container. This is very odd and I'll try to investigate some more.

[debarshiray]

These lines from toolbox enter ... are weird:

DEBU Looking for command /bin/bash in container f31 
Error: exec failed: container_linux.go:349: starting container process caused "exec: \"sh\": executable file not found in $PATH": OCI runtime command not found error
DEBU command /bin/bash not found in container f31; using /bin/bash instead

How can /bin/bash not be in the container? Were you in the middle of some RPM transactions?

[FlorianLudwig]

@debarshiray I did not create the image. I believe this image was created by the toolbox authors (I use toolbox create f31 --image fedora-toolbox:31 to create the container)

[debarshiray]

I did not create the image. I believe this image was created
by the toolbox authors

I know. That's why I was surprised. :)

Could you please mount the container's file system and take a peek if Bash is really missing:

$ podman unshare -- /bin/bash
# container_root=$(podman mount f31)
# ls -l $container_root/bin/bash
...
# ls -l $container_root/usr/bin/bash
...

[FlorianLudwig]

@debarshiray

looks fine to me:

[root@knight1 ~]# ls -l $container_root/bin/bash
-rwx------. 1 bin bin 1203992  6. Dez 2019  /home/f.ludwig/.local/share/containers/storage/overlay/2691c026589de34d450b4906db6449e0ced75f8b60db9d25be70ff22c10a1855/merged/bin/bash
[root@knight1 ~]# ls -l $container_root/usr/bin/bash
-rwx------. 1 bin bin 1203992  6. Dez 2019  /home/f.ludwig/.local/share/containers/storage/overlay/2691c026589de34d450b4906db6449e0ced75f8b60db9d25be70ff22c10a1855/merged/usr/bin/bash

[HarryMichal]

@FlorianLudwig, is the issue still present? Despite some investigation done by me, I couldn't find the culprit.

[FlorianLudwig]

@HarryMichal
I cannot reproduce this anymore with

toolbox --version
toolbox version 0.0.95

Therefore I am closing it.