Fail to enter Toolbox #169

floating-cat · 2019-05-21T08:14:03Z

Fedora 30, podman 1.3.1, Toolbox 0.0.9-1.
I failed to enter Toolbox.

~> toolbox -v enter
toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: Fedora generational core is f30
toolbox: base image is fedora-toolbox:30
toolbox: customized user-specific image is fedora-toolbox-my_name:30
toolbox: container is fedora-toolbox-my_name-30
toolbox: checking if container fedora-toolbox-my_name-30 exists
toolbox: container fedora-toolbox-my_name-30 was created from image localhost/fedora-toolbox-my_name:30
toolbox: checking if image localhost/fedora-toolbox-my_name:30 has volumes for host bind mounts
toolbox: trying to start container fedora-toolbox-my_name-30
toolbox: creating list of environment variables to forward
toolbox: COLORTERM=truecolor
toolbox: DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
toolbox: DBUS_SYSTEM_BUS_ADDRESS is unset
toolbox: DESKTOP_SESSION=/usr/share/xsessions/plasma
toolbox: DISPLAY=:0
toolbox: LANG=en_US.UTF-8
toolbox: SHELL=/usr/bin/fish
toolbox: SSH_AUTH_SOCK=/tmp/ssh-C5OHVekHJjzn/agent.1572
toolbox: TERM=xterm-256color
toolbox: TOOLBOX_PATH is unset
toolbox: VTE_VERSION=5603
toolbox: WAYLAND_DISPLAY is unset
toolbox: XDG_CURRENT_DESKTOP=KDE
toolbox: XDG_DATA_DIRS=/usr/share/kde-settings/kde-profile/default/share:/usr/local/share:/usr/share
toolbox: XDG_MENU_PREFIX=kf5-
toolbox: XDG_RUNTIME_DIR=/run/user/1000
toolbox: XDG_SEAT=seat0
toolbox: XDG_SESSION_DESKTOP=KDE
toolbox: XDG_SESSION_ID=3
toolbox: XDG_SESSION_TYPE=x11
toolbox: XDG_VTNR=1
toolbox: created options for environment variables to forward
--env=COLORTERM=truecolor --env=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus --env=DESKTOP_SESSION=/usr/share/xsessions/plasma --env=DISPLAY=:0 --env=LANG=en_US.UTF-8 --env=SHELL=/usr/bin/fish --env=SSH_AUTH_SOCK=/tmp/ssh-C5OHVekHJjzn/agent.1572 --env=TERM=xterm-256color --env=VTE_VERSION=5603 --env=XDG_CURRENT_DESKTOP=KDE --env=XDG_DATA_DIRS=/usr/share/kde-settings/kde-profile/default/share:/usr/local/share:/usr/share --env=XDG_MENU_PREFIX=kf5- --env=XDG_RUNTIME_DIR=/run/user/1000 --env=XDG_SEAT=seat0 --env=XDG_SESSION_DESKTOP=KDE --env=XDG_SESSION_ID=3 --env=XDG_SESSION_TYPE=x11 --env=XDG_VTNR=1
toolbox: looking for /usr/bin/fish in container fedora-toolbox-my_name-30
ERRO[0000] open /home/my_name/.local/share/containers/storage/overlay-containers/b39d66d68134c9cdb561ce2f57266bbe103463e5ad50bf2926151c1cc258c73e/userdata/conmon.pid: permission denied 
toolbox: /usr/bin/fish not found in fedora-toolbox-my_name-30; using /bin/bash instead
toolbox: trying to exec /bin/bash in container fedora-toolbox-my_name-30
ERRO[0000] open /home/my_name/.local/share/containers/storage/overlay-containers/b39d66d68134c9cdb561ce2f57266bbe103463e5ad50bf2926151c1cc258c73e/userdata/conmon.pid: permission denied

// I can't use some podman commands after I failed to enter Toolbar 
~> podman images
ERRO[0000] open /home/clj/.local/share/containers/storage/overlay-containers/b39d66d68134c9cdb561ce2f57266bbe103463e5ad50bf2926151c1cc258c73e/userdata/conmon.pid: permission denied

~> podman system migrate
// now podman images command works but I still fail to enter Toolbox

I am not sure whether this issue related to containers/podman#2935 or not.

The text was updated successfully, but these errors were encountered:

debarshiray · 2019-05-21T16:23:24Z

Since PR #166 the toolbox script will automatically migrate containers for you. This will be part of toolbox-0.0.10.

However, the migration only happens when all containers are stopped. eg., right after booting the system. In your case, your container was already started (ie. podman start had already been called). Try podman stop --all.

debarshiray · 2019-05-21T16:23:34Z

Thanks for getting in touch. Closing.

matthew-a-p · 2019-05-22T05:11:32Z

@debarshiray I seem to be stuck in this same state, but with newly created containers using toolbox in Silverblue 30:

[my_user@my_pc toolbox]$ sudo rm -rf ~/.local/share/containers

[my_user@my_pc toolbox]$ ./toolbox -v create -c testing00
toolbox: resolved absolute path for ./toolbox to /var/home/my_user/testing/toolbox/toolbox
toolbox: TOOLBOX_PATH is /var/home/my_user/testing/toolbox/toolbox
toolbox: checking if 'podman system migrate' exists
toolbox: migration not needed: 1.3.1 is unchanged
toolbox: Fedora generational core is f30
toolbox: base image is fedora-toolbox:30
toolbox: container is testing00
toolbox: checking value /var/run/.heim_org.h5l.kcm-socket (Stream) of property Listen in sssd-kcm.socket
toolbox: parsing value /var/run/.heim_org.h5l.kcm-socket (Stream) of property Listen in sssd-kcm.socket
toolbox: checking if 'podman create' supports --dns=none and --no-hosts
toolbox: 'podman create' supports --dns=none and --no-hosts
toolbox: looking for image localhost/fedora-toolbox:30
toolbox: looking for image registry.fedoraproject.org/f30/fedora-toolbox:30
Image required to create toolbox container.
Download registry.fedoraproject.org/f30/fedora-toolbox:30 (500MB)? [y/N]: y
toolbox: pulling image registry.fedoraproject.org/f30/fedora-toolbox:30
Trying to pull registry.fedoraproject.org/f30/fedora-toolbox:30...Getting image source signatures
Copying blob 4c4fc0337a75 done
Copying blob eb96ef222807 done
Copying config 42cdab313e done
Writing manifest to image destination
Storing signatures
toolbox: base image fedora-toolbox:30 resolved to registry.fedoraproject.org/f30/fedora-toolbox:30
toolbox: checking if container testing00 already exists
toolbox: trying to create container testing00
Created container: testing00
Enter with: toolbox enter --container testing00


[my_user@my_pc toolbox]$ ./toolbox -v enter -c testing00
toolbox: resolved absolute path for ./toolbox to /var/home/my_user/testing/toolbox/toolbox
toolbox: TOOLBOX_PATH is /var/home/my_user/testing/toolbox/toolbox
toolbox: checking if 'podman system migrate' exists
toolbox: migration not needed: 1.3.1 is unchanged
toolbox: Fedora generational core is f30
toolbox: base image is fedora-toolbox:30
toolbox: container is testing00
toolbox: checking if container testing00 exists
toolbox: trying to start container testing00
toolbox: looking for /etc/profile.d/toolbox.sh in container testing00
ERRO[0000] open /var/home/my_user/.local/share/containers/storage/overlay-containers/e301702200509ebec9c24f8daced60e745c5561af9b8c339215c94df4447c91d/userdata/conmon.pid: permission denied 
toolbox: copying /etc/profile.d/toolbox.sh to container testing00
ERRO[0000] open /var/home/my_user/.local/share/containers/storage/overlay-containers/e301702200509ebec9c24f8daced60e745c5561af9b8c339215c94df4447c91d/userdata/conmon.pid: permission denied 
toolbox: unable to copy /etc/profile.d/toolbox.sh to container testing00


[my_user@my_pc toolbox]$ sudo ls -la /var/home/my_user/.local/share/containers/storage/overlay-containers/e301702200509ebec9c24f8daced60e745c5561af9b8c339215c94df4447c91d/userdata/
total 28
drwx------. 3  100000  100000 4096 May 21 23:18 .
drwx--x--x. 3 my_user my_group 4096 May 21 23:18 ..
drwxr-xr-x. 2 my_user my_group 4096 May 21 23:18 artifacts
srwx------. 1 my_user my_group    0 May 21 23:18 attach
-rw-r--r--. 1 my_user my_group 7509 May 21 23:18 config.json
-rw-r--r--. 1 my_user my_group    4 May 21 23:18 conmon.pid
prw-r--r--. 1 my_user my_group    0 May 21 23:18 ctl
-rw-------. 1 my_user my_group  515 May 21 23:18 ctr.log


[my_user@my_pc ~]$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora-workstation:fedora/30/x86_64/silverblue
                   Version: 30.20190521.0 (2019-05-21T00:41:04Z)
                BaseCommit: c77995efc63ab7063d227468c614add3ad901ab98caa26ce4ae36925d2f7a6a0
              GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9
           LayeredPackages: chromium dmidecode gnome-tweaks htop tilix tilix-nautilus tmux

  ostree://fedora-workstation:fedora/30/x86_64/silverblue
                   Version: 30.20190520.0 (2019-05-20T00:34:36Z)
                BaseCommit: 7059cc8508f5d9a00374f43667a4934ab26f001e930c8b7f71e2f309cd8f1ea4
              GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9
           LayeredPackages: chromium dmidecode gnome-tweaks htop tilix tilix-nautilus tmux


[my_user@my_pc ~]$ rpm -q podman
podman-1.3.1-1.git7210727.fc30.x86_64
[my_user@my_pc ~]$

floating-cat · 2019-05-22T06:38:00Z

Actually I fail to enter Toolbox after using podman system migrate in the older Toolbox version (I said this in the last line in my log the first time, but I guess I need to bold that line to highlight this behavior).
Still same to the new Toolbox version, but even worse.

~> rpm -q podman
podman-1.3.1-1.git7210727.fc30.x86_64
~> rpm -q toolbox
toolbox-0.0.10-1.fc30.noarch
~> toolbox -v enter
toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: checking if 'podman system migrate' exists
toolbox: migration not needed: 1.3.1 is unchanged
toolbox: Fedora generational core is f30
toolbox: base image is fedora-toolbox:30
toolbox: container is fedora-toolbox-30
toolbox: checking if container fedora-toolbox-30 exists
toolbox: trying to start container fedora-toolbox-30
toolbox: looking for /etc/profile.d/toolbox.sh in container fedora-toolbox-30
ERRO[0000] open /home/my_name/.local/share/containers/storage/overlay-containers/a33655212b94c4b5eb9b59feec317f89dbc9771d2e0a3c51b573cccce4b1f68d/userdata/conmon.pid: permission denied 
toolbox: copying /etc/profile.d/toolbox.sh to container fedora-toolbox-30
ERRO[0000] open /home/my_name/.local/share/containers/storage/overlay-containers/a33655212b94c4b5eb9b59feec317f89dbc9771d2e0a3c51b573cccce4b1f68d/userdata/conmon.pid: permission denied 
toolbox: unable to copy /etc/profile.d/toolbox.sh to container fedora-toolbox-30

~> podman system migrate
container_linux.go:388: signaling init process caused "operation not permitted"
ERRO[0000] Error removing store for partially-created runtime: A layer is mounted: layer is in use by a container 
Error: error migrating containers: cannot stop container a33655212b94c4b5eb9b59feec317f89dbc9771d2e0a3c51b573cccce4b1f68d: operation not permitted

~> podman stop --all
ERRO[0000] open /home/my_name/.local/share/containers/storage/overlay-containers/a33655212b94c4b5eb9b59feec317f89dbc9771d2e0a3c51b573cccce4b1f68d/userdata/conmon.pid: permission denied

The podman system migrate can't fix my podman now (It works for the old version).

adamkrz · 2019-05-22T06:43:58Z

I have observed the same issue during "toolbox enter" after creating a new container with the toolbox in Fedora Silverblue 30.

debarshiray · 2019-05-22T11:13:25Z

podman system migrate doesn't work when one of the containers is already running.

Either way, I have seen the unable to copy error myself and it goes away on subsequent attempts. Most likely a Podman bug.

floating-cat · 2019-05-22T11:27:59Z

podman system migrate doesn't work when one of the containers is already running.

Either way, I have seen the unable to copy error myself and it goes away on subsequent attempts. Most likely a Podman bug.

I ran podman stop --all before all command I show in my last log if I remember correctly. I will try again tomorrow.
I agree this likely a bug in podman.

edit: Same result I try today.

matthew-a-p · 2019-05-22T13:14:18Z

What is the ownership of the userdata directory for a container that's working as intended?
${HOME}/.local/share/containers/storage/overlay-containers/<HASH>/userdata
I'm trying to understand if this is a creation problem or an run/enter problem. I'm hoping to find additional avenues to troubleshoot this from.

@debarshiray in my case this has nothing to do with the migrate function/feature of podman. it happens with a fresh new container.

toolbox: checking if 'podman system migrate' exists
toolbox: migration not needed: 1.3.1 is unchanged

S4ndyk · 2019-05-23T08:39:45Z

I'm having similar problems. This was done on a freshly installed fedora silverblue 30. Prior to this the only things I had done on the system were rpm-ostree upgrade, rebooting and toolbox create.

[myname@thinkpad ~]$ toolbox -v enter
toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: Fedora generational core is f30
toolbox: base image is fedora-toolbox:30
toolbox: customized user-specific image is fedora-toolbox-myname:30
toolbox: container is fedora-toolbox-myname-30
toolbox: checking if container fedora-toolbox-myname-30 exists
toolbox: container fedora-toolbox-myname-30 was created from image localhost/fedora-toolbox-myname:30
toolbox: checking if image localhost/fedora-toolbox-myname:30 has volumes for host bind mounts
toolbox: trying to start container fedora-toolbox-myname-30
toolbox: creating list of environment variables to forward
toolbox: COLORTERM=truecolor
toolbox: DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
toolbox: DBUS_SYSTEM_BUS_ADDRESS is unset
toolbox: DESKTOP_SESSION=gnome
toolbox: DISPLAY=:0
toolbox: LANG=en_US.UTF-8
toolbox: SHELL=/bin/bash
toolbox: SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
toolbox: TERM=xterm-256color
toolbox: TOOLBOX_PATH is unset
toolbox: VTE_VERSION=5603
toolbox: WAYLAND_DISPLAY=wayland-0
toolbox: XDG_CURRENT_DESKTOP=GNOME
toolbox: XDG_DATA_DIRS=/home/myname/.local/share/flatpak/exports/share/:/var/lib/flatpak/exports/share/:/usr/local/share/:/usr/share/
toolbox: XDG_MENU_PREFIX=gnome-
toolbox: XDG_RUNTIME_DIR=/run/user/1000
toolbox: XDG_SEAT=seat0
toolbox: XDG_SESSION_DESKTOP=gnome
toolbox: XDG_SESSION_ID=2
toolbox: XDG_SESSION_TYPE=wayland
toolbox: XDG_VTNR=2
toolbox: created options for environment variables to forward
--env=COLORTERM=truecolor --env=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus --env=DESKTOP_SESSION=gnome --env=DISPLAY=:0 --env=LANG=en_US.UTF-8 --env=SHELL=/bin/bash --env=SSH_AUTH_SOCK=/run/user/1000/keyring/ssh --env=TERM=xterm-256color --env=VTE_VERSION=5603 --env=WAYLAND_DISPLAY=wayland-0 --env=XDG_CURRENT_DESKTOP=GNOME --env=XDG_DATA_DIRS=/home/myname/.local/share/flatpak/exports/share/:/var/lib/flatpak/exports/share/:/usr/local/share/:/usr/share/ --env=XDG_MENU_PREFIX=gnome- --env=XDG_RUNTIME_DIR=/run/user/1000 --env=XDG_SEAT=seat0 --env=XDG_SESSION_DESKTOP=gnome --env=XDG_SESSION_ID=2 --env=XDG_SESSION_TYPE=wayland --env=XDG_VTNR=2
toolbox: looking for /bin/bash in container fedora-toolbox-myname-30
ERRO[0000] open /var/home/myname/.local/share/containers/storage/overlay-containers/2753373c5df60c4f03f395ce7e7512522bd5f7f7d01068f4af39b0dd40cfb0cc/userdata/conmon.pid: permission denied 
toolbox: /bin/bash not found in fedora-toolbox-myname-30; using /bin/bash instead
toolbox: trying to exec /bin/bash in container fedora-toolbox-myname-30
ERRO[0000] open /var/home/myname/.local/share/containers/storage/overlay-containers/2753373c5df60c4f03f395ce7e7512522bd5f7f7d01068f4af39b0dd40cfb0cc/userdata/conmon.pid: permission denied

w1gz · 2019-05-23T10:29:37Z

I am in the same situation and it seems to be a permission issue. A quick glance at a freshly created container in ~/.local/share/containers/storage/overlay-containers/<container-id> gives us:

drwx------. 3 100000 100000 100 May 23 12:19 userdata
drwx--x--x. 3 <myUser>    <myUser>     22 May 23 12:19 .
drwx--x--x. 3 <myUser>    <myUser>    124 May 23 12:19 ..

For testing purposes only, I recursively chown on that directory (e.g. chown -R myUser:myUser ~/.local/share/containers/storage/overlay-containers).

You can now successfully enter the toolbox container, however, by doing this you have screw up quite a few things (i.e. the container filesystem underneath, which will break sudo & such).

I am not familiar enough with toolbox, podman to debug this any further but please ask me If I can be of any help with logs etc.

ghost · 2019-05-23T11:39:05Z

I am also affected by the same error.

I lookup to podman git and I think this is related to #3187.

debarshiray · 2019-05-23T12:16:30Z

We are conflating a number of unrelated Podman bugs in this issue. None of those are actionable as Toolbox issues, and by now it's been proved that the podman-1.3.1 release had more than its fair share of regressions.

Let's just wait for a new Podman release to come out and see where things stand. In the meantime I'd suggest downgrading to podman-1.2.0.

debarshiray closed this as completed May 21, 2019

floating-cat mentioned this issue May 23, 2019

Running multiple rootless containers in parallel with toolbox-like UID mapping results in "conmon.pid: permission denied" containers/podman#3187

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fail to enter Toolbox #169

Fail to enter Toolbox #169

floating-cat commented May 21, 2019 •

edited

debarshiray commented May 21, 2019

debarshiray commented May 21, 2019

matthew-a-p commented May 22, 2019 •

edited

floating-cat commented May 22, 2019 •

edited

adamkrz commented May 22, 2019 •

edited

debarshiray commented May 22, 2019

floating-cat commented May 22, 2019 •

edited

matthew-a-p commented May 22, 2019

S4ndyk commented May 23, 2019

w1gz commented May 23, 2019 •

edited

ghost commented May 23, 2019 •

edited by ghost

debarshiray commented May 23, 2019

Fail to enter Toolbox #169

Fail to enter Toolbox #169

Comments

floating-cat commented May 21, 2019 • edited

debarshiray commented May 21, 2019

debarshiray commented May 21, 2019

matthew-a-p commented May 22, 2019 • edited

floating-cat commented May 22, 2019 • edited

adamkrz commented May 22, 2019 • edited

debarshiray commented May 22, 2019

floating-cat commented May 22, 2019 • edited

matthew-a-p commented May 22, 2019

S4ndyk commented May 23, 2019

w1gz commented May 23, 2019 • edited

ghost commented May 23, 2019 • edited by ghost

debarshiray commented May 23, 2019

floating-cat commented May 21, 2019 •

edited

matthew-a-p commented May 22, 2019 •

edited

floating-cat commented May 22, 2019 •

edited

adamkrz commented May 22, 2019 •

edited

floating-cat commented May 22, 2019 •

edited

w1gz commented May 23, 2019 •

edited

ghost commented May 23, 2019 •

edited by ghost