Containers remain running after exiting #114
Comments
|
But yes, I'd like to make this properly reference counted, but sadly, I don't know of a way to implement that using the existing Podman command line interface. |
|
Using Perhaps what would could be done is to detect if the container is running when deleting it and then give the user a prompt such as |
|
I am unable to delete a container created by toolbox even after using |
It will fail if you have currently active Otherwise, if that's not the case and you can reproduce at will, then I suggest trying Thanks for stopping by! |
|
Considering it was reported almost 1.5 years ago, I'm wondering if there was any progress since then. |
|
Seems I can't stop the containers left after Toolbox: |
The reason of the error is seems What is it? Is it an error, or it's by design? |
|
|
|
So far, I can think of two different ways to make Toolbox containers reference-counted so that they automatically stop once the last (Note that stopping the container is the same thing as terminating the container's entry point process.)
This can be implemented with Go channels, The downside of this is that it's not resilient against crashes in the
The nice thing about this is that locks are automatically released by the kernel when a process terminates. So, even if the |
This sounds like a Podman bug. If you can repeatedly reproduce this, then I'd suggest filing a Podman bug. It would be even better if you can reproduce this just with Podman commands. eg.,
I think those are UIDs, not PIDs. Those UIDs look big because they are inside a user namespace. |
Currently, once a toolbox container gets started with 'podman start', as part of the 'toolbox enter' command, it doesn't stop unless the host is shut down or someone explicitly calls 'podman stop'. This becomes annoying if someone tries to remove the container because commands like 'podman rm' and such don't work without the '--force' flag, even if all active 'toolbox enter' and 'toolbox run' sessions have terminated. A system of reference counting based on advisory file locks has been used to automatically terminate the container's entry point once all the active sessions have died. The 'toolbox enter' and 'toolbox run' sessions acquire shared file locks, and the container's entry point blocks trying to acquire an exclusive lock on a common file. The entry point will be unblocked once all shared locks have been released by the active sessions, and then it terminates. Once the container has been started, and the entry point has finished setting it up, the entry point waits for a while before trying to acquire its exclusive lock. This is meant to give some time to the first session to go ahead and acquire its shared lock. A duration of 25 seconds, the same interval as the default for D-Bus method calls, was chosen for this. containers#114
Seems I'm not. Not sure if it's good or bad :)
Of course they are UIDs, sorry. Thanks! |
Currently, once a toolbox container gets started with 'podman start', as part of the 'toolbox enter' command, it doesn't stop unless the host is shut down or someone explicitly calls 'podman stop'. This becomes annoying if someone tries to remove the container because commands like 'podman rm' and such don't work without the '--force' flag, even if all active 'toolbox enter' and 'toolbox run' sessions have terminated. A system of reference counting based on advisory file locks has been used to automatically terminate the container's entry point once all the active sessions have died. The 'toolbox enter' and 'toolbox run' sessions acquire shared file locks, and the container's entry point blocks trying to acquire an exclusive lock on a common file. The entry point will be unblocked once all shared locks have been released by the active sessions, and then it terminates. Once the container has been started, and the entry point has finished setting it up, the entry point waits for a while before trying to acquire its exclusive lock. This is meant to give some time to the first session to go ahead and acquire its shared lock. A duration of 25 seconds, the same interval as the default for D-Bus method calls, was chosen for this. containers#114
Currently, once a toolbox container gets started with 'podman start', as part of the 'toolbox enter' command, it doesn't stop unless the host is shut down or someone explicitly calls 'podman stop'. This becomes annoying if someone tries to remove the container because commands like 'podman rm' and such don't work without the '--force' flag, even if all active 'toolbox enter' and 'toolbox run' sessions have terminated. A system of reference counting based on advisory file locks has been used to automatically terminate the container's entry point once all the active sessions have died. The 'toolbox enter' and 'toolbox run' sessions acquire shared file locks, and the container's entry point blocks trying to acquire an exclusive lock on a common file. The entry point will be unblocked once all shared locks have been released by the active sessions, and then it terminates. Once the container has been started, and the entry point has finished setting it up, the entry point waits for a while before trying to acquire its exclusive lock. This is meant to give some time to the first session to go ahead and acquire its shared lock. A duration of 25 seconds, the same interval as the default for D-Bus method calls, was chosen for this. containers#114
I believe |
Another option, used by coreos/toolbox, is to call It's less sophisticated than the other alternatives, but simpler to implement. |
|
@debarshiray I would vote for the simple approach (just always stop after exec) as long as you suppress the spurious "Error: container ... has active exec sessions, refusing to clean up: container state improper"output when container cannot be stopped. |
|
Here's a simple fix that I'm trying out in https://github.com/castedo/cnest. So far seems to be working well. |
|
This is working, it finally kills all toolbox running in background |
Currently, once a toolbox container gets started with 'podman start', as part of the 'toolbox enter' command, it doesn't stop unless the host is shut down or someone explicitly calls 'podman stop'. This becomes annoying if someone tries to remove the container because commands like 'podman rm' and such don't work without the '--force' flag, even if all active 'toolbox enter' and 'toolbox run' sessions have terminated. A crude form of reference counting has been set up that depends on 'podman stop' failing as long there's any active 'podman exec' session left. Every invocation of 'podman exec' in 'enter' and 'run' is followed by 'podman stop', so that the container gets stopped once the last session finishes. While this approach looks very crude at first glance, it does have the advantage of being ridiculously simple to implement. Thus, it's a lot more robust and easier to verify than setting up some custom reference counting or synchronization using other means like POSIX signals or file locks. Based on the implementation in github.com/coreos/toolbox. containers#114
That's about cleaning up any active If so, then that's different from this issue. This issue is about stopping the container (ie., killing the entry point) when the last I think the problem you were trying to address might have been fixed in Podman through containers/podman#17025 |
It turns out that current implementations of |
Interesting. So you are doing: I am worried that there's a race. A |
Good eye! You are correct, there is that possibility. It's fair to say what I coded in cnest for this is a hack. I've been using it for more than a year now. Still working well. But I'm only using it for "nest" containers for which I enter and exit them manually at the command line. I'm not fast enough to be able to create the race condition. My hack might not be OK for more general uses of a container. Maybe there are single-user cases where someone has programs in the background entering/starting the container and not only entering from the command line manually. |
|
I've made a wrapper script to fix this issue (89luca89/distrobox#786) when using distrobox and adapted it for toolbox, pretty simple, it gets the conmon PID on start of the container then kills it afterwards using a background script Could easily be adapted to kill the container too when there is no more shells open #!/usr/bin/env bash
#
# toolbox-enter-wrapper - wrapper to call the shell properly in toolbox
if [ -z "$1" ]; then
echo "Please provide container name"
exit 1
fi
PIDFILE_DIR="$HOME/.local/state/toolbox"
PIDFILE="$PIDFILE_DIR/$$"
mkdir -p "$PIDFILE_DIR"
touch "$PIDFILE"
nohup sh <<EOF >/dev/null 2>&1 &
# wait for the main script to end
while ps -p $$ >/dev/null; do
sleep 1s
done
# get pid from the file
PID="\$(cat "$PIDFILE")"
rm -f "$PIDFILE"
# conmon already dead, quit
if ! ps -p "\$PID" >/dev/null; then
exit 0
fi
# kill conmon
kill -1 "\$PID"
# quit
exit 0
EOF
toolbox run -c "$1" sh -c "echo \$PPID > $PIDFILE; exec ${2:-$SHELL}" |
In particular, this means it's impossible to remove a toolbox-created container without first stopping/killing it with
podman:The text was updated successfully, but these errors were encountered: