[BUG] Compose gets stuck when dealing with failed service that's indirectly depended on by another service

[struffel]

Description

When creating a chain of init containers using using the depends_on: field in a compose file together with condition: service_completed_successfully, I expect docker compose to fail and stop if one container in the chain fails and there are no other dependencies left to meet, i.e. if there are no executable services left in the stack because they all completed or depend on a failed service.

However, having a chain on depends_on: for multiple services can cause docker compose to lock up and not quit, even though there are no containers left to run. At that point, force-quitting by pressing Ctrl+C multiple times is the only way to stop the stack.

This poses a problem for any kind of script or automation that acts on the success/failure of a docker compose command, as it can just get stuck forever.

Steps To Reproduce

Consider the following stack:

version: '3'
name: demo
services:

  step_1:
    image: alpine
    command: sh -c "sleep 2 ; echo 'Failure!' ; exit 1"

  step_2:
    image: alpine
    command: sh -c "sleep 2 ; echo 'Success!' ; exit 0"
    depends_on:
      step_1:
        condition: service_completed_successfully

This stack behaves as expected, quitting back to the console with an error message and leaving step_2 in the Created state:

PS C:\Users\Lennart\Desktop> docker compose up 
[+] Building 0.0s (0/0)
[+] Running 3/3
 ✔ Network demo_default     Created                                                                                                                                                                                                       0.1s 
 ✔ Container demo-step_1-1  Created                                                                                                                                                                                                       0.1s 
 ✔ Container demo-step_2-1  Created                                                                                                                                                                                                       0.1s 
Attaching to demo-step_1-1, demo-step_2-1
demo-step_1-1  | Failure!
service "step_1" didn't complete successfully: exit 1
PS C:\Users\Lennart\Desktop> 

Now expand the stack with one more step:

version: '3'
name: demo
services:

  step_1:
    image: alpine
    command: sh -c "sleep 2 ; echo 'Failure!' ; exit 1"

  step_2:
    image: alpine
    command: sh -c "sleep 2 ; echo 'Success!' ; exit 0"
    depends_on:
      step_1:
        condition: service_completed_successfully

  step_3:
    image: alpine
    command: sh -c "sleep 2 ; echo 'Success!' ; exit 0"
    depends_on:
      step_2:
        condition: service_completed_successfully

This stack never finishes! It reports the failure of step_1 and then does nothing, even after minutes of waiting:

PS C:\Users\Lennart\Desktop> docker compose up
[+] Building 0.0s (0/0)
[+] Running 2/0
 ✔ Container demo-step_1-1  Created                                                                                                                                                                                                       0.0s 
 ✔ Container demo-step_3-1  Created                                                                                                                                                                                                       0.1s 
Attaching to demo-step_1-1, demo-step_2-1, demo-step_3-1
demo-step_1-1  | Failure!
demo-step_1-1 exited with code 1

Pressing Ctrl+C once, causes compose to try to stop all containers. But they are already stopped, because they didn't even start. It just hangs again, requiring multiple presses of CTRL+C to finally quit:

PS C:\Users\Lennart\Desktop> docker compose up
[+] Building 0.0s (0/0)
[+] Running 2/0
 ✔ Container demo-step_1-1  Created                                                                                                                                                                                                       0.0s 
 ✔ Container demo-step_3-1  Created                                                                                                                                                                                                       0.1s 
Attaching to demo-step_1-1, demo-step_2-1, demo-step_3-1
demo-step_1-1  | Failure!
demo-step_1-1 exited with code 1
# PRESSED CTRL+C
Gracefully stopping... (press Ctrl+C again to force)
Aborting on container exit...
[+] Stopping 3/0
 ✔ Container demo-step_3-1  Stopped                                                                                                                                                                                                       0.0s 
 ✔ Container demo-step_2-1  Stopped                                                                                                                                                                                                       0.0s 
 ✔ Container demo-step_1-1  Stopped                                                                                                                                                                                                       0.0s 
# PRESSED AGAIN
no container to kill
# PRESSED AGAIN
time="2023-06-20T19:11:20+02:00" level=error msg="got 3 SIGTERM/SIGINTs, forcing shutdown"
PS C:\Users\Lennart\Desktop> 

Compose Version

PS C:\Users\Lennart\Desktop> docker compose version
Docker Compose version v2.18.1
PS C:\Users\Lennart\Desktop> docker-compose version
Docker Compose version v2.18.1
PS C:\Users\Lennart\Desktop>

Docker Environment

PS C:\Users\Lennart\Desktop> docker info
Client:
 Version:    24.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     C:\Users\Lennart\.docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     C:\Users\Lennart\.docker\cli-plugins\docker-compose.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.19
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scan.exe
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  v0.12.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 90
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.10.102.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 50.16GiB
 Name: docker-desktop
 ID: bdc81f07-086c-4c08-b0f8-1177c783ee5e
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Anything else?

I am using Docker Desktop with WSL2 on Win10 Home.
This issue was apparently introduced in Docker Desktop 4.20.

I tried it on my laptop (Also WSL2, but Win10 Pro) which still had 4.19 and it did not show this behavior before, but developed the same issue after installing the update to 4.20.

[ndeloof]

This has been fixed by #10725, release v2.19.0 includes this fix if you can give it a try and confirm

[struffel]

Thanks! I will try it out as soon as the update to 2.19 ships as part of Docker Desktop.

[jamshid]

Hi @ndeloof can you provide instructions for how to test the updated docker-compose 2.19? I'm currently running the latest Docker for Mac which uses this version:

% docker-compose --version
Docker Compose version v2.18.1

Docker for Mac installs it in /usr/local/bin/. I can replace it with the binary from https://github.com/docker/compose/releases/tag/v2.19.0 but how do I make docker compose use that version?

[ndeloof]

@jamshid download adequate binary from https://github.com/docker/compose/releases and move under ~/.docker/cli-plugins/ as docker-compose (must be set executable)

[FabienBoutantin]

Hello,

I've just tested all the configurations from @struffel and went to the same conclusions. I used my Ubuntu 22.04LTS deployed version (2.18.1).

Then I've checked with v2.19.0, as advertised by @ndeloof and everything is good now.

Regards,

[struffel]

Amazing! I think we can close this then.

[milas]

Closing as fixed, v2.19.0 will also be included with the next Docker Desktop release (v4.21).

Thanks for reporting and confirming the fix!

[jamshid]

I get this error when I try running docker-compose --version after following your instructions:

@jamshid download adequate binary from https://github.com/docker/compose/releases and move under ~/.docker/cli-plugins/ as docker-compose (must be set executable)

“docker-compose” is damaged and can’t be opened. You should move it to the Trash.

The file seems correct:

% pwd
/Users/j/.docker/cli-plugins
% file ./docker-compose
./docker-compose: Mach-O 64-bit executable x86_64

To woarkround this I was able to brew install docker-compose then followed its instructions and that worked:

%  mkdir -p ~/.docker/cli-plugins
% ln -sfn /usr/local/opt/docker-compose/bin/docker-compose ~/.docker/cli-plugins/docker-compose
% docker-compose --version
Docker Compose version 2.19.0
% docker compose version  
Docker Compose version 2.19.0

FWIW the failing github release version (docker-compose-darwin-x86_64) is a little larger than the working brew version:
-rwxr-xr-x@ 1 j DCSW\Domain Users 60695088 Jun 27 16:26 docker-compose
-r-xr-xr-x 1 j admin 60223264 Jun 21 06:02 /usr/local/opt/docker-compose/bin/docker-compose