Cannot upgrade dokku while using Caddy

[spiffytech]

Description of problem

When I apt upgrade dokku, I get this error:

Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xe" for details.
nginx.service is not active, cannot reload.
dpkg: error processing package dokku (--configure):
 installed dokku package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 dokku
E: Sub-process /usr/bin/dpkg returned an error code (1)

Steps to reproduce

1. Switch on Caddy
2. systemctl stop nginx
3. systemctl disable nginx
4. Upgrade Dokku

My workaround has been to do this: apt update && dokku caddy:stop && service nginx start && apt upgrade -y && service nginx stop && systemctl disable nginx && dokku caddy:start

dokku report $APP_NAME

-----> uname: Linux debian-2gb-ash-1 5.10.0-25-amd64 #1 SMP Debian 5.10.191-1 (2023-08-16) x86_64 GNU/Linux
-----> memory: 
                      total        used        free      shared  buff/cache   available
       Mem:            1922         268         216          29        1438        1441
       Swap:              0           0           0
-----> docker version: 
       Client: Docker Engine - Community
        Version:           24.0.6
        API version:       1.43
        Go version:        go1.20.7
        Git commit:        ed223bc
        Built:             Mon Sep  4 12:32:16 2023
        OS/Arch:           linux/amd64
        Context:           default
       
       Server: Docker Engine - Community
        Engine:
         Version:          24.0.6
         API version:      1.43 (minimum version 1.12)
         Go version:       go1.20.7
         Git commit:       1a79695
         Built:            Mon Sep  4 12:32:16 2023
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.6.24
         GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
        runc:
         Version:          1.1.9
         GitCommit:        v1.1.9-0-gccaecfc
        docker-init:
         Version:          0.19.0
         GitCommit:        de40ad0
-----> docker daemon info: 
       Client: Docker Engine - Community
        Version:    24.0.6
        Context:    default
        Debug Mode: true
        Plugins:
         buildx: Docker Buildx (Docker Inc.)
           Version:  v0.11.2
           Path:     /usr/libexec/docker/cli-plugins/docker-buildx
         compose: Docker Compose (Docker Inc.)
           Version:  v2.21.0
           Path:     /usr/libexec/docker/cli-plugins/docker-compose
       
       Server:
        Containers: 2
         Running: 2
         Paused: 0
         Stopped: 0
        Images: 3
        Server Version: 24.0.6
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Using metacopy: false
         Native Overlay Diff: true
         userxattr: false
        Logging Driver: json-file
        Cgroup Driver: systemd
        Cgroup Version: 2
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
        Swarm: inactive
        Runtimes: io.containerd.runc.v2 runc
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
        runc version: v1.1.9-0-gccaecfc
        init version: de40ad0
        Security Options:
         apparmor
         seccomp
          Profile: builtin
         cgroupns
        Kernel Version: 5.10.0-25-amd64
        Operating System: Debian GNU/Linux 11 (bullseye)
        OSType: linux
        Architecture: x86_64
        CPUs: 2
        Total Memory: 1.878GiB
        Name: debian-2gb-ash-1
        ID: 485daf43-e7e8-43eb-8baf-93c181333259
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
         File Descriptors: 37
         Goroutines: 50
         System Time: 2023-10-17T14:10:23.414637108Z
         EventsListeners: 1
        Experimental: false
        Insecure Registries:
         127.0.0.0/8
        Live Restore Enabled: false
       
-----> herokuish version: 
 !     Herokuish image gliderlabs/herokuish:latest-22 is not available
-----> dokku version: dokku version 0.32.0
 !     dokku-event-listener binary is not available
 !     dokku-update binary is not available
-----> docker-container-healthchecker version: 0.7.2                                                                           
-----> docker-image-labeler version: 0.6.1build+c6e15a9                                                              
-----> git version: git version 2.30.2
-----> lambda-builder version:        0.5.0                                                                           
-----> netrc version: 0.7.1build+677c033
 !     pack binary is not available
-----> plugn version: plugn: 0.13.0build+fd5297a
-----> sigil version: 0.10.1build+e443be0
-----> sshcommand version: sshcommand 0.17.1
-----> dokku plugins: 
         00_dokku-standard    0.32.0 enabled    dokku core standard plugin
         20_events            0.32.0 enabled    dokku core events logging plugin
         app-json             0.32.0 enabled    dokku core app-json plugin
         apps                 0.32.0 enabled    dokku core apps plugin
         builder              0.32.0 enabled    dokku core builder plugin
         builder-dockerfile   0.32.0 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.32.0 enabled    dokku core builder-herokuish plugin
         builder-lambda       0.32.0 enabled    dokku core builder-lambda plugin
         builder-nixpacks     0.32.0 enabled    dokku core builder-nixpacks plugin
         builder-null         0.32.0 enabled    dokku core builder-null plugin
         builder-pack         0.32.0 enabled    dokku core builder-pack plugin
         buildpacks           0.32.0 enabled    dokku core buildpacks plugin
         caddy-vhosts         0.32.0 enabled    dokku core caddy-vhosts plugin
         certs                0.32.0 enabled    dokku core certificate management plugin
         checks               0.32.0 enabled    dokku core checks plugin
         common               0.32.0 enabled    dokku core common plugin
         config               0.32.0 enabled    dokku core config plugin
         cron                 0.32.0 enabled    dokku core cron plugin
         docker-options       0.32.0 enabled    dokku core docker-options plugin
         domains              0.32.0 enabled    dokku core domains plugin
         enter                0.32.0 enabled    dokku core enter plugin
         git                  0.32.0 enabled    dokku core git plugin
         haproxy-vhosts       0.32.0 enabled    dokku core haproxy-vhosts plugin
         logs                 0.32.0 enabled    dokku core logs plugin
         network              0.32.0 enabled    dokku core network plugin
         nginx-vhosts         0.32.0 enabled    dokku core nginx-vhosts plugin
         openresty-vhosts     0.32.0 enabled    dokku core openresty-vhosts plugin
         plugin               0.32.0 enabled    dokku core plugin plugin
         ports                0.32.0 enabled    dokku core ports plugin
         proxy                0.32.0 enabled    dokku core proxy plugin
         ps                   0.32.0 enabled    dokku core ps plugin
         registry             0.32.0 enabled    dokku core registry plugin
         repo                 0.32.0 enabled    dokku core repo plugin
         resource             0.32.0 enabled    dokku core resource plugin
         run                  0.32.0 enabled    dokku core run plugin
         scheduler            0.32.0 enabled    dokku core scheduler plugin
         scheduler-docker-local 0.32.0 enabled    dokku core scheduler-docker-local plugin
         scheduler-null       0.32.0 enabled    dokku core scheduler-null plugin
         shell                0.32.0 enabled    dokku core shell plugin
         ssh-keys             0.32.0 enabled    dokku core ssh-keys plugin
         storage              0.32.0 enabled    dokku core storage plugin
         trace                0.32.0 enabled    dokku core trace plugin
         traefik-vhosts       0.32.0 enabled    dokku core traefik-vhosts plugin
=====> grav app-json information
       App json computed selected:    app.json
       App json global selected:      app.json
       App json selected:             
=====> grav app information
       App created at:                1697549554
       App deploy source:             docker-image
       App deploy source metadata:    linuxserver/grav:latest
       App dir:                       /home/dokku/grav
       App locked:                    false
=====> grav builder information
       Builder build dir:             
       Builder computed build dir:    
       Builder computed selected:     
       Builder global build dir:      
       Builder global selected:       
       Builder selected:              
=====> grav builder-dockerfile information
       Builder dockerfile computed dockerfile path: Dockerfile               
       Builder dockerfile global dockerfile path: Dockerfile               
       Builder dockerfile dockerfile path:                          
=====> grav builder-herokuish information
       Builder herokuish computed allowed: true                     
       Builder herokuish global allowed: true                     
       Builder herokuish allowed:                              
=====> grav builder-lambda information
       Builder lambda computed lambdayml path: lambda.yml               
       Builder lambda global lambdayml path: lambda.yml               
       Builder lambda lambdayml path:                          
=====> grav builder-nixpacks information
       Builder nixpacks computed nixpackstoml path: nixpacks.toml            
       Builder nixpacks global nixpackstoml path: nixpacks.toml            
       Builder nixpacks nixpackstoml path:                          
       Builder nixpacks computed no cache: false                    
       Builder nixpacks global no cache: false                    
       Builder nixpacks no cache:                              
=====> grav builder-pack information
       Builder pack computed projecttoml path: project.toml             
       Builder pack global projecttoml path: project.toml             
       Builder pack projecttoml path:                          
=====> grav buildpacks information
       Buildpacks computed stack:     gliderlabs/herokuish:latest-22
       Buildpacks global stack:       
       Buildpacks list:               
       Buildpacks stack:              
=====> grav caddy information
       Caddy image:                   lucaslorentz/caddy-docker-proxy:2.8
       Caddy letsencrypt email:       letsencrypt@spiffy.tech  
       Caddy letsencrypt server:      https://acme-v02.api.letsencrypt.org/directory
       Caddy log level:               ERROR                    
       Caddy polling interval:        5s                       
       Caddy tls internal:            false                    
=====> grav ssl information
       Ssl dir:                       /home/dokku/grav/tls     
       Ssl enabled:                   false                    
       Ssl hostnames:                                          
       Ssl expires at:                                         
       Ssl issuer:                                             
       Ssl starts at:                                          
       Ssl subject:                                            
       Ssl verified:                                           
=====> grav checks information
       Checks disabled list:          none                     
       Checks skipped list:           none                     
       Checks computed wait to retire: 60                       
       Checks global wait to retire:  60                       
       Checks wait to retire:                                  
=====> grav docker options information
       Docker options build:                                   
       Docker options deploy:         --restart=on-failure:10  
       Docker options run:                                     
=====> grav domains information
       Domains app enabled:           true                     
       Domains app vhosts:            gravtest-1.spiffy.tech   
       Domains global enabled:        false                    
       Domains global vhosts:                                  
=====> grav git information
       Git deploy branch:             master                   
       Git global deploy branch:      master                   
       Git keep git dir:              false                    
       Git rev env var:               GIT_REV                  
       Git sha:                       3542561                  
       Git source image:              linuxserver/grav:latest  
       Git last updated at:           1697549841               
=====> grav haproxy information
       Haproxy image:                 byjg/easy-haproxy:4.3.0  
       Haproxy letsencrypt email:                              
       Haproxy letsencrypt server:    https://acme-v02.api.letsencrypt.org/directory
       Haproxy log level:             ERROR                    
=====> grav logs information
       Logs computed max size:        10m
       Logs global max size:          10m
       Logs global vector sink:       
       Logs max size:                 
       Logs vector sink:              
=====> grav network information
       Network attach post create:           
       Network attach post deploy:           
       Network bind all interfaces:          false
       Network computed attach post create:  
       Network computed attach post deploy:  
       Network computed bind all interfaces: false
       Network computed initial network:     
       Network computed tld:                 
       Network global attach post create:    
       Network global attach post deploy:    
       Network global bind all interfaces:   false
       Network global initial network:       
       Network global tld:                   
       Network initial network:              
       Network static web listener:          
       Network tld:                          
       Network web listeners:                172.17.0.3:443
=====> grav nginx information
       Nginx access log format:                                
       Nginx access log path:         /var/log/nginx/grav-access.log
       Nginx bind address ipv4:                                
       Nginx bind address ipv6:       ::                       
       Nginx client max body size:                             
       Nginx disable custom config:   false                    
       Nginx error log path:          /var/log/nginx/grav-error.log
       Nginx global hsts:             true                     
       Nginx computed hsts:           true                     
       Nginx hsts:                                             
       Nginx hsts include subdomains: true                     
       Nginx hsts max age:            15724800                 
       Nginx hsts preload:            false                    
       Nginx computed nginx conf sigil path: nginx.conf.sigil         
       Nginx global nginx conf sigil path: nginx.conf.sigil         
       Nginx nginx conf sigil path:                            
       Nginx proxy buffer size:       4096                     
       Nginx proxy buffering:         on                       
       Nginx proxy buffers:           8 4096                   
       Nginx proxy busy buffers size: 8192                     
       Nginx proxy read timeout:      60s                      
       Nginx last visited at:                                  
       Nginx x forwarded for value:   $remote_addr             
       Nginx x forwarded port value:  $server_port             
       Nginx x forwarded proto value: $scheme                  
       Nginx x forwarded ssl:                                  
=====> grav openresty information
       Openresty access log format:                            
       Openresty access log path:     /var/log/nginx/grav-access.log
       Openresty bind address ipv4:                            
       Openresty bind address ipv6:   ::                       
       Openresty client max body size:                          
       Openresty error log path:      /var/log/nginx/grav-error.log
       Openresty global hsts:         true                     
       Openresty computed hsts:       true                     
       Openresty hsts:                                         
       Openresty hsts include subdomains: true                     
       Openresty hsts max age:        15724800                 
       Openresty hsts preload:        false                    
       Openresty image:               dokku/openresty-docker-proxy:0.5.6
       Openresty letsencrypt email:                            
       Openresty letsencrypt server:  https://acme-v02.api.letsencrypt.org/directory
       Openresty proxy buffer size:   4096                     
       Openresty proxy buffering:     on                       
       Openresty proxy buffers:       8 4096                   
       Openresty proxy busy buffers size: 8192                     
       Openresty proxy read timeout:  60s                      
       Openresty x forwarded for value: $remote_addr             
       Openresty x forwarded port value: $server_port             
       Openresty x forwarded proto value: $scheme                  
       Openresty x forwarded ssl:                              
=====> grav ports information
       Ports map:                     
       Ports map detected:            http:443:443 http:80:80 https:443:80
=====> grav proxy information
       Proxy enabled:                 true
       Proxy type:                    caddy
=====> grav ps information
       Deployed:                      true
       Processes:                     1
       Ps can scale:                  true
       Ps computed procfile path:     Procfile
       Ps global procfile path:       Procfile
       Ps procfile path:              
       Ps restart policy:             on-failure:10
       Restore:                       true
       Running:                       true
       Status web 1:                  running (CID: 5bdc1019167)
=====> grav registry information
       Registry computed image repo:      dokku/grav
       Registry computed push on release: false
       Registry computed server:          
       Registry global push on release:   
       Registry global server:            
       Registry image repo:               
       Registry push on release:          
       Registry server:                   
       Registry tag version:              
=====> grav resource information
=====> grav scheduler information
       Scheduler computed selected:   docker-local
       Scheduler global selected:     docker-local
       Scheduler selected:            
=====> grav scheduler-docker-local information
       Scheduler docker local disable chown:                          
       Scheduler docker local init process: true                     
       Scheduler docker local parallel schedule count:                          
=====> grav storage information
       Storage build mounts:                                   
       Storage deploy mounts:                                  
       Storage run mounts:                                     
=====> grav traefik information
       Traefik api enabled:           false                    
       Traefik api vhost:             traefik.dokku.me         
       Traefik basic auth password:                            
       Traefik basic auth username:                            
       Traefik dashboard enabled:     false                    
       Traefik image:                 traefik:v2.10            
       Traefik letsencrypt email:                              
       Traefik letsencrypt server:    https://acme-v02.api.letsencrypt.org/directory
       Traefik log level:             ERROR                    
       Traefik priority:                                       

Additional information

No response

Output of failing commands after running: dokku trace:off

No response

Output of failing commands after running: dokku trace:on

No response

[josegonzalez]

What version did you upgrade from, and which did you upgrade to? Do you have any apps where the proxy is set to nginx?

[spiffytech]

I've been experiencing this for several months.

This morning I reproduced it on a clean machine, upgrading from 0.31.5 to 0.32.0., on Debian 11.

I'd only created one app on that machine, and I'd configured it to use Caddy. I've confirmed on one of my servers experiencing the issue that all apps have DOKKU_APP_PROXY_TYPE: caddy.

[josegonzalez]

Awesome, that's enough for me to repro and fix in the next patch tonight.

In the future, please feel more than free to file tickets around things that bother you as it helps me improve Dokku.