Certain log configuration reliably triggers a segfault

[paulgb]

Description

A segfault occurs reliably under these circumstances:

• The fluentd log driver is in use
• The collector address uses TLS
• The collector address cannot be connected to
• The fluentd-async option is provided and is true.

Reproduce

Run this:

docker run --log-driver=fluentd \
  --log-opt fluentd-address=tls://foo:1234 \
  --log-opt fluentd-async=true \
  -t debian:buster-slim echo "anything"

View dockerd logs (e.g. sudo journalctl -u docker.service) and observe that the daemon has been restarted due to a segfault.

Expected behavior

Docker should error/warn, but not segfault.

docker version

Client: Docker Engine - Community
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:17:48 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:17:48 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.25.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 28
  Running: 4
  Paused: 0
  Stopped: 24
 Images: 7
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc runsc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.0-1016-aws
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.857GiB
 Name: ip-172-31-44-70
 ID: 016a89fb-6276-4d6a-b982-c6ee1a692214
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

It may be related to this open issue in fluentd.

Relevant part of my dockerd logs (sudo journalctl -u docker.service):

Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]: [2024-04-11T14:42:00Z] Unable to send logs to fluentd, reconnecting...
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]: panic: runtime error: invalid memory address or nil pointer dereference
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x358 pc=0x6132cc733837]
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]: goroutine 8871 [running]:
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]: crypto/tls.(*Conn).Close(0xc001a3b2c0?)
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]:         /usr/local/go/src/crypto/tls/conn.go:1403 +0x17
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]: github.com/fluent/fluent-logger-golang/fluent.(*Fluent).close(...)
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]:         /root/build-deb/engine/vendor/github.com/fluent/fluent-logger-golang/fluent/fluent.go:430
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]: github.com/fluent/fluent-logger-golang/fluent.(*Fluent).Close(0xc000ff6b40)
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]:         /root/build-deb/engine/vendor/github.com/fluent/fluent-logger-golang/fluent/fluent.go:398 +0xe9
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]: github.com/docker/docker/daemon/logger/fluentd.(*fluentd).Close(0xc00037bce0?)
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]:         /root/build-deb/engine/daemon/logger/fluentd/fluentd.go:133 +0x17
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]: github.com/docker/docker/daemon/logger/loggerutils/cache.(*loggerWithCache).Close(0xc001b17aa0)
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]:         /root/build-deb/engine/daemon/logger/loggerutils/cache/local_cache.go:93 +0x27
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]: github.com/docker/docker/container.(*Container).Reset(0xc00103af00, 0x70?)
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]:         /root/build-deb/engine/container/monitor.go:46 +0x4ad
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]: github.com/docker/docker/daemon.(*Daemon).handleContainerExit(0xc000868500, 0xc00103af00, 0xc001873e50)
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]:         /root/build-deb/engine/daemon/monitor.go:62 +0x4a7
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]: github.com/docker/docker/daemon.(*Daemon).ProcessEvent(0xc000868500, {0xc0026caa80, 0x40}, {0x6132ce0ff9eb, 0x4}, {{0xc0026caa80, 0x40}, {0xc0026cab00, 0x>
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]:         /root/build-deb/engine/daemon/monitor.go:175 +0x72b
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]: github.com/docker/docker/libcontainerd/remote.(*client).processEventStream.(*client).processEvent.func3()
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]:         /root/build-deb/engine/libcontainerd/remote/client.go:561 +0x9d
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]: github.com/docker/docker/libcontainerd/queue.(*Queue).Append.func1()
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]:         /root/build-deb/engine/libcontainerd/queue/queue.go:28 +0x5f
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]: created by github.com/docker/docker/libcontainerd/queue.(*Queue).Append in goroutine 89
Apr 11 14:42:00 ip-172-31-44-70 dockerd[636]:         /root/build-deb/engine/libcontainerd/queue/queue.go:24 +0x1f6
Apr 11 14:42:00 ip-172-31-44-70 systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 11 14:42:00 ip-172-31-44-70 systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 11 14:42:00 ip-172-31-44-70 systemd[1]: docker.service: Consumed 1min 24.472s CPU time.
Apr 11 14:42:02 ip-172-31-44-70 systemd[1]: docker.service: Scheduled restart job, restart counter is at 1.
Apr 11 14:42:02 ip-172-31-44-70 systemd[1]: Stopped Docker Application Container Engine.
Apr 11 14:42:02 ip-172-31-44-70 systemd[1]: docker.service: Consumed 1min 24.472s CPU time.
Apr 11 14:42:02 ip-172-31-44-70 systemd[1]: Starting Docker Application Container Engine...```