Docker with BuildKit uses the wrong cache

[enasca]

moby/buildkit#1368

There are two Dockerfiles with the same size and timestamp in this repository:

• https://github.com/kaapana/kaapana/blob/9e206e5688052170791909946db24f3dc6b6fe41/services/monitoring/node-exporter/docker/Dockerfile
• https://github.com/kaapana/kaapana/blob/9e206e5688052170791909946db24f3dc6b6fe41/services/monitoring/alert-manager/docker/Dockerfile

This triggers a subtle bug in BuildKit whereby the same base image is used for both builds. As a result, one image with two tags is produced instead of two different images. The build completes successfully, then one of the two container fails to run when deployed.

This is apparently fixed in Docker Engine 23 (moby/buildkit#1368). I haven't verified. I simply worked around it by disabling BuildKit. Perhaps a check could be added to the build scripts to disable BuildKit if Docker is not recent enough.

[enasca]

Docker Engine v23 indeed fixes the issue.

Reproduction steps for Docker v20:

$ docker version
Client: Docker Engine - Community
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.19.7
 Git commit:        297e128
 Built:             Tue Apr  4 18:21:00 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
[...]

$ docker builder prune -a
WARNING! This will remove all build cache. Are you sure you want to continue? [y/N] y

# Build alert-manager.
$ pushd kaapana/services/monitoring/alert-manager/docker
$ DOCKER_BUILDKIT=1 docker build -t local/alertmanager .
[+] Building 3.0s (5/5) FINISHED                                                                                                                                                                
 => [internal] load build definition from Dockerfile                                                                                                                                       0.0s
 => => transferring dockerfile: 160B                                                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                                            0.0s
 => [internal] load metadata for docker.io/prom/alertmanager:v0.25.0                                                                                                                       1.1s
 => [1/1] FROM docker.io/prom/alertmanager:v0.25.0@sha256:fd4d9a3dd1fd0125108417be21be917f19cc76262347086509a0d43f29b80e98                                                                 1.8s
[...]
 => => writing image sha256:1b8c7fce1ec13b79dbc4a891a82025174aab7963a3e368d30eadb5aab6e2a1cd                                                                                               0.0s

$ popd

# Build node-exporter. Observe that the base image from alertmanager is used.
$ pushd kaapana/services/monitoring/node-exporter/docker
$ DOCKER_BUILDKIT=1 docker build -t local/nodeexporter .
[+] Building 0.8s (5/5) FINISHED                                                                                                                                                                
 => [internal] load build definition from Dockerfile                                                                                                                                       0.0s
 => => transferring dockerfile: 37B                                                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                                            0.0s
 => [internal] load metadata for docker.io/prom/alertmanager:v0.25.0                                                                                                                       0.7s
 => CACHED [1/1] FROM docker.io/prom/alertmanager:v0.25.0@sha256:fd4d9a3dd1fd0125108417be21be917f19cc76262347086509a0d43f29b80e98                                                          0.0s
 => exporting to image                                                                                                                                                                     0.0s
 => => exporting layers                                                                                                                                                                    0.0s
 => => writing image sha256:1b8c7fce1ec13b79dbc4a891a82025174aab7963a3e368d30eadb5aab6e2a1cd                                                                                               0.0s