Enable dockerfile-based image build logs with buildkit

[andrew-demb]

I tried to use the mount cache but get some problems while using it.

Dockerfile

# ...
COPY package.json yarn.lock ./

RUN --mount=type=cache,mode=0777,target=/root/.yarn yarn install --immutable --mode skip-build --frozen-lockfile
# ...

Error:

Error: phase build on image back stage dockerfile handler failed: failed to build image for stage dockerfile with digest 1bce8b66c3ccea0e1cb5a772ca54a12892eaf8bd5925535550865de1: error building dockerfile with local-docker-server: Status: the --mount option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled, Code: 1

After visiting https://docs.docker.com/go/buildkit/ I found that I need to enable it with env variable: DOCKER_BUILDKIT=1

After that my build going to be run, but there is no logs of build and warning message in logs refers to the closed issue in docker/cli:

┌ ⛵ image back
│ WARNING: There is no way to ignore the Dockerfile due to docker limitation when building an image for a compressed context that reads from STDIN.
│ WARNING: To hide this message, remove the Dockerfile ignore rule or add an exception rule.
│ ┌ Building stage back/dockerfile
│ │ back/dockerfile  WARNING: BuildKit output in background tasks is not supported (--quiet) due to https://github.com/docker/cli/issues/2889
│ │ back/dockerfile  sha256:527c3a5e720e8fc2ee183a1ad0eb03cb437391c34ce850a96074928136f03386
│ │ ┌ Store stage into .../back
│ │ └ Store stage into .../back (11.85 seconds)
│ ├ Info
│ │      name: .../back:1bce8b66c3ccea0e1cb5a772ca54a12892eaf8bd5925535550865de1-1681832320278
│ │        id: 527c3a5e720e
│ │   created: 2023-04-18 18:38:38 +0300 EEST
│ │      size: 55.5 MiB
│ └ Building stage back/dockerfile (379.20 seconds)
└ ⛵ image back (382.98 seconds)

Issues I found while searching for source of problem:

• Improve mount logic: allow safe shared mounts between builds #919
• Support of RUN --mount flags #1767
• Is Dockerfile.dockerignore supported? #3460
• [dockerfile] The BuildKit support with explicit user activation #3367

---

The question is: can werf provide a bit more detail in warning about the required steps to use BuilKit with enabled build logs?

---

Werf:

werf version
v1.2.186

Docker:

$ docker -v                                 
Docker version 23.0.4, build f480fb1

[andrew-demb]

While emulating failed build, I found that the build log will be displayed, but in the success case, it's not displayed.

It is intended or a tech limitation?

# success build emulation - build logs not displayed
werf build --repo="..." app
Version: v1.2.214+fix3
Using werf config render file: /tmp/werf-config-render-1643290864

WARNING: There is no way to ignore the Dockerfile due to docker limitation when building an image for a compressed context that reads from STDIN.
WARNING: To hide this message, remove the Dockerfile ignore rule or add an exception rule.
┌ Concurrent builds plan (no more than 5 images at the same time)
│ Set #0:
│ - ⛵ image app
└ Concurrent builds plan (no more than 5 images at the same time)

┌ ⛵ image app
│ WARNING: There is no way to ignore the Dockerfile due to docker limitation when building an image for a compressed context that reads from STDIN.
│ WARNING: To hide this message, remove the Dockerfile ignore rule or add an exception rule.
│ ┌ Building stage app/dockerfile
│ │ app/dockerfile  WARNING: BuildKit output in background tasks is not supported (--quiet) due to https://github.com/docker/cli/issues/2889
│ │ app/dockerfile  sha256:fd0b5cf2102807676c56a63c17ecaf01137411f59dfff232b563aa45053c905d
│ │ ┌ Store stage into ...
│ │ └ Store stage into ... (11.51 seconds)
│ ├ Info
│ │      name: ...:6e0ea8b32b749fb75637081eba676a568541dab64616c3b312842b38-1681889874643
│ │        id: fd0b5cf21028
│ │   created: 2023-04-19 10:37:53 +0300 EEST
│ │      size: 55.5 MiB
│ └ Building stage app/dockerfile (259.39 seconds)
└ ⛵ image app (264.38 seconds)

Running time 265.58 seconds

# failed build emulation - build logs displayed
$ werf build --repo="..." app
Version: v1.2.214+fix3
Using werf config render file: /tmp/werf-config-render-1443257149

┌ Concurrent builds plan (no more than 5 images at the same time)
│ Set #0:
│ - ⛵ image app
└ Concurrent builds plan (no more than 5 images at the same time)

┌ ⛵ image app
│ ┌ Building stage app/dockerfile
│ │ app/dockerfile  WARNING: BuildKit output in background tasks is not supported (--quiet) due to https://github.com/docker/cli/issues/2889
[+] Building -0.2s (15/23)                                                                                                                                                                                         
 => CACHED [internal] load remote build context                                                                                                                                                               0.0s
 => CACHED copy /context /                                                                                                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/node:16-alpine                                                                                                                                             0.0s
 => [internal] load metadata for docker.io/library/node:16-slim                                                                                                                                               0.0s
 => CACHED [internal] settings cache mount permissions                                                                                                                                                        0.0s
 => [runner 1/9] FROM docker.io/library/node:16-alpine                                                                                                                                                        0.0s
 => [builder 1/9] FROM docker.io/library/node:16-slim                                                                                                                                                         0.0s
 => CACHED [builder 2/9] WORKDIR /app                                                                                                                                                                         0.0s
 => CACHED [builder 3/9] COPY .yarnrc.yml ./                                                                                                                                                                  0.0s
 => CACHED [builder 4/9] COPY .yarn ./.yarn                                                                                                                                                                   0.0s
 => CACHED [builder 5/9] COPY package.json yarn.lock ./                                                                                                                                                       0.0s
 => CACHED [builder 6/9] RUN yarn -v                                                                                                                                                                          0.0s
 => CACHED [builder 7/9] RUN --mount=type=cache,mode=0777,target=/root/.yarn yarn install --immutable --mode skip-build --frozen-lockfile                                                                     0.0s
 => CACHED [builder 8/9] COPY . .                                                                                                                                                                             0.0s
 => ERROR [builder 9/9] RUN --mount=type=cache,mode=0777,target=/app/.yarn/cache yarn build                                                                                                                  71.7s
------
 > [builder 9/9] RUN --mount=type=cache,mode=0777,target=/app/.yarn/cache yarn build:
#18 2.741 warn  - You have enabled experimental feature (modularizeImports) in next.config.js.
#18 2.741 warn  - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use at your own risk.
#18 2.741 
#18 2.742 info  - SWC minify release candidate enabled. https://nextjs.link/swcmin
#18 2.745 warn  - No build cache found. Please configure build caching for faster rebuilds. Read more: https://nextjs.org/docs/messages/no-cache
#18 2.890 info  - Linting and checking validity of types...
#18 3.214 warn  - No ESLint configuration detected. Run next lint to begin setup
#18 70.41 info  - Creating an optimized production build...
#18 71.45 Browserslist: caniuse-lite is outdated. Please run:
#18 71.45   npx browserslist@latest --update-db
#18 71.45   Why you should do it regularly: https://github.com/browserslist/browserslist#browsers-data-updating
#18 71.65 
#18 71.65 > Build error occurred
#18 71.65 asd
------
│ ├ Info
│ └ Building stage app/dockerfile (71.91 seconds) FAILED
└ ⛵ image app (72.28 seconds) FAILED

Running time 72.73 seconds
Error: phase build on image app stage dockerfile handler failed: failed to build image for stage dockerfile with digest 7cac189f4d7f762439839d8c02125e6f4ee6756f6e107c395d0c7218: error building dockerfile with local-docker-server: Status: process "/bin/sh -c yarn build" did not complete successfully: exit code: 1, Code: 1

[ilya-lesikov]

Hi, we need to look into this.

[distorhead]

There was a technical limitation with parallel building of images using buildkit. When embedding buildkit it does not allow customizing stdout stream, which would certainly break parallel builds output. That's why buildkit works in quiet mode when building several images. Though buildkit output should work as usual when building only one image, or when parallel mode is disabled.

[distorhead]

There was a technical limitation with parallel building of images using buildkit. When embedding buildkit it does not allow customizing stdout stream, which would certainly break parallel builds output. That's why buildkit works in quiet mode when building several images. Though buildkit output should work as usual when building only one image, or when parallel mode is disabled.

This warning in the log says just that:

│ │ back/dockerfile  WARNING: BuildKit output in background tasks is not supported (--quiet) due to https://github.com/docker/cli/issues/2889

This issue docker/cli#2889 was closed in favor of docker/cli#4056

[andrew-demb]

Thank you.
So there is no way to improve it for now because upstream is not fixed to be fully ready to be used as a library.

One question remains - should the build log always be displayed at the end of the build execution (ref: #5546 (comment))?

[alexey-igrychev]

Thank you. So there is no way to improve it for now because upstream needs to be fixed to be fully ready to be used as a library.

One question remains - should the build log always be displayed at the end of the build execution (ref: #5546 (comment))?

No, it should not be due to limitations of buildkit.
Only foreground tasks have a log; background tasks are executed in quiet mode.