build and push Docker image based on Chainguard base image #13137

v1v · 2024-05-10T09:59:33Z

Motivation/summary

Release two flavours of Docker images:

the one we always do
the one based on chainguard from Chainguard

Please note that we are going to preserve the current Dockerfile, so that users will still be able to build their own custom images based on ubuntu: this is needed because docker.elastic.co/wolfi/* is not a public base image, so docker build would fail for unauthenticated users.

Implementation details

I changed the Makefiles to support different Dockerfiles when running the docker build generation.
I added a tag for helping testing this in GitHub actions and then be able to push the docker images to an internal docker registry.
I fixed some warning regarding missing files when running the go build in the docker.

Checklist

Update CHANGELOG.asciidoc
Documentation has been updated

For functional changes, consider:

Is it observable through the addition of either logging or metrics?
Is its use being published in telemetry to enable product improvement?
Have system tests been added to avoid regression?

How to test these changes

CI

Will create the docker images automatically when working on a feature branch or main/releases

docker images docker.elastic.co/observability-ci/apm-server-internal
REPOSITORY                                               TAG                     IMAGE ID       CREATED          SIZE
docker.elastic.co/observability-ci/apm-server-internal   8.15.0-SNAPSHOT-wolfi   dce4529d1383   11 minutes ago   126MB
docker.elastic.co/observability-ci/apm-server-internal   8.15.0-SNAPSHOT         d2e84aceb911   12 minutes ago   209MB

Locally

$ make package-snapshot

TBC what kind of tests should be done and how.

Related issues

#13139 will help with testing these changes in Buildkite without the need to be merged.
#12671 is also another in-progress task

this will help with pushing docker images to the internal docker registry and test this out

…ny issues

packaging.mk

v1v · 2024-05-10T10:25:12Z

packaging.mk

@@ -186,7 +196,10 @@ build/dependencies-$(APM_SERVER_VERSION)-SNAPSHOT.csv: build/dependencies-$(APM_
 	cp $< $@

 package-docker: $(DOCKER_IMAGE_RELEASE_TARBALLS)
+	@echo ">> $(DOCKER_IMAGE_RELEASE_TARBALLS)"


just to help with printing what it did

v1v · 2024-05-10T10:25:16Z

packaging.mk

 package-docker-snapshot: $(DOCKER_IMAGE_SNAPSHOT_TARBALLS)
+	@echo ">> $(DOCKER_IMAGE_SNAPSHOT_TARBALLS)"


just to help with printing what it did

v1v · 2024-05-10T10:26:18Z

packaging.mk

+		--tag $(INTERNAL_DOCKER_IMAGE):$(VERSION)$(if $(findstring arm64,$(GOARCH)),-arm64)$(if $(findstring wolfi,$(@)),-wolfi) \
+		-f packaging/docker/Dockerfile .
+
+# Docker image tarballs. We distribute UBI Docker images only for AMD64.


docs: no ubi8 but ubi, #11977

v1v · 2024-05-10T10:28:23Z

packaging.mk


 build/docker/%.txt: DOCKER_IMAGE_TAG := docker.elastic.co/apm/apm-server:%
 build/docker/%.txt: VERSION := $(APM_SERVER_VERSION)
 build/docker/%-SNAPSHOT.txt: VERSION := $(APM_SERVER_VERSION)-SNAPSHOT
 build/docker/apm-server-ubi-%.txt: DOCKER_BUILD_ARGS+=--build-arg BASE_IMAGE=docker.elastic.co/ubi9/ubi-minimal
+build/docker/apm-server-wolfi-%.txt: DOCKER_BUILD_ARGS+=--build-arg BASE_IMAGE=docker.elastic.co/wolfi/chainguard-base:20230214 --build-arg BASE_GOLANG_IMAGE=docker.elastic.co/wolfi/go
+
+INTERNAL_DOCKER_IMAGE := docker.elastic.co/observability-ci/apm-server-internal


This should help create docker images and publish them to our internal docker registry - this is handy for validating changes before anything related to the DRA and in GitHub actions.

We are safe since these docker images won't be public available for the time being:

$ docker pull docker.elastic.co/observability-ci/apm-server-internal:8.15.0-SNAPSHOT-wolfi Error response from daemon: pull access denied for docker.elastic.co/observability-ci/apm-server-internal, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

v1v · 2024-05-10T10:28:44Z

packaging/docker/Dockerfile

 WORKDIR /src
-COPY go.mod go.sum /src/
+COPY go.mod go.sum .go-version /src/


Fixes a warning when building the docker image

v1v · 2024-05-10T10:28:48Z

packaging/docker/Dockerfile

 COPY internal/glog/go.mod /src/internal/glog/go.mod
 RUN --mount=type=cache,target=/go/pkg/mod go mod download
 COPY Makefile *.mk /src
 COPY cmd /src/cmd
 COPY internal /src/internal
 COPY x-pack /src/x-pack
 COPY .git /src/.git
+COPY script /src/script


Fixes a warning when building the docker image

v1v · 2024-05-10T10:29:26Z

packaging/docker/Dockerfile


-ENV GOROOT /usr/local/go


Unrequired for building the wolfi version and in fact it's not needed for the traditional docker image

v1v · 2024-05-10T10:30:06Z

.github/workflows/ci.yml

      - run: make package-snapshot
+        env:
+          GH_TOKEN: ${{ github.token }}


Remove some warnings in the github logs, regarding missing variable when running gh in the GitHub runner

v1v · 2024-05-10T10:30:45Z

.github/workflows/ci.yml

+          url: ${{ secrets.VAULT_ADDR }}
+          roleId: ${{ secrets.VAULT_ROLE_ID }}
+          secretId: ${{ secrets.VAULT_SECRET_ID }}
+        if: |


Only when working on branches in the upstream or PRs coming from the upstream then login, GitHub secrets are not available on forked PRs.

.github/workflows/ci.yml

kruskall

we should use chainguard static image as apm-server is a static go binary now

v1v · 2024-05-15T07:34:17Z

we should use chainguard static image as apm-server is a static go binary now

Would you agree that changing this PR to support a vanilla Dockerfile.<TBD> will be enough? - as long as, the Dockerfile.<TBD> is implemented in #12671

.<TBD> -> can be chainguard, minimal, static ... To Be Defined

Changes will be about:

Validating docker image generation on GitHub actions
Supporting docker image generate in Buildkite.
Supporting conditional docker generation - to avoid external contributors having failures because docker base images are not available by default.

kruskall · 2024-05-17T03:28:39Z

Would you agree that changing this PR to support a vanilla Dockerfile. will be enough? - as long as, the Dockerfile. is implemented in #12671

That works 😄

v1v · 2024-05-17T07:35:27Z

packaging/docker/Dockerfile

 COPY internal/glog/go.mod /src/internal/glog/go.mod
 RUN --mount=type=cache,target=/go/pkg/mod go mod download
-COPY Makefile *.mk /src
+COPY Makefile *.mk /src/


fixed linting, when copy multiple files then the target should end with /

v1v · 2024-05-23T16:34:45Z

I think I need to enable some docker login in Buildkite... (I'll disable the docker generation for Buildkite for the time being) while I work on that in a separated PR, sorry @kruskall , I need your review in 1 minutes

0db7be1 is the one disabling it

.buildkite/package.yml

kruskall · 2024-05-24T12:28:56Z

@v1v can you sign the commits ?

(cherry picked from commit b070b3c)

…13225) (cherry picked from commit b070b3c) Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

v1v added 5 commits May 10, 2024 11:32

support wolfi

7d0feee

Use one Dockerfile with args to configure wolfi

a89834b

tag docker images

369c5be

ci: push docker images for no forked PRs

a1ea9f8

this will help with pushing docker images to the internal docker registry and test this out

avoid failing for the time being just in case parallel build scause a…

b7527f1

…ny issues

v1v self-assigned this May 10, 2024

v1v added the backport-8.14 Automated backport with mergify label May 10, 2024

login beforehand

de52938

v1v commented May 10, 2024

View reviewed changes

packaging.mk Outdated Show resolved Hide resolved

v1v commented May 10, 2024

View reviewed changes

ci: enable wolfi docker image generation with GENERATE_WOLFI_IMAGES

c30bbfd

v1v commented May 10, 2024

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

mergify bot mentioned this pull request May 10, 2024

test conditionals #13138

Draft

v1v added 3 commits May 10, 2024 13:01

simplify logic ci

569a5b9

set env and rename jobs

de225d7

fix

1bb262f

v1v marked this pull request as ready for review May 10, 2024 12:10

v1v requested a review from a team as a code owner May 10, 2024 12:10

v1v requested a review from a team May 10, 2024 12:11

reakaleek previously approved these changes May 14, 2024

View reviewed changes

kruskall requested changes May 14, 2024

View reviewed changes

use placeholder for the chainguard

8509876

v1v dismissed reakaleek’s stale review via 8509876 May 15, 2024 16:37

v1v marked this pull request as draft May 15, 2024 16:39

fix duplicated

e403623

v1v marked this pull request as ready for review May 15, 2024 16:51

v1v requested a review from kruskall May 15, 2024 16:51

kruskall approved these changes May 17, 2024

View reviewed changes

v1v commented May 17, 2024

View reviewed changes

kruskall previously approved these changes May 23, 2024

View reviewed changes

v1v commented May 23, 2024

View reviewed changes

.buildkite/package.yml Outdated Show resolved Hide resolved

Update .buildkite/package.yml

f1fe3f9

v1v dismissed kruskall’s stale review via f1fe3f9 May 23, 2024 16:35

v1v commented May 23, 2024

View reviewed changes

.buildkite/package.yml Outdated Show resolved Hide resolved

Update .buildkite/package.yml

0db7be1

kruskall approved these changes May 23, 2024

View reviewed changes

v1v mentioned this pull request May 23, 2024

buildkite(dra): support list for feature branches #13139

Merged

2 tasks

Merge branch 'main' into feature/docker-images-wolfi

88fcc8b

v1v enabled auto-merge (squash) May 24, 2024 13:03

v1v merged commit b070b3c into main May 24, 2024
11 checks passed

v1v deleted the feature/docker-images-wolfi branch May 24, 2024 13:03

mergify bot pushed a commit that referenced this pull request May 24, 2024

build and push Docker image based on Chainguard base image (#13137)

b023d18

(cherry picked from commit b070b3c)

mergify bot mentioned this pull request May 24, 2024

[8.14] build and push Docker image based on Chainguard base image (backport #13137) #13225

Merged

2 tasks

mergify bot added a commit that referenced this pull request May 24, 2024

build and push Docker image based on Chainguard base image (#13137) (#…

26c4223

…13225) (cherry picked from commit b070b3c) Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build and push Docker image based on Chainguard base image #13137

build and push Docker image based on Chainguard base image #13137

v1v commented May 10, 2024 •

edited

v1v May 10, 2024

v1v May 10, 2024

v1v May 10, 2024

v1v May 10, 2024

v1v May 10, 2024

v1v May 10, 2024

v1v May 10, 2024

v1v May 10, 2024

v1v May 10, 2024

kruskall left a comment

v1v commented May 15, 2024 •

edited

kruskall commented May 17, 2024

v1v May 17, 2024

v1v commented May 23, 2024 •

edited

kruskall commented May 24, 2024

		package-docker-snapshot: $(DOCKER_IMAGE_SNAPSHOT_TARBALLS)
		@echo ">> $(DOCKER_IMAGE_SNAPSHOT_TARBALLS)"

build and push Docker image based on Chainguard base image #13137

build and push Docker image based on Chainguard base image #13137

Conversation

v1v commented May 10, 2024 • edited

Motivation/summary

Implementation details

Checklist

How to test these changes

CI

Locally

Related issues

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kruskall left a comment

Choose a reason for hiding this comment

v1v commented May 15, 2024 • edited

kruskall commented May 17, 2024

Choose a reason for hiding this comment

v1v commented May 23, 2024 • edited

kruskall commented May 24, 2024

v1v commented May 10, 2024 •

edited

v1v commented May 15, 2024 •

edited

v1v commented May 23, 2024 •

edited