Container labels inherited from UBI image

[stianst]

Before reporting an issue

• I have searched existing issues
• I have reproduced the issue with the latest nightly release

Area

dist/quarkus

Describe the bug

A number of labels are set on the container metadata that are simply inherited from UBI:

$ podman image inspect quay.io/keycloak/keycloak | jq .[0].Config.Labels
{
  "architecture": "x86_64",
  "build-date": "2022-06-17T04:19:16.231674",
  "com.redhat.build-host": "cpt-1001.osbs.prod.upshift.rdu2.redhat.com",
  "com.redhat.component": "ubi8-minimal-container",
  "com.redhat.license_terms": "https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI",
  "description": "The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
  "distribution-scope": "public",
  "io.k8s.description": "The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
  "io.k8s.display-name": "Red Hat Universal Base Image 8 Minimal",
  "io.openshift.expose-services": "",
  "io.openshift.tags": "minimal rhel8",
  "maintainer": "Red Hat, Inc.",
  "name": "ubi8-minimal",
  "org.opencontainers.image.created": "2022-07-29T11:17:24.398Z",
  "org.opencontainers.image.description": "",
  "org.opencontainers.image.licenses": "Apache-2.0",
  "org.opencontainers.image.revision": "55316db91317e8a22aa91fa9abf37b10e21cb6c9",
  "org.opencontainers.image.source": "https://github.com/keycloak-rel/keycloak-rel",
  "org.opencontainers.image.title": "keycloak-rel",
  "org.opencontainers.image.url": "https://github.com/keycloak-rel/keycloak-rel",
  "org.opencontainers.image.version": "19.0.1",
  "release": "854",
  "summary": "Provides the latest release of the minimal Red Hat Universal Base Image 8.",
  "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.6-854",
  "vcs-ref": "50293e4ba66ca2194d8ee5a149a665cb9a375c71",
  "vcs-type": "git",
  "vendor": "Red Hat, Inc.",
  "version": "8.6"
}

We should investigate if we can replace some (like name, maintainer, and io.k8s.*), and potentially remove some (com.redhat.*).

Version

main

Expected behavior

Container metadata labels contain information relevant to Keycloak and not the base image

Actual behavior

Container metadata labels contain information relevant to the base image

How to Reproduce?

$ podman image inspect quay.io/keycloak/keycloak | jq .[0].Config.Labels

Anything else?

No response

[shawkins]

There isn't a convienent way to remove the inherited labels from what I can see. A workaround is mentioned here: moby/moby#3465 (comment)

@vmuzikar what jobs are responsible for pushing the images to quay? Presumably we can use maven / application.properties for the operator - I've confirmed you can't even specify an empty string for a quarkus.container-image.labels value, so removing is not expected.

[vmuzikar]

what jobs are responsible for pushing the images to quay?

@shawkins Please see https://github.com/keycloak-rel/keycloak-rel/blob/main/.github/workflows/x-keycloak-container.yml

[shawkins]

On the operator side, I tried out:

quarkus.container-image.labels.version=1
quarkus.container-image.labels.release=2
quarkus.container-image.labels.url=3
quarkus.container-image.labels."com.redhat.component"=4
quarkus.container-image.labels."com.redhat.license_terms"=5
quarkus.container-image.labels.name=${quarkus.container-image.name}

quarkus.container-image.labels.maintainer=https://www.keycloak.org/
quarkus.container-image.labels.summary=Keycloak Operator Image
quarkus.container-image.labels."io.k8s.display-name"=Keycloak Operator

quarkus.container-image.labels.description=${quarkus.container-image.labels.summary}
quarkus.container-image.labels."io.k8s.description"=${quarkus.container-image.labels.description}
quarkus.container-image.labels.vendor=${quarkus.container-image.labels.maintainer}

With overrides in application-rhbk.properties.However even though the image building logs shows adding all of the labels, the acutal image does not have summary, vendor and all of dummy values (version, release, url, etc.) overriden.

I tried some of the same overrides with a LABEL command directly in the Dockerfile, and still hit the same problem. For whatever reason it looks like quite a few of these labels are fixed.

What we seem to be able to set:

name
maintainer
description
io.k8s.display-name
io.k8s.description

For consistency I'm now thinking that I'll just do this directly in the respective Docker files with appropriate env interpolation, such that the defaults will apply to the community.

@stianst @vmuzikar Does this sound good enough, or should I try to track down what Docker issue is in play with not being able to override everything?

[shawkins]

@stianst @vmuzikar Please ignore the previous comment it is invalid.

After looking over keycloak-rel, it looks like keycloak-rel/keycloak-rel#103 overlaps with this concern - we should consider merging that.

To have consistency with how labels are set we'll need to change the x-keycloak-operator workflow to directly use the Dockerfile, rather than the maven build. @ASzc does that logic already exist downstream somewhere?

[ASzc]

@shawkins Downstream builds server and operator from a plain Dockerfile that ADDs a zip of the server or operator contents. These dockerfiles override the labels from the parent UBI image by the LABEL instruction, quarkus properties aren't considered at all

[shawkins]

@ASzc can you point me to where than lives? I'd like to line everything up as closely as possible.

[shawkins]

Current state:

	Server	Operator
Local	Dockerfile s1, no labels accounted for	Dockerfile o1 used via a maven build, no labels accounted for
Community Release	Dockerfile s1 - some labels applied with https://github.com/keycloak-rel/keycloak-rel/blob/main/.github/workflows/x-keycloak-container.yml#L99C11-L99C17 (these are mostly incorrect) and there's a pr for more keycloak-rel/keycloak-rel#103	Dockerfile o1 used via a maven build, no labels accounted for
Product Release	Dockerfile s2 - maintains some labels in the Dockerfile, does not override the org.opencontainers.image labels as proposed in the pr above	Dockerfile o2 - maintains some labels in the Dockerfile, does not override the org.opencontainers.image labels as proposed in the pr above

The base proposal here is to convert the community release of the operator into one that looks like the product release. The labels being manuplated by the product release can be accounted for by the community release as well.

The next possibility would be to see if the Dockerfiles could be made identical, if not reused, across the releases. At the very least it would require parameterizing the label values. So we'll have one set of labels to maintain in both the operator and the keycloak Dockerfiles.

The last option is that it seems possible to create a secondary Dockerfile that will apply the set of labels we are interested in via a lot of build args. We'd then use that against in an secondary build against the image created by the particular release. Ideally this would give us just one place to maintain the labels.

@ASzc @vmuzikar is it good enough to stop at the base proposal or is there a value in trying to align this even more?

[ASzc]

Re: Dockerfile parameters, I'm limited by the capabilities of OSBS, and I don't think it supports setting these as far as I can see in its docs or schema. So you could use them, but the defaults would have to be for product, which is awkward. Similar story for more than one Dockerfile per component, it's not easily supported by OSBS

[shawkins]

@ASzc thank you for the perspective, so the best we can do is having 4 different Dockerfiles each maintaining their own labels.

[vmuzikar]

@ASzc @shawkins What if we generated the Dockerfile via Maven? It would then include a lengthy list of label overrides. A different Dockefile would be generated based on activation of -Dproduct property (and its profile) – the property and profile is already part of our POMs. But it's never ideal to include prod specific stuff in upstream...

If that's not feasible then I'd then vote for handling the labels in the existing 4 Dockerfiles independently.

[ASzc]

@vmuzikar Unfortunately there's no capability provided by CPaaS to alter containers' source files during build execution. So we can't replace the Dockerfile itself. However we can ADD stuff from the keycloak maven build into the container's Dockerfile (we do that with the server and operator zips now).

If we ADD'd a yaml to the first stage of a multi-stage build, that could pass the info in from maven. The problem then is I can't find a way to read that yaml into Dockerfile ENV variables, which would be required to pass them to the LABEL command. So, this approach probably won't work either for what we want to do

[shawkins]

What if we generated the Dockerfile via Maven?

@vmuzikar are you thinking about the build producing not only the tar.gz, but the associated Dockerfile artifacts as well?

[vmuzikar]

are you thinking about the build producing not only the tar.gz, but the associated Dockerfile artifacts as well?

Exactly. :)

But seems like we can't use a generated Dockerfile in prod based on @ASzc answer.

[vmuzikar]

I looked into it as well and I can confirm it really looks like there's not way to load ENV or ARG from a file. And if I understand correctly, another CPaaS limitations are that the Dockerfile needs to be "immutable" (we can't generate it or otherwise modify it via automation) and we can't provide any build args either. That leaves us only with a single option:

the best we can do is having 4 different Dockerfiles each maintaining their own labels

[ASzc]

4 discrete Dockerfiles does look like the only way to go at present. Maybe if we move the product containers to Konflux sometime, things will change. The code duplication is unfortunate but they are relatively simple to synchronize by hand, since they all do the same thing more or less

[shawkins]

Okay, let me start down this path.