🚀 Add Dockerhub publish during release

[giom-l]

What changes did you make? (Give an overview)
Add login to dockerhub and push image to it in main workflow
Fixes #237

Is there anything you'd like reviewers to focus on?
Secrets names may need to be changed based on what is currently available (no visibility on it)

How Has This Been Tested? (put an "x" (case-sensitive!) next to an item)

• Yes, on my forked project with own credentials

Checklist (put an "x" (case-sensitive!) next to all the items, otherwise the build will fail)

• I have performed a self-review of my own code

Check out Contributing and Code of Conduct

[github-actions]

Hi giom-l! 👋

Welcome, and thank you for opening your first PR in the repo!

Please wait for triaging by our maintainers.

Please take a look at our contributing guide.

[Haarolean]

Hi, could you make this a separate job (within the same workflow) please? So we don't have to rebuild (or re-push) everything if one of the things fail.
Ideally:

• build
• ghcr push
• dockerhub push
• aws ecr push

You'd need to transfer artifacts between the jobs, feel free to use .github/workflows/e2e-*.yml workflows as reference.

[giom-l]

Hi,
I spent some time last days to on that and mostly struggled with my own station to have quicker local builds...
Anyway I also took into account #349 to go with a solution that would support it.

So here are 2 versions that work the same, let me know which one your prefer.
In both, I had to activate containerd as builder, for some different reasons (being able to load multi platforms images & preserve provenance attestations)

With docker save / docker load mechanism :
https://github.com/giom-l/kafka-ui/blob/feat/improve_docker_publish/.github/workflows/save_load.yml

With containerd all along (using oci output & ctr to reuse the produced image)
https://github.com/giom-l/kafka-ui/blob/feat/improve_docker_publish/.github/workflows/with_ctr.yml

All produced images can be found in all 3 repositories :

• Github packages: https://github.com/giom-l/kafka-ui/pkgs/container/kafka-ui
• Dockerhub : https://hub.docker.com/repository/docker/gocho/kafka-ui/tags
• ECR public : https://gallery.ecr.aws/j4u0y1h1/kafka-ui?page=1

For ECR, my workflow uses OIDC provider .
It would be the same with classic credentials (I saw some in your other workflows) and even be less verbose by using this action that would allow something like

- name: Push to ECR
  id: ecr
  uses: jwalton/gh-ecr-push@v2
  with:
    access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    region: us-east-1
    local-image: kafka-ui:temp
    image: <repo>/kafka-ui:main, <repo>/kafka-ui:<sha>

[Haarolean]

Hi, I spent some time last days to on that and mostly struggled with my own station to have quicker local builds... Anyway I also took into account #349 to go with a solution that would support it.

So here are 2 versions that work the same, let me know which one your prefer. In both, I had to activate containerd as builder, for some different reasons (being able to load multi platforms images & preserve provenance attestations)

With docker save / docker load mechanism : https://github.com/giom-l/kafka-ui/blob/feat/improve_docker_publish/.github/workflows/save_load.yml

With containerd all along (using oci output & ctr to reuse the produced image) https://github.com/giom-l/kafka-ui/blob/feat/improve_docker_publish/.github/workflows/with_ctr.yml

All produced images can be found in all 3 repositories :

• Github packages: https://github.com/giom-l/kafka-ui/pkgs/container/kafka-ui
• Dockerhub : https://hub.docker.com/repository/docker/gocho/kafka-ui/tags
• ECR public : https://gallery.ecr.aws/j4u0y1h1/kafka-ui?page=1

For ECR, my workflow uses OIDC provider . It would be the same with classic credentials (I saw some in your other workflows) and even be less verbose by using this action that would allow something like

- name: Push to ECR
  id: ecr
  uses: jwalton/gh-ecr-push@v2
  with:
    access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    region: us-east-1
    local-image: kafka-ui:temp
    image: <repo>/kafka-ui:main, <repo>/kafka-ui:<sha>

@giom-l thank you for the analysis. I'd prefer using docker save/load mechanism to have the same approach between different workflows (for example, e2e-run.yml, already uses this approach).

Can you update the PR branch with this approach so we can proceed with the review? Thank you

[giom-l]

Sorry for the delay !
I updated the PR using the save/load mechanism as requested.
I also impacted the release workflow that I forgot initially.

For the ECR publish, it's still "WIP" as I would need some informations about how you want to authenticate and where (ecr public ?)

[Haarolean]

I updated the PR using the save/load mechanism as requested.

Thank you.

about how you want to authenticate

I guess, we could auth the same way we do here (unless you have a better suggestion?)

and where (ecr public ?)

Yes, as we (accidentally) had one before (source).

Can I also ask you to refactor the workflows to use reusable workflows to reduce the copypaste in these workflows? I can take a look myself later as an alternative.

[giom-l]

Sure.
Just a question about how you see that :)
One reusable action per target registry (ecr, github, ecr) ?
Or one reusable action (something like image-publish) that push to all registries (but I'm not sure how it behave if one of them fails. Can we still rerun just a part of it ?) ?

I thought about one generic reusable action that will take parameters (like registry, credentials) but since ECR login is not the same as the other, I'm not sure it's doable.

Let me know what you think

[Haarolean]

I see that as follows:

1. A workflow to run on a push to main
2. A workflow to run on a release
3. Both 1) and 2) call another workflow to build an image (docker-build.yml) and a workflow to publish one (docker-publish.yml)
4. docker-publish.yml contains 3 jobs for each registry (gh, ecr, dockerhub)
   In this case we can extract common parts for each workflow and keep publishing as separate jobs within a separate calling workflow.

Feel free to use e2e-*.yml workflows as templates, we have the basic required stuff there (calling other workflows, passing arguments through, transferring artifacts between jobs, etc.)

Can we still rerun just a part of it

Yes, we would be able to rerun separate jobs within a workflow like this:

[giom-l]

Hello

I gave this some time yesterday and made it work as follows (PR udpated) :

• 1 docker_build.yml workflow
• 1 docker_publish.yml workflow, which itself calls 3 differents workflows :
  • 1 publish_ecr.yml
  • 1 publish_dockerhub.yml
  • 1 publish_ghcr.yml

About extracting common parts for the three publish jobs :
It seems not really useful to me since it's only a matter of downloading artifacts & configuring docker daemon, which can't be shared between jobs.

However, I also experienced another solution that can be found in this branch :
The main & release job will call the docker_publish and this one is more generic (only the logging part isn't) and uses matrix instead of calling others workflows.
I find this solution to be more elegant (less duplicated code, no intermediary workflow)

Let me know what you think and I'll finalize the PR with the solution that is preferred on your side.

[Haarolean]

let's do 1, the jars are pretty fat to hold in the cache for too long

[Haarolean]

I don't think we need packages here anymore as we're building image in a separate job

[Haarolean]

why is this containerd-snapshotter thing required?

[Haarolean]

Hey, sorry for the delay.

One more thing, can we get separate workflows like publish_x.yml to be separate jobs within one workflow instead? I don't see any point in making them separate workflows rather than jobs, it just adds additional overhead. Let me know what you think.