ci: uffizzi ephemeral environments for every PR

[waveywaves]

Thank you for contributing to Harbor!

Comprehensive Summary of your change

This PR adds a ephemeral environment solution for harbor to support creating virtual clusters for every pull request opened in this repo. After this PR is merged, all subsequent PRs created will each have a virtual cluster spun up and a harbor deployments done in the cluster. The harbor services deployed would be built from the changes in the pull request.

The uffizzi cluster lifecycle and ingress creation

When the uffizzi cluster creation workflow succeeds in the PR, a comment, similar to the one below will be posted. The comment will also contains a link to to the ingress for the instances of harbor services running in the cluster

waveywaves#1 (comment)

https://my-release-harbor-ingress-default-pr-1-c916.uclusters.app.uffizzi.com/

The above is an example of what the url to the harbor instance will look like. Here, my-release-harbor-ingress is the ingress name, default is the namespace where harbor is deployed pr-1 is the cluster name which can be accessed by the user.

Once the PR is closed, the cluster and therefore the resources within will be purged. The ingress will also not work thereafter.

Creation of Uffizzi Clusters is quick and easy. The clusters will be accessible to the developer opening the PR and the admins of the project. These will improve collaboration through a singular ephemeral environment which can be shared between developers and other stakeholders.

Issue being fixed

Fixes #18006

Please indicate you've done the following:

• Well Written Title and Summary of the PR
• Label the PR as needed. "release-note/ignore-for-release, release-note/new-feature, release-note/update, release-note/enhancement, release-note/community, release-note/breaking-change, release-note/docs, release-note/infra, release-note/deprecation"
• Accepted the DCO. Commits without the DCO will delay acceptance.
• Made sure tests are passing and test coverage is added if needed.
• Considered the docs impact and opened a new docs issue or PR with docs changes if needed in website repository.

cc @OrlinVasilev

[OrlinVasilev]

@waveywaves as we discussed it in the call can you add the skip for few paths like we have in CI:

on:
pull_request:
paths-ignore:
- 'docs/'
- '.md'
- 'tests/'
- '!tests/.sh'
- '!tests/apitests/'
- '!tests/ci/'
push:
paths-ignore:
- 'docs/'
- '.md'
- 'tests/'
- '!tests/.sh'
- '!tests/apitests/'
- '!tests/ci/'

thank you

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 65.31%. Comparing base (45b41d4) to head (06199e4).
Report is 141 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #19415      +/-   ##
==========================================
- Coverage   67.52%   65.31%   -2.22%     
==========================================
  Files         991     1242     +251     
  Lines      109168   124292   +15124     
  Branches     2719     5443    +2724     
==========================================
+ Hits        73721    81182    +7461     
- Misses      31476    38767    +7291     
- Partials     3971     4343     +372     

Flag	Coverage Δ
unittests	65.31% <ø> (-2.22%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 304 files with indirect coverage changes

[waveywaves]

@OrlinVasilev I have updated the workflow with the skips as you suggested. This workflow does not work on pushes and only does on pull requests so I have added the skips to the pull request bits.

[OrlinVasilev]

@waveywaves Does that mean if you push another commit to a PR it will not recreate the env?

[waveywaves]

@OrlinVasilev it will recreate the env if the pushes are on the pr. What I was referring to in the earlier comment was the

on:
  pull_request:
    paths-ignore:
    - 'docs/'
    - '.md'
    - 'tests/'
    - '!tests/.sh'
    - '!tests/apitests/'
    - '!tests/ci/'
  push:
    paths-ignore:
    - 'docs/'
    - '.md'
    - 'tests/'
    - '!tests/.sh'
    - '!tests/apitests/'
    - '!tests/ci/'

push part in the above section. this is not necessary, it will be redundant is what I meant. Because the environments only get triggered on pushes to a branch from which a pull request has been opened.

[OrlinVasilev]

@waveywaves can you add few lines of the implementations under https://goharbor.io/docs/2.9.0/build-customize-contribute/ will be great if we have that documented and also will help out with the use case:
what cli do you need to take a advantage of the envs
how to set it up
how to troubleshoot
how to ...

I can help out writing this if you need me!

[OrlinVasilev]

LGTM
I think that would be a great addition to the testing capabilities that we have and will be adding great value for the new contributors to experiment

[waveywaves]

@waveywaves can you add few lines of the implementations under https://goharbor.io/docs/2.9.0/build-customize-contribute/ will be great if we have that documented and also will help out with the use case: what cli do you need to take a advantage of the envs how to set it up how to troubleshoot how to ...

I can help out writing this if you need me!

@OrlinVasilev yep let me know how I should go about it

[OrlinVasilev]

@waveywaves you can add new section here: https://github.com/goharbor/website/tree/main/docs/build-customize-contribute
adding new file like :
using-uffizzi.md
following the style that we have and adding the information about will be great!

[waveywaves]

@OrlinVasilev I have created a basic PR to the docs goharbor/website#501.

[OrlinVasilev]

@Vad1mo @wy65701436 can you review please!

[OrlinVasilev]

@goharbor/all-maintainers can you please address your concerns(if any) here! Thank you so much!
@yanji09 @wy65701436 @Vad1mo

[zyyw]

As the PR author shared in the community meeting, the ephemeral Harbor env is installed via harbor-helm. The concern is that some Harbor oss features need corresponding changes in the goharbor/harbor-helm repository to make it work. Thus enable this uffizzi ephemeral Harbor env in harbor oss may not cover validating all PR features.
If we enable uffzzi in harbor-helm repository, then it only validates the harbor-helm PR changes. It does not validate PR opened in goharbor/harbor repository.

[OrlinVasilev]

@zyyw we wanted to have the helm implementation so we can test all that, if we think we can revert back to the docker-compose way.. With each new PR we will be testing against/with the current helm chart, which I don't see to be an issue. If we want to have a special helm chart version maybe @waveywaves can comment on this if there is a possibility to make that flexible.
Keep in mind that deployment is not a decision-making check for the PR merge, but rather to be able to test the changes you have implemented.

[zyyw]

Keep in mind that deployment is not a decision-making check for the PR merge, but rather to be able to test the changes you have implemented.

@OrlinVasilev I have a different opinion on this though. As mentioned above, if a feature implemented in one PR within goharbor/harbor repository and it requires corresponding changes in goharbor/harbor-helm as well to make it work for those deployment via harbor-helm. Then, uffizzi, deploying harbor via harbor-helm, does not actually test the PR changes on goharbor/harbor. Because this PR is not merged into the main branch of goharbor/harbor yet.

One concrete example would be this:
Say we have rise a PR1 in goharbor/harbor to enable cache layer like this. We also need to make a PR2 in goharbor/harbor-helm to enable cache layer configuration as well like this.
When PR1 is submitted, if we have uffizzi, uffizzi deployed Harbor via harbor-helm. However, PR1 is not merged in goharbor/harbor (harbor-helm use the dev tag image for harbor components, which is built based on main branch), so the ephemeral Harbor env created by uffizzi does not actually test PR1 at all.

[OrlinVasilev]

@zyyw Agree with you, but that is a corner case, right? What's your process for testing cases like this so maybe we can have uffizzi implement it :)

[waveywaves]

@zyyw That is correct, considering the harbor helm chart naturally won't be able to keep up with the developments in harbor as is the case with any projects and their deployments, you won't be able to test the latest harbor features with your existing helm chart. But you would be able to test developments in existing harbor features with the available helm chart deployment, and I am sure that would benefit developers working on existing features quite a bit.

On that note, we are working on multi PR environments through which you should be able to test changes from multiple PRs in one cluster.

@OrlinVasilev you are right this is an edge case but, Uffizzi should still be very helpful for developers working on existing features and we can work on extended solutions which can help drive productivity here.

[OrlinVasilev]

@waveywaves thanks for your response! Indeed I think even a bit limited we still can benefit quite extensively by be able to test with Uffizzi!

[OrlinVasilev]

@zyyw can you please share your workflow , thank you!

[zyyw]

@zyyw can you please share your workflow , thank you!

IMO, if Uffizzi can manage to install harbor via docker-compose, then it would be beneficial to the goharbor/harbor project to test/validate PRs within goharbor/harbor project. Otherwise, Uffizzi's currently implementation, which deploys Harbor via harbor-helm, actually tests/validates PRs within goharbor/harbor-helm project.

As Uffizzi deploys harbor via harbor-helm, and harbor-helm images use dev tag images, for example goharbor/harbor-core:dev, which are built against all merged PRs in the main branch of goharbor/harbor project. Does it test/validate pre-merged PRs in goharbor/harbor project at all?

@waveywaves @OrlinVasilev please correct me if I misunderstand anything. Thanks

[OrlinVasilev]

@waveywaves how hard is it to rework it ?

[wy65701436]

Following the community meeting, it appears that Uffizzi can exclusively deploy Harbor using the Helm chart. However, within the Harbor core, there's a need to establish a process for synchronizing code changes from the current pull request to Helm. Otherwise, given we merge this, it still cannot get the environment with the latest code change.

@waveywaves this should be an blocker that needs to be figured out.

[github-actions]

This PR is being marked stale due to a period of inactivty. If this PR is still relevant, please comment or remove the stale label. Otherwise, this PR will close in 30 days.

[github-actions]

This PR was closed because it has been stalled for 30 days with no activity. If this PR is still relevant, please re-open a new PR against main.