K8SPS-331 - chore: migrate from kuttl to chainsaw

[eddycharly]

This PR migrates e2e tests from kuttl to chainsaw.

Chainsaw was created because kuttl stopped to be actively maintained (it's actually a single maintainer now).

We are also solving most of the kuttl limitations that exist today and wrote a command to automatically migrate from kuttl to chainsaw. It supercharges e2e tests with better logs, config maps assertions, assertions trees and many more things 💪

This is a relatively young project but it has already been adopted by a couple of e2e tests heavy projects (kyverno, keptn, fairwinds rbac-manager, grafana-operator, redis-operator and others).

To give more context, you can see similar PRs at:

• chore: replace kuttl tests with chainsaw keptn/lifecycle-toolkit#3000
• feat: add chainsaw scorecard test support operator-framework/operator-sdk#6676
• chore: migrate from kuttl to chainsaw grafana/tempo-operator#786 in this PR the tempo-operator maintainers asked some good questions and I explained the current status, that I suggest taking a look at.

Sadly, kuttl haven't done a release since jan 2023.

Here you can find a good video showcasing chainsaw: https://www.youtube.com/watch?v=Ejof-wtAdQM&ab_channel=CamilaMacedo

To make things more concrete, there are a couple of issues in this repo, a few files are not valid and kuttl doesn't complain because unmarshalling is not strict so the non-valid fields are simply ignored. This is a problem because the behaviour of the tests is not the expected one.

If you need more infos 👇

• GitHub: https://github.com/kyverno/chainsaw
• Docs: https://kyverno.github.io/chainsaw
• Slack: https://kubernetes.slack.com/archives/C067LUFL43U

[it-percona-cla]

All committers have signed the CLA.

[eddycharly]

Ooops, i think i need to do something for jenkins 🤔

[hors]

Ooops, i think i need to do something for jenkins 🤔

please check https://github.com/percona/percona-server-mysql-operator/blob/main/Jenkinsfile

[eddycharly]

Thanks @hors !

How does chainsaw gets installed on jenkins ?

[eddycharly]

How does chainsaw gets installed on jenkins ?

Found it.

[eddycharly]

@hors @tplavcic why Error: open ./e2e-tests/kuttl.yaml: no such file or directory in the logs ?

[eddycharly]

There's something i'm not getting here

[eddycharly]

@hors @tplavcic a bit of help would be greatly appreciated.

[JNKPercona]

Test name	Status
async-ignore-annotations	failure
auto-config	failure
config	failure
config-router	failure
demand-backup	failure
gr-demand-backup	failure
gr-demand-backup-haproxy	failure
gr-haproxy	failure
gr-ignore-annotations	failure
gr-init-deploy	failure
gr-one-pod	failure
gr-recreate	failure
gr-scaling	failure
gr-self-healing	failure
gr-tls-cert-manager	failure
gr-users	failure
haproxy	failure
init-deploy	failure
limits	failure
monitoring	failure
one-pod	failure
operator-self-healing	failure
recreate	failure
scaling	failure
service-per-pod	failure
sidecars	failure
smart-update	failure
tls-cert-manager	failure
users	failure
version-service	failure
We run 30 out of 30

commit: 8c68083
image: perconalab/percona-server-mysql-operator:PR-560-8c68083

[eddycharly]

@tplavcic let me know when i should resolve conflicts

[tplavcic]

@eddycharly We have this problem: https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/client-and-managed-controllers/why-jenkinsfile-changes-are-not-reflected-in-pr-build
So we need to find some workaround, because I have tried to create a draft PR and it seems it is running chainsaw code: #566

[eddycharly]

@tplavcic got it !

Maybe we can have both kuttl and chainsaw first, then remove kuttl from jenkins, and finally remove kuttl tests ?

I think in this order it should work 🤷

[tplavcic]

Hi @eddycharly !
We have added chainsaw binary into our pipeline so you can continue and try to run the tests. So please revert your changes in Jenkinsfile (jenkins will not run them anyway) and merge the current main branch to your branch.

FYI: Currently we are working on 0.7.0 release of this operator and we cannot include chainsaw tests into this release, but we will consider it for the next one (we will need to inspect the tool more, have more test runs etc. to be more confident).

Is there a possibility to share a tool for migration? Can this tool be re-run simply/quickly if we have many test changes?

Thank you very much for your work!

[eddycharly]

@tplavcic i will work on this next week, thanks for the update.

[tplavcic]

@eddycharly There are some discussions in the team about the direction for our E2E tests so I propose for the moment you don't spend more time on this PR until some decision is made. Thanks!

[eddycharly]

Hi @tplavcic thanks for letting me know !
Ping me when you make a decision.