chore: migrate from kuttl to chainsaw

[eddycharly]

This PR migrates e2e tests from kuttl to chainsaw.

Chainsaw was created because kuttl stopped to be actively maintained (it's actually a single maintainer now).

We are also solving most of the kuttl limitations that exist today and wrote a command to automatically migrate from kuttl to chainsaw. It supercharges e2e tests with better logs, config maps assertions, assertions trees and many more things 💪

This is a relatively young project but it has already been adopted by a couple of e2e tests heavy projects (kyverno, keptn, fairwinds rbac-manager, grafana-operator, redis-operator and others).

To give more context, you can see similar PRs at:

• chore: replace kuttl tests with chainsaw keptn/lifecycle-toolkit#3000
• feat: add chainsaw scorecard test support operator-framework/operator-sdk#6676
• chore: migrate from kuttl to chainsaw grafana/tempo-operator#786 in this PR the tempo-operator maintainers asked some good questions and I explained the current status, that I suggest taking a look at.

Sadly, kuttl haven't done a release since jan 2023.

Here you can find a good video showcasing chainsaw: https://www.youtube.com/watch?v=Ejof-wtAdQM&ab_channel=CamilaMacedo

To make things more concrete, there are a couple of issues in this repo, a few files are not valid and kuttl doesn't complain because unmarshalling is not strict so the non-valid fields are simply ignored. This is a problem because the behaviour of the tests is not the expected one, for example:

• opentelemetry-operator/tests/e2e-opampbridge/opampbridge/00-assert.yaml

  Lines 67 to 70 in c534eaf

  	apiVersion: kuttl.dev/v1beta1
  	kind: TestAssert
  	opampbridges:
  	- selector: app.kubernetes.io/component=opentelemetry-opamp-bridge

• opentelemetry-operator/tests/e2e-prometheuscr/create-sm-prometheus-exporters/07-delete.yaml

  Lines 6 to 8 in 94c8420

  	metadata:
  	name: simplest
  	namespace: create-sm-prometheus

If you need more infos 👇

• GitHub: https://github.com/kyverno/chainsaw
• Docs: https://kyverno.github.io/chainsaw
• Slack: https://kubernetes.slack.com/archives/C067LUFL43U

[eddycharly]

Replaces #2607 🙈

[eddycharly]

For reference, we should really avoid this kind of thing

opentelemetry-operator/tests/e2e/prometheus-config-validation/00-promreceiver-allocatorconfig.yaml

Lines 16 to 19 in b32c855

	apiVersion: kuttl.dev/v1beta1
	kind: TestStep
	commands:
	- command: kubectl -n $NAMESPACE create clusterrolebinding default-view-$NAMESPACE --clusterrole=promreceiver-allocatorconfig --serviceaccount=$NAMESPACE:ta

Chainsaw doesn't know anything about resources created with kubectl and therefore won't clean them up.

[swiatekm-sumo]

For reference, we should really avoid this kind of thing

opentelemetry-operator/tests/e2e/prometheus-config-validation/00-promreceiver-allocatorconfig.yaml

Lines 16 to 19 in b32c855

	apiVersion: kuttl.dev/v1beta1
	kind: TestStep
	commands:
	- command: kubectl -n $NAMESPACE create clusterrolebinding default-view-$NAMESPACE --clusterrole=promreceiver-allocatorconfig --serviceaccount=$NAMESPACE:ta

Chainsaw doesn't know anything about resources created with kubectl and therefore won't clean them up.

We can use templating for it, right? That can happen as a follow-up PR though.

[eddycharly]

@swiatekm-sumo we can use templating indeed, will do in a follow-up.

This one is already big enough 😂

[swiatekm-sumo]

LGTM overall, left some minor comments and questions about the Makefile and potential improvements to the tests using chainsaw's features.

[swiatekm-sumo]

I think we can just delete this line and adjust the next one. All of these tools are now installed automatically by the make targets.

[eddycharly]

I will do that.

[eddycharly]

Follow up PR if it's ok.

[swiatekm-sumo]

We have a small utility to make this nicer.

Suggested change

	@{ \
	set -e ;\
	go install github.com/kyverno/chainsaw@v0.1.4 ;\
	}
	$(call go-get-tool,$(CHAINSAW), github.com/kyverno/chainsaw,v0.1.4)

Can we also move the version to its own variable, like the other tools?

[eddycharly]

👍

[eddycharly]

Follow up PR if it's ok.

[swiatekm-sumo]

LGTM, we can take care of minor fixes in a follow up. Thanks a lot for your work on this contribution @eddycharly !

[eddycharly]

Ok, i'll address those comments in follow ups once this one merges :)

[jaronoff97]

Incredible work here @eddycharly !!! Thank you very much for your contributions. I'm looking forward to those follows to clean this up further.