Verification of Chaos Experiment

[Bhumika1997]

Hi, We have executed the chaos experiment using kubectl apply -f disruption_file.yaml and it says that it is created but I need to understand that from where we can verify our fault that fault is injected or not and the impact of our experiments.
Can somebody help me on this?

[nikos912000]

@Devatoria @ptnapoleon related to the question above, I was thinking we could introduce end-to-end tests where we apply the demo CRs and verify the injection.

Idea:

• Use the Kubernetes E2E testing framework which relies on Ginkgo. The latter is already used in this repo.
• Run the tests in a CI environment; this could either be the provided minikube environment or a kind environment that is easy to set up
• Users can then run the same tests in their local/internal clusters as a post-deploy verification test. Different users have different setups and this will provide a more realistic and reliable testing strategy.
• Tests can start simple and evolve. A good starting point is https://github.com/DataDog/chaos-controller/blob/main/CONTRIBUTING.md#testing-local-changes-in-minikube.

[Devatoria]

Hi Nikos and sorry for the late reply, summer and everything...

Implementing integration/e2e tests against a real cluster is in our OKRs for this quarter and we will start working on it very soon. It'll likely be manual and local at the beginning (understand not automatically through the CI but through a single command similar to make test) and we'll then make it evolve to something more automated.

Thank you for all the suggestions which are definitely very close to what we have in mind. The very first step would be to keep the tests we have now but run them against an existing/real cluster which is possible with envtest (that we are using in our controller tests currently). We'll probably rely on minikube at first since it is the "common" platform for people working on the controller but we'll allow to run them against any cluster for the exact same reason you mentioned: everybody, including Datadog, has specific cluster setups and we need to be able to test the controller against all those setups.

We'll keep you posted as soon as we have a draft!

[Devatoria]

@nikos912000 PR #401 is a first shot to have the same controllers tests as we have now but running against a real minikube cluster in the CI (and it works locally as well of course)
We'll do some more PRs later to add better test cases with real disruptions (they are currently dry-run only)