Improve Disruption status

[nikos912000]

Is your feature request related to a problem? Please describe.
Right now the Disruption status is one of the following:

• NotInjected
• PartiallyInjected
• Injected

The Injected status indicates a successful injection. However, there is no state that reflects a disruption that was not injected successfully.

Especially when integrating with CI/CD platforms it is critical to know the exact state of the experiments:

• This allows the pipeline to succeed or fail
• It also gives accurate feedback to users who do not have kubectl access

Describe the solution you'd like
A possible implementation can use stages and states for each stage.

Stage	State
Attack	Initialising
Attack	Running
Attack	Succeeded
Attack	Failed
Rollback	Initialising
Rollback	Running
Rollback	Succeeded
Rollback	Failed

This way, CI/CD platforms can expose these 2 stages to users and accurately reflect the status of each stage.

[nikos912000]

Hi @Devatoria @ptnapoleon 👋
Do you have any thoughts on the above?

[ptnapoleon]

Hi @nikos912000, sorry about the delay. I'll let others on the team give their opinions as well, but I think this is more state than we want to calculate or track. We are planning on making targeting and injecting no longer a one-time operation, so the concept of stages would be restricted to "has it been deleted yet and thus is trying to clean up". That's easy enough to know and communicate to the user. Would adding "Cleaning" and "FailedInjecting" states be sufficient for your needs?

[nikos912000]

Thanks for checking @ptnapoleon.
We are planning on making targeting and injecting no longer a one-time operation
Would you mind providing more context on this? Do you mean the controller will be continuously running disruptions?

As regards the recommended states, let me get some feedback.
I feel a simple state machine with clear end states would be sensible for this.

[ptnapoleon]

Would you mind providing more context on this? Do you mean the controller will be continuously running disruptions?

So currently if you have 5 possible targets that match your selector, and set count: 5 or count: 100%, we will inject into those 5 targets, and will not re-select or re-inject into new targets after that. We are planning on modifying the behavior of the controller so that we create a Watcher on the targeted selector, and when new possible targets appear|leave, we continually inject or clean to make sure the count of injector pods matches the spec.count as closely as possible.

As an example, right now if there were only 3 targets with a count: 5, we would inject into those 3 and be done with injecting, even if two new targets appeared. With this upcoming change, the controller will notice the two new targets, re-reconcile and inject into those as well.

[nikos912000]

I see, this will be very useful.
It might make sense to control that behavior through a field in the Custom Resource. Cause it may have unintended consequences.

[Devatoria]

Sorry for the delay as well, it seems that my email filters are still not working as I'd expect and I totally missed the ping here. Those unintended consequences is what we are trying to spot and avoid, indeed. Even at Datadog, we have users with different expectations on this behavior. The simplest example is what should we do with a disruption with a 100% target count if the targets start to fail because of the disruption (let's say the liveness probe failing)? It'd continuously kill the targets and could of course lead to some undesired issues. We are working on adding safeguards around that behavior so we can avoid such scenarios (for instance, if a targeted pod fails while a disruption is ongoing, maybe should we not try to inject a new target when it comes up).

About this more detailed disruption states, as Philip said, I think it is worth waiting a bit for this behavior to be implemented because the state would not be an end state (when a new target comes, it could move from injected to running before going back to injected or failed). Though I'm not against having something a bit more elaborated especially about the cleanup that we do not have as of today. We should just wait for that target count reconciliation feature to be implemented before going further so we can see how much sense it makes to have those stages.