User Request: Support for pod state disruptions #378
Replies: 3 comments
-
I'm happy to take a stab on this soon once we decide on these implementation details. |
Beta Was this translation helpful? Give feedback.
-
Hi, sorry for the very late reply but we couldn't find time to answer to this yet. We never implemented this disruption in the controller because at Datadog we consider a pod being killed (either softly or violently) should not impact us at all as long as the pod can be rescheduled elsewhere to keep the scale. But we definitely agree that it could be an interesting feature outside of the Datadog usecase. If you want to tackle this one, we can work together on a design document in this issue. In the CRD, I'd go with a new field In the injector, as we always try to be container engine agnostic, I'd go with signals which are often an issue (with apps in containers not catching termination signals so they end up with a What do you think? |
Beta Was this translation helpful? Give feedback.
-
Sure thing. |
Beta Was this translation helpful? Give feedback.
-
Is your feature request related to a problem? Please describe.
Pod state failures (e.g. graceful/non-graceful deletion) are common disruptions in the Chaos Engineering community.
The reasoning behind pod failures is that Kubernetes pods are ephemeral resources; they get destroyed, restarted, recreated.
This happens in many cases:
Pod state disruptions can expose a number of reliability concerns including:
Describe the solution you'd like
Pod deletions can be executed in many different ways. The easiest is through the Kubernetes client which supports graceful and non-graceful deletions through its gracePeriodSeconds parameter. This is how tools like Kube Monkey and our internal controller execute that disruption.
The other option would be to do this at container level which provides more granularity. This is how Pumba executes these disruptions.
A few more implementation details/ideas:
pod
for these disruptions.podFailure
, similar tonodeFailure
, with options (e.g. graceful/non-graceful deletion).containers
field which would allow targeting containers.Beta Was this translation helpful? Give feedback.
All reactions