User Request: HTTP Disruptions

[blazebissar]

Note: While chaos-controller is open to the public and we consider all suggestions for improvement, we prioritize feature development that is immediately applicable to chaos engineering initiatives within Datadog. We encourage users to contribute ideas to the repository directly in the form of pull requests!

Is your feature request related to a problem? Please describe.
@scottjr632 and I are interested in disrupting some web services at the application HTTP layer.

Describe the solution you'd like
If this is a desirable feature for Datadog, we are willing to design and implement an HTTP disruption capability. We would provide the design for review by early July to ensure that it lines up with Datadog plans for the controller and then proceed with implementation. We are currently looking into how possible this is to do at the interface/pod level, though it may require some kind of application capability due to the protocol's layer and the encryption of HTTPS.

Describe alternatives you've considered
A network disruption could approximate this functionality, but would be too coarse to target specific HTTP request types or URI endpoints without also disrupting other requests.

[ptnapoleon]

This is a desirable feature, and we're willing to collaborate on both/either of design or implementation. We're definitely happy to take suggestions for a design. We hadn't planned to start this anytime soon, so there's no risk of duplicate work (at least not in the near future)

[Devatoria]

Thank you for the request. It is, indeed, a desirable feature. We haven't designed anything related to this yet but I have some (eventually wrong) ideas in mind which can help as a starting point but feel free to go with your own ideas as well of course.

If the idea is to apply the same disruptions as a network disruption but with filters based on HTTP headers (like the verb, the path, the host, etc.), there is a potential implementation with a combination of iptables and existing network disruptions with tc. iptables rules being applied before tc rules on outgoing packets, you could probably rely on the string extension to match the header and apply a specific mark on the packet so tc can catch it later. However, because headers are encrypted when using HTTPS, it won't work as expected.

The second idea is having a more advanced HTTP feature with response mocking. In this case, like the DNS disruption, you would need to setup a local proxy and redirect outgoing packets to that proxy so it handles them when needed. DNS disruption relies on iptables to redirect packets and it could eventually work the same way. For HTTPS though, it means that you also need to handle the SSL connection and everything that comes with it.

[scottjr632]

@blazebissar and I have created an initial design doc for this enhancement. Our current goal is to finalize the design doc and create a POC for the disruption working with HTTPS by 19 July. Any feedback or critique is welcomed!

[ptnapoleon]

I'm curious to see how well the poc performs, but the design looks reasonable

[blazebissar]

We made some modifications to the design based on a small-scale POC and the comments everyone has made. Later this week we plan to make the POC within the chaos-controller.

[blazebissar]

We're running a little behind schedule and are anticipating completion of the POC by 24 July.