Adaptive concurrency limiting for circuit breakers

[jhalterman]

Similar to #27, I've been thinking about how to make circuit breaking more effective. One of the things that make time based circuit breakers effective is that they use recent error % to decide when to open, which is a very accurate and easy to configure signal. But once they're open, the delay before transitioning to half-open is statically configured, ex: 1 minute, which is not a very accurate way of knowing whether the system being protected is still overloaded. This delay is an indirect way of waiting for concurrency in the system to drop, without actually knowing what the concurrency is. A more direct, and better approach, would be to involve concurrency in the circuit breaker's configuration.

For example:

• open threshold: 10% error rate
• half-open threshold: 25% decrease in concurrency from where it was when opened

This perhaps starts to look a bit more like an adaptive concurrency limiter than a circuit breaker, but there are some differences:

• An adaptive concurrency limiter uses recent error rates or latency to continuously adjust concurrency limits up or down. This approach has some drawbacks:
  • Gradually lowering or raising limits can be slow to respond
  • Concurrency can only be lowered to 1
• The idea above would use recent error rates as a signal to allow concurrency to drop, effectively limiting it downwards.

The benefit of this approach is we would continue to have a load signal to drive our decision making, rather than cutting off all load (open state) and guessing about how long to keep it cutoff.