Readiness and Liveness Probe Failing with Overload Manager Configuration

[sekar-saravanan]

Hi, I am working on emissary-ingress for quite sometime and configured the setup in kubernetes with below probs.

livenessProbe:
    httpGet:
        path: /ambassador/v0/check_alive
        port: 8877
        scheme: HTTP
readinessProbe:
    httpGet:
        path: /ambassador/v0/check_ready
        port: 8877
        scheme: HTTP

I'm in need of using envoy overload manager feature (envoy.overload_actions.stop_accepting_requests) to block some percent of request when cpu usage is too high and I made some modification on envoy.json to use the same.

  "overload_manager": {
    "actions": [
      {
        "name": "envoy.overload_actions.stop_accepting_requests",
        "triggers": [
          {
            "name": "envoy.resource_monitors.injected_resource",
            "scaled": {
              "saturation_threshold": 1,
              "scaling_threshold": 0.01
            }
          }
        ]
      }
    ],
    "refresh_interval": "3s",
    "resource_monitors": [
      {
        "name": "envoy.resource_monitors.injected_resource",
        "typed_config": {
          "@type": "type.googleapis.com/envoy.extensions.resource_monitors.injected_resource.v3.InjectedResourceConfig",
          "filename": "/tmp/pressure"
        }
      }
    ]
  }

when overload_actions triggered, its started to block some percent of requests. But unexpectedly, readiness and liveness probes (/ambassador/v0/check_ready, /ambassador/v0/check_alive) also getting failed.

Is there any other way, we can perform healthcheck on emissary-ingress instead (/ambassador/v0/check_ready, /ambassador/v0/check_alive) ?

[cindymullins-dw]

Those are the only health-check endpoints I know of. I'm not sure there's another way to access the readiness/liveness probes if blocked by a custom config. Will mark this as a feature request. You could look at Active Health Checking w/ the Endpoint Resolver.