bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily #2176

kworkbee · 2024-03-06T08:35:26Z

Current Behavior

There is a problem with the unhealthy external service being delivered as it is without being excluded from routing targets.

Expected Behavior

Two external services (ALB configured in front of each) are configured as upstream nodes and should be temporarily excluded from routing if a 5XX error occurs through health check configuration.

Error Logs

No response

Steps to Reproduce

For reproducing the issue, one service is deployed, and the other one is not deployed (only ALB's are set up.)

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: route
spec:
  http:
    - match:
        hosts:
          - service.corp.com
        methods:
          - POST
        paths:
          - /svc/*
      name: route
      plugins:
        - enable: true
          name: traffic-split
        - config:
            regex_uri:
              - ^\/svc\/(.+)$
              - /$1
          enable: true
          name: proxy-rewrite
      upstreams:
        - name: upstream
          weight: 100

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  name: upstream
  namespace: apisix
spec:
  externalNodes:
    - name: svc01.corp.com
      port: 443
      type: Domain
      weight: 50
    - name: svc02.corp.com
      port: 443
      type: Domain
      weight: 50
  healthCheck:
    active:
      healthy:
        httpCodes:
          - 200
          - 404
        interval: 3s
        successes: 1
      httpPath: /
      type: https
      unhealthy:
        httpCodes:
          - 503
          - 504
        httpFailures: 1
        interval: 3s
        timeouts: 3
    passive:
      healthy:
        httpCodes:
          - 200
          - 404
        successes: 1
      type: https
      unhealthy:
        httpCodes:
          - 503
          - 504
        httpFailures: 1
        timeouts: 3
  loadbalancer:
    type: roundrobin
  passHost: node
  scheme: https

Environment

APISIX Ingress controller version (run apisix-ingress-controller version --long)
Kubernetes cluster version (run kubectl version)
OS version if running APISIX Ingress controller in a bare-metal environment (run uname -a)

Runs on an AWS EKS Cluster (Kubernetes v1.25). Uses APISIX Helm Chart (1.11.0, App 3.8.0).

The text was updated successfully, but these errors were encountered:

kworkbee · 2024-03-06T10:35:22Z

In Admin API, a state different from the .spec.healthCheck.passive.healthy.httpCodes defined in ApisixUpstream is being returned.

"active": {
        "type": "http",
        "concurrency": 10,
        "http_path": "/",
        "timeout": 1,
        "healthy": {
          "http_statuses": [
            404
          ],
          "successes": 1,
          "interval": 3
        },
        "unhealthy": {
          "timeouts": 3,
          "interval": 3,
          "http_statuses": [
            503,
            504
          ],
          "http_failures": 1,
          "tcp_failures": 2
        },
        "https_verify_certificate": true
      },
      "passive": {
        "healthy": {
          "http_statuses": [
            // Not 200/404?
            200,
            201,
            202,
            203,
            204,
            205,
            206,
            207,
            208,
            226,
            300,
            301,
            302,
            303,
            304,
            305,
            306,
            307,
            308
          ],
          "successes": 5
        },
        "type": "http",
        "unhealthy": {
          "http_statuses": [
            503,
            504
          ],
          "timeouts": 7,
          "http_failures": 1,
          "tcp_failures": 2
        }
      }

kworkbee · 2024-03-07T04:49:42Z

It seems that health checks are made as they are without being rewritten to each host defined in externalNodes on the log.

2024/03/07 13:27:02 [warn] 51#51: *1196791 [lua] balancer.lua:82: fetch_health_nodes(): failed to get health check target status, addr: 123.456.789.876:443, host: nil, err: target not found, client: 123.789.345.678, server: _, request: "POST /svc HTTP/1.1", host: "svc01.corp.com"

kworkbee · 2024-03-07T04:50:11Z

@tokers @tao12345666333
@Revolyssup @shreemaan-abhishek

Revolyssup · 2024-03-12T06:34:17Z

@kworkbee Thanks for reporting. I'll take a look

Revolyssup · 2024-03-12T06:57:00Z

@kworkbee Meanwhile can you paste any relevant logs from ingress controller?

kworkbee · 2024-03-12T07:00:04Z

@Revolyssup Logs below:

2024-03-12T14:58:40+08:00	info	ingress/ingress.go:121	init apisix ingress controller
2024-03-12T14:58:40+08:00	info	ingress/ingress.go:123	version:
Version: 1.8.0
Git SHA: no-git-module
Go Version: go1.20.13
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64
2024-03-12T14:58:40+08:00	info	ingress/ingress.go:133	use configuration
{
  "cert_file": "/etc/webhook/certs/cert.pem",
  "key_file": "/etc/webhook/certs/key.pem",
  "log_level": "info",
  "log_output": "stderr",
  "log_rotate_output_path": "",
  "log_rotation_max_size": 100,
  "log_rotation_max_age": 0,
  "log_rotation_max_backups": 0,
  "http_listen": ":8080",
  "https_listen": ":8443",
  "ingress_publish_service": "",
  "ingress_status_address": [],
  "enable_profiling": true,
  "kubernetes": {
    "kubeconfig": "",
    "resync_interval": "6h0m0s",
    "namespace_selector": [],
    "election_id": "ingress-apisix-leader",
    "ingress_class": "apisix",
    "ingress_version": "networking/v1",
    "watch_endpoint_slices": false,
    "api_version": "apisix.apache.org/v2",
    "enable_gateway_api": false,
    "disable_status_updates": false,
    "enable_admission": false
  },
  "apisix": {
    "admin_api_version": "v3",
    "default_cluster_name": "default",
    "default_cluster_base_url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin",
    "default_cluster_admin_key": "******"
  },
  "apisix_resource_sync_interval": "1h0m0s",
  "apisix_resource_sync_comparison": true,
  "plugin_metadata_cm": "",
  "etcdserver": {
    "enabled": false,
    "prefix": "/apisix",
    "listen_address": ":12379",
    "ssl_key_encrypt_salt": "edd1c9f0985e76a2"
  }
}
2024-03-12T14:58:40+08:00	info	ingress/ingress.go:145	start ingress controller
2024-03-12T14:58:40+08:00	info	providers/controller.go:172	start leader election
I0312 14:58:40.791429       1 leaderelection.go:250] attempting to acquire leader lease apisix/ingress-apisix-leader...
2024-03-12T14:58:40+08:00	info	providers/controller.go:157	start api server
2024-03-12T14:58:40+08:00	info	providers/controller.go:143	LeaderElection	{"message": "dev-tool-apisix-ingress-controller-56bc5c7ff-5q6wp became leader", "event_type": "Normal"}
I0312 14:58:40.814029       1 leaderelection.go:260] successfully acquired lease apisix/ingress-apisix-leader
2024-03-12T14:58:40+08:00	warn	providers/controller.go:222	found a new leader dev-tool-apisix-ingress-controller-56bc5c7ff-5q6wp
2024-03-12T14:58:40+08:00	info	providers/controller.go:214	controller now is running as leader	{"namespace": "apisix", "pod": "dev-tool-apisix-ingress-controller-56bc5c7ff-5q6wp"}
2024-03-12T14:58:40+08:00	info	providers/controller.go:389	controller tries to leading ...	{"namespace": "apisix", "pod": "dev-tool-apisix-ingress-controller-56bc5c7ff-5q6wp"}
2024-03-12T14:58:41+08:00	info	apisix/stream_route.go:38	resource stream_routes is disabled
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:423	waiting cluster default to ready, it may takes a while
2024-03-12T14:58:41+08:00	info	apisix/cluster.go:248	syncing cache	{"cluster": "default"}
2024-03-12T14:58:41+08:00	info	apisix/cluster.go:463	syncing schema	{"cluster": "default"}
2024-03-12T14:58:41+08:00	info	apisix/cluster.go:252	cache synced	{"cost_time": "23.597419ms", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:434	cluster default now is ready, cost time 23.795508ms
2024-03-12T14:58:41+08:00	info	providers/controller.go:433	creating controller
2024-03-12T14:58:41+08:00	info	providers/controller.go:508	init namespaces
2024-03-12T14:58:41+08:00	info	providers/controller.go:515	wait for resource sync
2024-03-12T14:58:41+08:00	info	providers/controller.go:523	init providers
2024-03-12T14:58:41+08:00	info	providers/controller.go:535	try to run providers
2024-03-12T14:58:41+08:00	info	configmap/configmap.go:87	configmap controller started
2024-03-12T14:58:41+08:00	info	apisix/apisix_global_rule.go:65	ApisixGlobalRule controller started
2024-03-12T14:58:41+08:00	info	ingress/ingress.go:85	ingress controller started
2024-03-12T14:58:41+08:00	info	k8s/secret.go:79	secret controller started
2024-03-12T14:58:41+08:00	info	endpoint/endpoint.go:70	endpoints controller started
2024-03-12T14:58:41+08:00	info	apisix/apisix_upstream.go:91	ApisixUpstream controller started
2024-03-12T14:58:41+08:00	info	apisix/apisix_route.go:104	ApisixRoute controller started
2024-03-12T14:58:41+08:00	info	apisix/apisix_tls.go:78	ApisixTls controller started
2024-03-12T14:58:41+08:00	info	apisix/apisix_cluster_config.go:65	ApisixClusterConfig controller started
2024-03-12T14:58:41+08:00	info	apisix/apisix_consumer.go:69	ApisixConsumer controller started
2024-03-12T14:58:41+08:00	info	apisix/apisix_plugin_config.go:70	ApisixPluginConfig controller started
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "32eb11c7", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/32eb11c7", "cluster": "default"}
2024-03-12T14:58:41+08:00	error	apisix/apisix_upstream.go:376	failed to get upstream apisix_feature-flags-nodes-upstream: not found
2024-03-12T14:58:41+08:00	error	apisix/apisix_upstream.go:845	update status	{"status": {"conditions":[{"type":"ResourcesAvailable","status":"False","observedGeneration":1,"lastTransitionTime":"2024-03-12T06:58:41Z","reason":"ResourceSyncAborted","message":"not found"}]}}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "9756d1ef", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/9756d1ef", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "5d575ecc", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/5d575ecc", "cluster": "default"}
W0312 14:58:41.373281       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"
2024-03-12T14:58:41+08:00	warn	apisix/apisix_upstream.go:489	sync ApisixUpstream failed, will retry	{"object": {"Type":1,"Object":{"Key":"apisix/feature-flags-nodes-upstream","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "not found"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "3f53b397", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/3f53b397", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "c11717fc", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/c11717fc", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "bbc21df0", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/bbc21df0", "cluster": "default"}
W0312 14:58:41.418441       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "4586b99b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/4586b99b", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "971e4f30", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/971e4f30", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "c661d56", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/c661d56", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "f7be7887", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/f7be7887", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "738f1a00", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/738f1a00", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "6bbfb9e", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/6bbfb9e", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "2b57804a", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/2b57804a", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "307b185f", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/307b185f", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "477c28c9", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/477c28c9", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "b552709e", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/b552709e", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "7a4e2355", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/7a4e2355", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "715168ec", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/715168ec", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "eb4e7b5d", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/eb4e7b5d", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "1595d1c9", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/1595d1c9", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "4dfc631e", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/4dfc631e", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "606b3be0", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/606b3be0", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "dcf65d48", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/dcf65d48", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "abf16dde", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/abf16dde", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "269a612a", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/269a612a", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "55bfce9b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/55bfce9b", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "aaa91dc", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/aaa91dc", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "7dada14a", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/7dada14a", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "3669021e", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/3669021e", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "1d272733", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/1d272733", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "2ed9e1f1", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/2ed9e1f1", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "90640ae5", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/90640ae5", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "a51b5d6", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/a51b5d6", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "c18026ba", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/c18026ba", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "80649f40", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/80649f40", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "2cac3481", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/2cac3481", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "5c1c1952", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/5c1c1952", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "345a1e06", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/345a1e06", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "811258cf", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/811258cf", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "1dd1830e", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/1dd1830e", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "c662ff0", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/c662ff0", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "a9aefac7", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/a9aefac7", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "3a3b07f7", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/3a3b07f7", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "2ba1a8b9", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/2ba1a8b9", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "b232187b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/b232187b", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "30555848", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/30555848", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "e3ca777b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/e3ca777b", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "105d31bd", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/105d31bd", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "f3118c88", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/f3118c88", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "f3118c88", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/f3118c88", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "23ef4d0", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/23ef4d0", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "4a344633", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/4a344633", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "40be5e95", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/40be5e95", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "cb84cd70", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/cb84cd70", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "b85d412e", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/b85d412e", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "1580b33d", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/1580b33d", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "28399d36", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/28399d36", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "9ee809e8", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/9ee809e8", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "ee82fd67", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/ee82fd67", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "21194849", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/21194849", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "f28d69a9", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/f28d69a9", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "4815345b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/4815345b", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "7a7eff70", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/7a7eff70", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "b4b958f6", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/b4b958f6", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "77fa0f3b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/77fa0f3b", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "8f79a5c3", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/8f79a5c3", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "fe03eb03", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/fe03eb03", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "ded031c", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/ded031c", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "3686d301", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/3686d301", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "34aa912b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/34aa912b", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "2194b02b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/2194b02b", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "a2a099bb", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/a2a099bb", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "a428cf6f", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/a428cf6f", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "6bff0afa", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/6bff0afa", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "f661101d", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/f661101d", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "39b6d588", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/39b6d588", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "d63de15d", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/d63de15d", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "ac24d8ca", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/ac24d8ca", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "63f31d5f", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/63f31d5f", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "6e45694f", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/6e45694f", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "a192acda", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/a192acda", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "6fbc2a7f", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/6fbc2a7f", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "bce313ae", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/bce313ae", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "7334d63b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/7334d63b", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "426a4118", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/426a4118", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "8dbd848d", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/8dbd848d", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "9bccea51", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/9bccea51", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "541b2fc4", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/541b2fc4", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "cd127e7e", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/cd127e7e", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "c85d3ec9", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/c85d3ec9", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "78afb5c", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/78afb5c", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "9e83aae6", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/9e83aae6", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "c8628e4c", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/c8628e4c", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "631510da", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/631510da", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "c2b246fe", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/c2b246fe", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "5283d6b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/5283d6b", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "771ab980", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/771ab980", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "2cd3d003", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/2cd3d003", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "264dfd09", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/264dfd09", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "c661d56", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/c661d56", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "f7be7887", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/f7be7887", "cluster": "default"}
2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "738f1a00", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/738f1a00", "cluster": "default"}
2024-03-12T14:58:42+08:00	error	apisix/apisix_upstream.go:845	update status	{"status": {"conditions":[{"type":"ResourcesAvailable","status":"True","observedGeneration":1,"lastTransitionTime":"2024-03-12T06:58:42Z","reason":"ResourcesSynced","message":"Sync Successfully"}]}}
W0312 14:58:42.417094       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"

Revolyssup · 2024-03-13T05:31:28Z

@kworkbee Can you show the spec of apisix_feature-flags-nodes-upstream?

kworkbee · 2024-03-13T05:34:58Z

@Revolyssup The domains are modified and provided for security reasons.

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  name: feature-flags-nodes-upstream
spec:
  loadbalancer:
    type: roundrobin
  passHost: node
  scheme: https
  externalNodes:
  - type: Domain
    name: svc01.corp.com
    port: 443
    weight: 50
  - type: Domain
    name: svc02.corp.com
    port: 443
    weight: 50
  healthCheck:
    active:
      type: https
      port: 443
      strictTLS: true
      httpPath: /
      healthy:
        successes: 1
        interval: 3s
        httpCodes:
        - 404
      unhealthy:
        httpCodes:
        - 503
        - 504
        httpFailures: 1
        interval: 3s
        timeouts: 3
    passive:
      healthy:
        successes: 1
        httpCodes:
        - 404
      unhealthy:
        httpCodes:
        - 503
        - 504
        httpFailures: 1
        timeouts: 3

Revolyssup · 2024-03-13T05:42:35Z

@kworkbee What logs do you get immediately after kubectl apply -f <the above provided upstream> ?

Revolyssup · 2024-03-13T05:46:19Z

@kworkbee What logs do you get immediately after kubectl apply -f <the above provided upstream> ?

Can you delete both ApisixRoute and ApisixUpstream, reapply and show logs?

kworkbee · 2024-03-13T05:46:59Z

@Revolyssup Got these two lines below:

2024-03-13T13:46:13+08:00	error	apisix/apisix_upstream.go:845	update status	{"status": {"conditions":[{"type":"ResourcesAvailable","status":"True","observedGeneration":1,"lastTransitionTime":"2024-03-13T05:46:13Z","reason":"ResourcesSynced","message":"Sync Successfully"}]}}
W0313 13:46:13.985770       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"

kworkbee · 2024-03-13T05:52:11Z

@Revolyssup As you just mentioned, this is the result of erasing and re-applying both ApisixRoute and ApisixUpstream.

2024-03-13T13:49:24+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
2024-03-13T13:49:29+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
2024-03-13T13:49:34+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
2024-03-13T13:49:34+08:00	warn	providers/controller.go:598	failed to check health for default cluster: timed out waiting for the condition, give up leader
2024-03-13T13:49:34+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
2024-03-13T13:49:39+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
2024-03-13T13:49:44+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
2024-03-13T13:49:44+08:00	warn	providers/controller.go:598	failed to check health for default cluster: timed out waiting for the condition, give up leader
2024-03-13T13:49:44+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
2024-03-13T13:49:44+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "971e4f30", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/971e4f30", "cluster": "default"}
2024-03-13T13:49:44+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "9756d1ef", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/9756d1ef", "cluster": "default"}
2024-03-13T13:50:42+08:00	error	translation/apisix_route.go:238	failed to translate ApisixUpstream at Upstream[0]	{"error": "apisixupstream.apisix.apache.org \"feature-flags-nodes-upstream\" not found", "apisix_upstream": "apisix/feature-flags-nodes-upstream"}
2024-03-13T13:50:42+08:00	error	apisix/route.go:138	failed to create route: unexpected status code 400; error message: {"error_msg":"failed to fetch upstream info by upstream id [32eb11c7], response code: 404"}
2024-03-13T13:50:42+08:00	error	apisix/apisix_route.go:380	failed to sync ApisixRoute to apisix	{"error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
2024-03-13T13:50:42+08:00	warn	apisix/apisix_route.go:482	sync ApisixRoute failed, will retry	{"object": {"Type":1,"Object":{"Key":"apisix/feature-flags-route","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
W0313 13:50:42.477556       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"
2024-03-13T13:50:43+08:00	error	translation/apisix_route.go:238	failed to translate ApisixUpstream at Upstream[0]	{"error": "apisixupstream.apisix.apache.org \"feature-flags-nodes-upstream\" not found", "apisix_upstream": "apisix/feature-flags-nodes-upstream"}
2024-03-13T13:50:43+08:00	error	apisix/route.go:138	failed to create route: unexpected status code 400; error message: {"error_msg":"failed to fetch upstream info by upstream id [32eb11c7], response code: 404"}
2024-03-13T13:50:43+08:00	error	apisix/apisix_route.go:380	failed to sync ApisixRoute to apisix	{"error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
2024-03-13T13:50:43+08:00	warn	apisix/apisix_route.go:482	sync ApisixRoute failed, will retry	{"object": {"Type":1,"Object":{"Key":"apisix/feature-flags-route","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
2024-03-13T13:50:44+08:00	error	translation/apisix_route.go:238	failed to translate ApisixUpstream at Upstream[0]	{"error": "apisixupstream.apisix.apache.org \"feature-flags-nodes-upstream\" not found", "apisix_upstream": "apisix/feature-flags-nodes-upstream"}
2024-03-13T13:50:44+08:00	error	apisix/route.go:138	failed to create route: unexpected status code 400; error message: {"error_msg":"failed to fetch upstream info by upstream id [32eb11c7], response code: 404"}
2024-03-13T13:50:44+08:00	error	apisix/apisix_route.go:380	failed to sync ApisixRoute to apisix	{"error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
2024-03-13T13:50:44+08:00	warn	apisix/apisix_route.go:482	sync ApisixRoute failed, will retry	{"object": {"Type":1,"Object":{"Key":"apisix/feature-flags-route","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
2024-03-13T13:50:45+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "32eb11c7", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/32eb11c7", "cluster": "default"}
2024-03-13T13:50:45+08:00	error	apisix/apisix_upstream.go:376	failed to get upstream apisix_feature-flags-nodes-upstream: not found
2024-03-13T13:50:45+08:00	error	apisix/apisix_upstream.go:845	update status	{"status": {"conditions":[{"type":"ResourcesAvailable","status":"False","observedGeneration":1,"lastTransitionTime":"2024-03-13T05:50:45Z","reason":"ResourceSyncAborted","message":"not found"}]}}
W0313 13:50:45.220898       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"
2024-03-13T13:50:45+08:00	warn	apisix/apisix_upstream.go:489	sync ApisixUpstream failed, will retry	{"object": {"Type":1,"Object":{"Key":"apisix/feature-flags-nodes-upstream","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "not found"}
W0313 13:50:45.273798       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"
2024-03-13T13:50:46+08:00	error	apisix/apisix_upstream.go:845	update status	{"status": {"conditions":[{"type":"ResourcesAvailable","status":"True","observedGeneration":1,"lastTransitionTime":"2024-03-13T05:50:46Z","reason":"ResourcesSynced","message":"Sync Successfully"}]}}
W0313 13:50:46.274203       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"

Revolyssup · 2024-03-13T06:08:27Z

@kworkbee I just noticed that you have deployed apisixroute and apisixupstream in different namespaces. This is why while deploying apisixroute, upstream can't be found.

kworkbee · 2024-03-13T06:13:32Z

@Revolyssup It is distributed in the form of Helm Chart using ArgoCD, a CD tool, so I think you should see that it is distributed in the same namespace.

Revolyssup · 2024-03-13T06:16:44Z

@kworkbee In the original issue description, the route and upstream you have provided are not in same namespace. So you mean to say in your case, both are in default namespace?

Revolyssup · 2024-03-13T06:18:12Z

@Revolyssup The domains are modified and provided for security reasons.

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  name: feature-flags-nodes-upstream
spec:
  loadbalancer:
    type: roundrobin
  passHost: node
  scheme: https
  externalNodes:
  - type: Domain
    name: svc01.corp.com
    port: 443
    weight: 50
  - type: Domain
    name: svc02.corp.com
    port: 443
    weight: 50
  healthCheck:
    active:
      type: https
      port: 443
      strictTLS: true
      httpPath: /
      healthy:
        successes: 1
        interval: 3s
        httpCodes:
        - 404
      unhealthy:
        httpCodes:
        - 503
        - 504
        httpFailures: 1
        interval: 3s
        timeouts: 3
    passive:
      healthy:
        successes: 1
        httpCodes:
        - 404
      unhealthy:
        httpCodes:
        - 503
        - 504
        httpFailures: 1
        timeouts: 3

Also then provide the original route configuration

kworkbee · 2024-03-13T06:19:32Z

In the process of copying YAML, the information .metadata.namespace was incorrectly written, but in reality, both the ApisixRoute object and the ApisixUpstream object were deployed in the same apisix namespace.

Revolyssup · 2024-03-13T06:23:19Z

In the process of copying YAML, the information .metadata.namespace was incorrectly written, but in reality, both the ApisixRoute object and the ApisixUpstream object were deployed in the same apisix namespace.

Got it. Checking.....

kworkbee · 2024-03-13T06:25:24Z

@Revolyssup The intended structure is as follows:

Just in case, this is the result of double-checking whether namespace is the same or not:

Thank you very much for actively watching.

Revolyssup · 2024-03-13T06:29:27Z

200,
201,
202,
203,
204,
205,
206,
207,
208,
226,
300,
301,
302,
303,
304,
305,
306,
307,
308

Is this the only upstream? Can you send the output of

 curl http://127.0.0.1:9180/apisix/admin/upstreams -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' | jq

After port forwarding to apisix pod.

Also

 curl http://127.0.0.1:9180/apisix/admin/routes -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' | jq

will be helpful

Revolyssup · 2024-03-13T06:35:37Z

@kworkbee What you're seeing for healthcheck.active.healthy.httpCodes is the default value which is passed when you don't explicitly pass the value. It looks like at some point in time ApisixUpstream resource was deployed without passing the healthcheck.active.healthy.httpCodes. The upstream you showed from admin api might have been from that time. I can confirm that httpCodes passed in Upstream is succesfully translated to httpStatuses.

kworkbee · 2024-03-13T06:36:18Z

@Revolyssup Yes, only one upstream is currently �deployed.

Revolyssup · 2024-03-13T06:38:09Z

@Revolyssup Yes, only one upstream is currently �deployed.

please click on the passive and active, I wanted to see those values. Please show whole json

kworkbee · 2024-03-13T06:39:06Z

@Revolyssup

Revolyssup · 2024-03-13T06:41:40Z

@Revolyssup

Okay so there is no sync issue. This upstream configuration matches exactly what you provided via ApisixUpstream. So there is no issue at ingress controller level. Are you still running into the issue where unhealthy upstream is not excluded?

kworkbee · 2024-03-13T06:42:02Z

@Revolyssup Unlike a few days ago, Status Code is reflected well. But the phenomenon still seems to continue.

Revolyssup · 2024-03-13T06:42:14Z

@Revolyssup

Okay so there is no sync issue. This upstream configuration matches exactly what you provided via ApisixUpstream. So there is no issue at ingress controller level. Are you still running into the issue where unhealthy upstream is not excluded? Can you try again?

kworkbee · 2024-03-13T06:43:28Z

The following specific logs are visible in the APISIX Log.
Did SSL Handshake Failure Affect?

2024/03/13 15:42:17 [error] 50#50: *33439 [lua] healthcheck.lua:1383: log(): [healthcheck] (upstream#/apisix/upstreams/32eb11c7) failed SSL handshake with 'X.X.X.X (X.X.X.X:443)', using server name (sni) 'svc01.corp.com': 19: self-signed certificate in certificate chain, context: ngx.timer, client: X.X.X.X, server: 0.0.0.0:9080

Revolyssup · 2024-03-14T08:49:33Z

The following specific logs are visible in the APISIX Log. Did SSL Handshake Failure Affect?

2024/03/13 15:42:17 [error] 50#50: *33439 [lua] healthcheck.lua:1383: log(): [healthcheck] (upstream#/apisix/upstreams/32eb11c7) failed SSL handshake with 'X.X.X.X (X.X.X.X:443)', using server name (sni) 'svc01.corp.com': 19: self-signed certificate in certificate chain, context: ngx.timer, client: X.X.X.X, server: 0.0.0.0:9080

he active and passive health checks on upstreams work on HTTP codes but in case the HTTP connection is never established with the upstream, like in the case of SSL handshake failure then the upstream will not be excluded from the list of available upstreams. As the healthchecks wont work, the client will get 503.
This is a tcp connection failure, you can use something like healtheck.active.unhealthy.tcpFailures

Revolyssup · 2024-03-15T07:45:43Z

@kworkbee Did you try the above solution?

kworkbee · 2024-03-15T08:07:30Z

@Revolyssup It will take some time to check. I will mention it again as soon as I check. Thank you for your quick response.

kworkbee · 2024-03-18T02:35:47Z

@Revolyssup TCP Failures option also doesn't work. There is no special log in the ingress controller.

kworkbee · 2024-03-18T06:24:01Z

In APISIX Log 19: self-signed certificate in certificate chain I suspect this point.

Revolyssup · 2024-03-18T06:28:17Z

@Revolyssup TCP Failures option also doesn't work. There is no special log in the ingress controller.

What configuration did you use to add tcp healthcheck?

kworkbee · 2024-03-18T06:31:31Z

@Revolyssup TCP Failures option also doesn't work. There is no special log in the ingress controller.

What configuration did you use to add tcp healthcheck?

I set both healthcheck.active.unhealthy.tcpFailures and healthcheck.passive.unhealthy.tcpFailures to 1.

Revolyssup · 2024-03-18T14:21:05Z

@kworkbee Thats weird as you can see here that this error log corresponds to reporting a tcp healthcheck failure https://github.com/api7/lua-resty-healthcheck/blob/09a672e0784c75231ceca51e0a4cb39313c020f5/lib/resty/healthcheck.lua#L1078.
Can you send the configuration directly from admin API showing the exact configuration?

kworkbee · 2024-03-19T02:07:21Z

@kworkbee Thats weird as you can see here that this error log corresponds to reporting a tcp healthcheck failure https://github.com/api7/lua-resty-healthcheck/blob/09a672e0784c75231ceca51e0a4cb39313c020f5/lib/resty/healthcheck.lua#L1078. Can you send the configuration directly from admin API showing the exact configuration?

{
  "value": {
    "name": "apisix_feature-flags-nodes-upstream",
    "labels": {
      "k8slens-edit-resource-version": "v2",
      "meta_weight": "100",
      "managed-by": "apisix-ingress-controller",
      "argocd.argoproj.io/instance": "dev-tool-apisix"
    },
    "desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
    "type": "roundrobin",
    "id": "32eb11c7",
    "pass_host": "node",
    "nodes": [
      {
        "host": "svc01.corp.com",
        "weight": 50,
        "port": 443,
        "priority": 0
      },
      {
        "host": "svc02.corp.com",
        "weight": 50,
        "port": 443,
        "priority": 0
      }
    ],
    "update_time": 1710793944,
    "checks": {
      "active": {
        "http_path": "/",
        "https_verify_certificate": true,
        "port": 443,
        "type": "https",
        "timeout": 1,
        "healthy": {
          "http_statuses": [
            404
          ],
          "successes": 1,
          "interval": 3
        },
        "unhealthy": {
          "http_statuses": [
            503,
            504
          ],
          "http_failures": 1,
          "tcp_failures": 1,
          "timeouts": 3,
          "interval": 3
        },
        "concurrency": 10
      },
      "passive": {
        "healthy": {
          "http_statuses": [
            404
          ],
          "successes": 1
        },
        "unhealthy": {
          "http_statuses": [
            503,
            504
          ],
          "timeouts": 7,
          "tcp_failures": 1,
          "http_failures": 1
        },
        "type": "http"
      }
    },
    "create_time": 1710309045,
    "hash_on": "vars",
    "scheme": "https"
  },
  "key": "/apisix/upstreams/32eb11c7",
  "createdIndex": 151,
  "modifiedIndex": 538
}

kworkbee · 2024-03-20T10:56:51Z

In APISIX Log 19: self-signed certificate in certificate chain I suspect this point.

Related to this error above, any prerequisites needed to resolve? e.g. Using Admin SSL API to register ACM Certificate, ...?

kworkbee · 2024-03-28T05:41:33Z

The certificate problem was solved. It was a problem that occurred when terminated traffic went to Secure ALB. However, the ApisixUpstream health check still does not work.

@Revolyssup @shreemaan-abhishek

kworkbee changed the title ~~bug: Upstream HealthCheck Issue - Unhealthy Upstream won't be excluded temporarily~~ bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily Mar 7, 2024

kworkbee mentioned this issue Mar 8, 2024

bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily apache/apisix#11016

Open

Revolyssup self-assigned this Mar 12, 2024

bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily #2176

bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily #2176

Comments

kworkbee commented Mar 6, 2024 • edited

Current Behavior

Expected Behavior

Error Logs

Steps to Reproduce

Environment

kworkbee commented Mar 6, 2024 • edited

kworkbee commented Mar 7, 2024

kworkbee commented Mar 7, 2024 • edited

Revolyssup commented Mar 12, 2024

Revolyssup commented Mar 12, 2024

kworkbee commented Mar 12, 2024 • edited

Revolyssup commented Mar 13, 2024

kworkbee commented Mar 13, 2024

Revolyssup commented Mar 13, 2024

Revolyssup commented Mar 13, 2024

kworkbee commented Mar 13, 2024

kworkbee commented Mar 13, 2024

Revolyssup commented Mar 13, 2024

kworkbee commented Mar 13, 2024

Revolyssup commented Mar 13, 2024

Revolyssup commented Mar 13, 2024

kworkbee commented Mar 13, 2024 • edited

Revolyssup commented Mar 13, 2024

kworkbee commented Mar 13, 2024 • edited

Revolyssup commented Mar 13, 2024 • edited

Revolyssup commented Mar 13, 2024

kworkbee commented Mar 13, 2024

Revolyssup commented Mar 13, 2024

kworkbee commented Mar 13, 2024

Revolyssup commented Mar 13, 2024

kworkbee commented Mar 13, 2024

Revolyssup commented Mar 13, 2024

kworkbee commented Mar 13, 2024 • edited

Revolyssup commented Mar 14, 2024 • edited

Revolyssup commented Mar 15, 2024

kworkbee commented Mar 15, 2024

kworkbee commented Mar 18, 2024 • edited

kworkbee commented Mar 18, 2024

Revolyssup commented Mar 18, 2024

kworkbee commented Mar 18, 2024

Revolyssup commented Mar 18, 2024 • edited

kworkbee commented Mar 19, 2024

kworkbee commented Mar 20, 2024 • edited

kworkbee commented Mar 28, 2024

kworkbee commented Mar 6, 2024 •

edited

kworkbee commented Mar 6, 2024 •

edited

kworkbee commented Mar 7, 2024 •

edited

kworkbee commented Mar 12, 2024 •

edited

kworkbee commented Mar 13, 2024 •

edited

kworkbee commented Mar 13, 2024 •

edited

Revolyssup commented Mar 13, 2024 •

edited

kworkbee commented Mar 13, 2024 •

edited

Revolyssup commented Mar 14, 2024 •

edited

kworkbee commented Mar 18, 2024 •

edited

Revolyssup commented Mar 18, 2024 •

edited

kworkbee commented Mar 20, 2024 •

edited