Addressing 'Upstream Not Found' Warning in APISIX

[rkazak07]

Issue description

Hello,
I installed Apisix. Even though I did not set up anything inside , I always get the following warnings. When I examined it in Debug mode, I saw that it was plugged into the Rabbitmq service ports running inside. I added these ports as upstream. Warnings have been stopped, but it sometimes sends warnings for different applications like redis, kafka... How can I proceed with the solution?
When I include the application with Apisix ingress, it works stably.

Note: I installed Apisix as daemonset

2024-04-07T18:36:43+08:00 warn apisix/cluster.go:1096 route not found {"id": "fb148866", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/fb148866", "cluster": "default"} 2024-04-07T19:54:01+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "8c0a7c8f", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/8c0a7c8f", "cluster": "default"} 2024-04-07T19:54:01+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "fb0d4c19", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/fb0d4c19", "cluster": "default"} 2024-04-07T14:59:10.711119455+03:00 2024-04-07T19:59:10+08:00 warn apisix/cluster.go:1096 route not found {"id": "1be8a9cd", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/1be8a9cd", "cluster": "default"} 2024-04-07T20:00:36+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "8c0a7c8f", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/8c0a7c8f", "cluster": "default"} 2024-04-07T15:00:36.322330526+03:00 2024-04-07T20:00:36+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "fb0d4c19", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/fb0d4c19", "cluster": "default"} 2024-04-07T20:00:37+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "b785aa59", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/b785aa59", "cluster": "default"} 2024-04-07T20:00:37+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "c0829acf", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/c0829acf", "cluster": "default"} 2024-04-07T20:01:43+08:00 warn apisix/cluster.go:1096 route not found {"id": "19221c15", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/19221c15", "cluster": "default"} 2024-04-07T15:02:43.559923042+03:00 2024-04-07T20:02:43+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "8c0a7c8f", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/8c0a7c8f", "cluster": "default"} 2024-04-07T15:02:43.719498269+03:00 2024-04-07T20:02:43+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "fb0d4c19", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/fb0d4c19", "cluster": "default"} 2024-04-07T15:02:45.045476422+03:00 2024-04-07T20:02:45+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "b785aa59", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/b785aa59", "cluster": "default"} 2024-04-07T20:02:45+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "c0829acf", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/c0829acf", "cluster": "default"} 2024-04-07T15:03:15.520423951+03:00 2024-04-07T20:03:15+08:00 warn apisix/cluster.go:1096 route not found {"id": "6dd4f7e8", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/6dd4f7e8", "cluster": "default"} 2024-04-07T15:04:15.849063365+03:00 2024-04-07T20:04:15+08:00 error ingress/ingress.go:527 failed to record status change for IngressV1 {"error": "Operation cannot be fulfilled on ingresses.networking.k8s.io \"bff-tapp\": the object has been modified; please apply your changes to the latest version and try again", "name": "bff-tapp", "namespace": "dev-bff"} 2024-04-07T20:04:15+08:00 warn apisix/cluster.go:1096 route not found {"id": "eec00fc1", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/eec00fc1", "cluster": "default"} 2024-04-07T20:04:53+08:00 warn apisix/cluster.go:1096 route not found {"id": "beecb92d", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/beecb92d", "cluster": "default"} 2024-04-07T15:05:49.522312841+03:00 2024-04-07T20:05:49+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "b785aa59", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/b785aa59", "cluster": "default"} 2024-04-07T20:05:49+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "c0829acf", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/c0829acf", "cluster": "default"} 2024-04-07T15:06:46.130431720+03:00 2024-04-07T20:06:46+08:00 error ingress/ingress.go:527 failed to record status change for IngressV1 {"error": "Operation cannot be fulfilled on ingresses.networking.k8s.io \"bff-appd\": the object has been modified; please apply your changes to the latest version and try again", "name": "bff-appd", "namespace": "dev-bff"} 2024-04-07T15:06:46.168746503+03:00 2024-04-07T20:06:46+08:00 warn apisix/cluster.go:1096 route not found {"id": "37e34cb7", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/37e34cb7", "cluster": "default"} 2024-04-07T15:08:49.527680464+03:00 2024-04-07T20:08:49+08:00 error ingress/ingress.go:527 failed to record status change for IngressV1 {"error": "Operation cannot be fulfilled on ingresses.networking.k8s.io \"ui-atapp\": the object has been modified; please apply your changes to the latest version and try again", "name": "ui-atapp", "namespace": "dev-ui"} 2024-04-07T15:08:49.739153847+03:00 2024-04-07T20:08:49+08:00 warn apisix/cluster.go:1096 route not found {"id": "70110b8a", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/70110b8a", "cluster": "default"} 2024-04-07T15:11:04.884775966+03:00 2024-04-07T20:11:04+08:00 error ingress/ingress.go:527 failed to record status change for IngressV1 {"error": "Operation cannot be fulfilled on ingresses.networking.k8s.io \"ui-tapp\": the object has been modified; please apply your changes to the latest version and try again", "name": "ui-tapp", "namespace": "dev-ui"} 2024-04-07T15:11:04.928498466+03:00 2024-04-07T20:11:04+08:00 warn apisix/cluster.go:1096 route not found {"id": "e3bcc2f5", "url": "http://apisix-ad min.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/e3bcc2f5", "cluster": "default"} 2024-04-07T15:13:47.631911060+03:00 2024-04-07T20:13:47+08:00 error ingress/ingress.go:527 failed to record status change for IngressV1 {"error": "Operation cannot be fulfilled on ingresses.networking.k8s.io \"security-idp-capp\": the object has been modified; please apply your changes to the latest version and try again", "name": "security-idp-capp", "namespace": "dev-security"} 2024-04-07T20:13:47+08:00 warn apisix/cluster.go:1096 route not found {"id": "b26166dd", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/b26166dd", "cluster": "default"} 2024-04-07T15:15:19.907450226+03:00 2024-04-07T20:15:19+08:00 error ingress/ingress.go:527 failed to record status change for IngressV1 {"error": "Operation cannot be fulfilled on ingresses.networking.k8s.io \"security-idp-papp\": the object has been modified; please apply your changes to the latest version and try again", "name": "security-idp-papp", "namespace": "dev-security"} 2024-04-07T15:16:26.397665034+03:00 2024-04-07T20:16:26+08:00 warn apisix/cluster.go:1096 route not found {"id": "321f3c87", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/321f3c87", "cluster": "default"} 2024-04-07T15:22:35.869151820+03:00 2024-04-07T20:22:35+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "8c0a7c8f", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/8c0a7c8f", "cluster": "default"} 2024-04-07T15:22:35.888250101+03:00 2024-04-07T20:22:35+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "fb0d4c19", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/fb0d4c19", "cluster": "default"} 2024-04-07T20:31:10+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "8c0a7c8f", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/8c0a7c8f", "cluster": "default"} 2024-04-07T20:31:10+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "fb0d4c19", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/fb0d4c19", "cluster": "default"} 2024-04-07T20:33:18+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "8c0a7c8f", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/8c0a7c8f", "cluster": "default"} 2024-04-07T20:33:18+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "fb0d4c19", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/fb0d4c19", "cluster": "default"}

Thanks,

Environment

Apisix: 3.8.0
Apisix Ingress Controller: 1.8.0
Apisix Dashboard: 3.0.0
K8S Version: 1.27.10