The apisix ingress controller doesn't start as expected behind the VPN proxy

[thirumurthis]

Issue description

Deployed APISIX with latest version in Kind cluster. The apisix-ingress-controller instance throws exception and not able to create routes. This happens when I am connected to the VPN behind my official network.

The application works when the VPN is stopped. For some reason the ingress controller pod is not able to access the apisix-admin instance for routes.
I used a Nginx pod and was able to curl the routes with the api token. Is there any specific reason for this behavior?

Below is the error I see,

Defaulted container "ingress-controller" out of: ingress-controller, wait-apisix-admin (init)
2024-05-03T08:54:18+08:00       info    ingress/ingress.go:128  init apisix ingress controller
2024-05-03T08:54:18+08:00       info    ingress/ingress.go:129  version:
Version: 1.8.0
Git SHA: no-git-module
Go Version: go1.20.14
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64

2024-05-03T08:54:18+08:00       info    ingress/ingress.go:139  use configuration
{
  "cert_file": "/etc/webhook/certs/cert.pem",
  "key_file": "/etc/webhook/certs/key.pem",
  "log_level": "info",
  "log_output": "stderr",
  "log_rotate_output_path": "",
  "log_rotation_max_size": 100,
  "log_rotation_max_age": 0,
  "log_rotation_max_backups": 0,
  "http_listen": ":8080",
  "https_listen": ":8443",
  "ingress_publish_service": "",
  "ingress_status_address": [],
  "enable_profiling": true,
  "kubernetes": {
    "kubeconfig": "",
    "resync_interval": "6h0m0s",
    "namespace_selector": [],
    "election_id": "ingress-apisix-leader",
    "ingress_class": "apisix",
    "ingress_version": "networking/v1",
    "watch_endpoint_slices": false,
    "api_version": "apisix.apache.org/v2",
    "enable_gateway_api": false,
    "disable_status_updates": false,
    "enable_admission": false
  },
  "apisix": {
    "admin_api_version": "v3",
    "default_cluster_name": "apisix-zitadel",
    "default_cluster_base_url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin",
    "default_cluster_admin_key": "******"
  },
  "apisix_resource_sync_interval": "1h0m0s",
  "apisix_resource_sync_comparison": true,
  "plugin_metadata_cm": "",
  "etcdserver": {
    "enabled": false,
    "prefix": "/apisix",
    "listen_address": ":12379",
    "ssl_key_encrypt_salt": "edd1c9f0985e76a2"
  }
}
2024-05-03T08:54:18+08:00       info    providers/controller.go:169     start leader election
2024-05-03T08:54:18+08:00       info    providers/controller.go:154     start api server
I0503 08:54:18.279238       1 leaderelection.go:250] attempting to acquire leader lease ingress-apisix/ingress-apisix-leader...
2024-05-03T08:54:18+08:00       info    providers/controller.go:143     LeaderElection  {"message": "apisix-ingress-controller-f4cfccb69-j22lv became leader", "event_type": "Normal"}
2024-05-03T08:54:18+08:00       warn    providers/controller.go:219     found a new leader apisix-ingress-controller-f4cfccb69-j22lv
I0503 08:54:18.286094       1 leaderelection.go:260] successfully acquired lease ingress-apisix/ingress-apisix-leader
2024-05-03T08:54:18+08:00       info    providers/controller.go:211     controller now is running as leader     {"namespace": "ingress-apisix", "pod": "apisix-ingress-controller-f4cfccb69-j22lv"}
2024-05-03T08:54:18+08:00       info    providers/controller.go:386     controller tries to leading ... {"namespace": "ingress-apisix", "pod": "apisix-ingress-controller-f4cfccb69-j22lv"}
2024-05-03T08:54:21+08:00       warn    apisix/cluster.go:423   waiting cluster apisix-zitadel to ready, it may takes a while
2024-05-03T08:54:21+08:00       info    apisix/cluster.go:248   syncing cache   {"cluster": "apisix-zitadel"}
2024-05-03T08:54:21+08:00       info    apisix/cluster.go:463   syncing schema  {"cluster": "apisix-zitadel"}
2024-05-03T08:54:24+08:00       error   apisix/route.go:90      failed to list routes: Get "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes": dial tcp: lookup apisix-admin.ingress-apisix.svc.cluster.local: i/o timeout
2024-05-03T08:54:24+08:00       error   apisix/cluster.go:298   failed to list routes in APISIX: Get "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes": dial tcp: lookup apisix-admin.ingress-apisix.svc.cluster.local: i/o timeout
2024-05-03T08:54:24+08:00       error   apisix/plugin.go:46     failed to list plugins' names: Get "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/plugins?all=true": dial tcp: lookup apisix-admin.ingress-apisix.svc.cluster.local: i/o timeout
2024-05-03T08:54:24+08:00       error   apisix/cluster.go:483   failed to list plugin names in APISIX: Get "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/plugins?all=true": dial tcp: lookup apisix-admin.ingress-apisix.svc.cluster.local: i/o timeout
2024-05-03T08:54:24+08:00       error   apisix/cluster.go:446   failed to sync schema: Get "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/plugins?all=true": dial tcp: lookup apisix-admin.ingress-apisix.svc.cluster.local: i/o timeout

Environment

Using kind cluster

• apisix-ingress-controller version: v1.8.0 (used helm chart latest 0.14.0
• your Kubernetes cluster version (output of kubectl version):

 Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.2

• if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a):
  Used WSL2
  Linux wsl-local 5.15.150.1-microsoft-standard-WSL2 # 1 SMP Thu Mar 7 03:22:57 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux