SpiderPool Controller Crash On k8s 1.29 (k0s installer) #3204

infinitydon · 2024-02-07T00:43:01Z

Spiderpool Version

chart spiderpool-0.9.1

Bug Type

Installation Issue

Main CNI

kube-router

What happened?

Tried to install spiderpool using the following commands:

helm repo add spiderpool https://spidernet-io.github.io/spiderpool
helm repo update spiderpool
helm install spiderpool spiderpool/spiderpool --namespace kube-system --set ipam.enableStatefulSet=true

spirderpool controller pod keeps crashing with the following error logs:

kubectl get no -o wide
NAME                 STATUS   ROLES           AGE   VERSION       INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
talos-mgmt-cluster   Ready    control-plane   32m   v1.29.1+k0s   192.168.100.59   <none>        Ubuntu 20.04.6 LTS   5.4.0-170-generic   containerd://1.7.13

kubectl -n kube-system get po
NAME                                     READY   STATUS             RESTARTS      AGE
kube-proxy-cgkfw                         1/1     Running            0             29m
kube-router-znjd5                        1/1     Running            0             29m
coredns-6cd46fb86c-wcrtl                 1/1     Running            0             29m
metrics-server-7556957bb7-6m97x          1/1     Running            0             29m
metallb-controller-648b76f565-ss79k      1/1     Running            0             17m
metallb-speaker-zrmxv                    4/4     Running            0             17m
spiderpool-agent-ktb4n                   1/1     Running            0             13m
multus-cni-whbrh                         1/1     Running            0             10m
spiderpool-init                          0/1     Error              0             13m
spiderpool-controller-786774fdc4-tq97q   0/1     CrashLoopBackOff   7 (64s ago)   13m

kubectl -n kube-system logs spiderpool-controller-786774fdc4-tq97q
{"level":"INFO","ts":"2024-02-07T00:34:13.014Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:57","msg":"CommitVersion: 32c8359256d008cbd73e04757f8085887de1aec2"}
{"level":"INFO","ts":"2024-02-07T00:34:13.014Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:60","msg":"CommitTime: 2024-01-29T17:37:04+08:00"}
{"level":"INFO","ts":"2024-02-07T00:34:13.014Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:63","msg":"AppVersion: v0.9.1"}
{"level":"INFO","ts":"2024-02-07T00:34:13.014Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:68","msg":"Default max golang procs: 8"}
{"level":"INFO","ts":"2024-02-07T00:34:13.014Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:78","msg":"Spiderpool-controller config: {CommitVersion:32c8359256d008cbd73e04757f8085887de1aec2 CommitTime:2024-01-29T17:37:04+08:00 AppVersion:v0.9.1 GoMaxProcs:8 ConfigPath:/tmp/spiderpool/config-map/conf.yml TlsServerCertPath:/etc/tls/tls.crt TlsServerKeyPath:/etc/tls/tls.key LogLevel:info EnableMetric:false EnableDebugLevelMetric:false MetricRenewPeriod:120 HttpPort:5720 MetricHttpPort:5721 WebhookPort:5722 GopsListenPort:5724 PyroscopeAddress: DefaultCniConfDir:/etc/cni/net.d CiliumConfigName:kube-system/cilium-config ControllerPodNamespace:kube-system ControllerPodName:spiderpool-controller-786774fdc4-tq97q LeaseDuration:15 LeaseRenewDeadline:10 LeaseRetryPeriod:2 LeaseRetryGap:1 IPPoolMaxAllocatedIPs:5000 SubnetInformerResyncPeriod:300 SubnetInformerWorkers:5 SubnetInformerMaxWorkqueueLength:10000 SubnetAppControllerWorkers:5 IPPoolInformerResyncPeriod:300 IPPoolInformerWorkers:3 IPPoolInformerMaxWorkQueueLength:10000 WorkQueueMaxRetries:500 WorkQueueRequeueDelayDuration:5 CoordinatorInformerResyncPeriod:60 EnableMultusConfig:true MultusConfigInformerResyncPeriod:60 EnableIPv4:true EnableIPv6:true EnableStatefulSet:true EnableKubevirtStaticIP:true EnableSpiderSubnet:true ClusterSubnetDefaultFlexibleIPNum:1}"}
{"level":"INFO","ts":"2024-02-07T00:34:13.066Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:91","msg":"gops is listening on 127.0.0.1:5724"}
{"level":"INFO","ts":"2024-02-07T00:34:13.066Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:120","msg":"Begin to initialize spiderpool-controller metrics HTTP server"}
{"level":"INFO","ts":"2024-02-07T00:34:13.066Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:123","msg":"Begin to initialize spiderpool-controller runtime manager"}
{"level":"INFO","ts":"2024-02-07T00:34:13.075Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:255","msg":"Begin to initialize StatefulSet manager"}
{"level":"INFO","ts":"2024-02-07T00:34:13.075Z","logger":"Lease-Lock-Election","caller":"election/lease_election.go:163","msg":"'kube-system/spiderpool-controller-leases/spiderpool-controller-786774fdc4-tq97q' is trying to elect"}
I0207 00:34:13.075737       1 leaderelection.go:250] attempting to acquire leader lease kube-system/spiderpool-controller-leases...
{"level":"INFO","ts":"2024-02-07T00:34:13.076Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:151","msg":"Starting spiderpool-controller runtime manager"}
I0207 00:34:13.083544       1 leaderelection.go:260] successfully acquired lease kube-system/spiderpool-controller-leases
{"level":"INFO","ts":"2024-02-07T00:34:13.083Z","logger":"Lease-Lock-Election","caller":"election/lease_election.go:133","msg":"leader elected: kube-system/spiderpool-controller-leases/spiderpool-controller-786774fdc4-tq97q"}
{"level":"INFO","ts":"2024-02-07T00:34:13.177Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:161","msg":"Begin to initialize OpenAPI HTTP server"}
{"level":"INFO","ts":"2024-02-07T00:34:13.180Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:177","msg":"Begin to initialize IP GC Manager"}
{"level":"INFO","ts":"2024-02-07T00:34:13.181Z","logger":"spiderpool-controller","caller":"cmd/http_server.go:28","msg":"Serving spiderpool controller API at http://[[::]:5720]"}
{"level":"INFO","ts":"2024-02-07T00:34:13.182Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:180","msg":"Set spiderpool-controller Startup probe ready"}
{"level":"INFO","ts":"2024-02-07T00:34:13.183Z","logger":"IP-GarbageCollection","caller":"gcmanager/pod_informer.go:17","msg":"try to register pod informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.183Z","logger":"IP-GarbageCollection","caller":"gcmanager/gc_manager.go:160","msg":"running IP garbage collection"}
{"level":"INFO","ts":"2024-02-07T00:34:13.183Z","logger":"IP-GarbageCollection","caller":"gcmanager/pod_informer.go:49","msg":"create Pod informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.183Z","caller":"limiter/queue.go:135","msg":"Starting limiter"}
{"level":"INFO","ts":"2024-02-07T00:34:13.183Z","logger":"IP-GarbageCollection","caller":"gcmanager/tracePod_worker.go:74","msg":"Starting running 'releaseIPPoolIPExecutor'","IPPoolIP_Worker":1}
{"level":"INFO","ts":"2024-02-07T00:34:13.183Z","logger":"IP-GarbageCollection","caller":"gcmanager/tracePod_worker.go:74","msg":"Starting running 'releaseIPPoolIPExecutor'","IPPoolIP_Worker":3}
{"level":"INFO","ts":"2024-02-07T00:34:13.183Z","logger":"IP-GarbageCollection","caller":"gcmanager/scanAll_IPPool.go:58","msg":"receive CLI GC request, execute scan all right now!"}
{"level":"INFO","ts":"2024-02-07T00:34:13.183Z","logger":"IP-GarbageCollection","caller":"gcmanager/tracePod_worker.go:74","msg":"Starting running 'releaseIPPoolIPExecutor'","IPPoolIP_Worker":2}
{"level":"INFO","ts":"2024-02-07T00:34:13.183Z","logger":"IP-GarbageCollection","caller":"gcmanager/tracePod_worker.go:28","msg":"starting trace pod worker"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:461","msg":"Begin to set up Coordinator informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:474","msg":"Begin to set up IPPool informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"SpiderIPPool-Informer","caller":"ippoolmanager/ippool_informer.go:77","msg":"try to register SpiderIPPool informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:494","msg":"Begin to set up Subnet informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:507","msg":"Begin to set up auto-created IPPool controller"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"SpiderSubnet-Application-Controllers","caller":"applicationcontroller/app_controller.go:106","msg":"try to register SpiderSubnet App informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"spiderpool-controller","caller":"cmd/daemon.go:532","msg":"Begin to set up MultusConfig informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"SpiderIPPool-Informer","caller":"ippoolmanager/ippool_informer.go:109","msg":"create SpiderIPPool informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"MultusConfig-Informer","caller":"multuscniconfig/multusconfig_informer.go:73","msg":"try to register MultusConfig informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"MultusConfig-Informer","caller":"multuscniconfig/multusconfig_informer.go:105","msg":"create MultusConfig informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"Subnet-Informer","caller":"subnetmanager/subnet_informer.go:125","msg":"Initialize Dynamic informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"Coordinator-Informer","caller":"coordinatormanager/coordinator_informer.go:144","msg":"Initialize Coordinator informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"SpiderSubnet-Application-Controllers","caller":"applicationcontroller/app_controller.go:138","msg":"create SpiderSubnet App informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"Subnet-Informer","caller":"subnetmanager/subnet_informer.go:129","msg":"Initialize Subnet informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"SpiderSubnet-Application-Controllers","caller":"applicationinformers/deployment_informer.go:15","msg":"Setting up Deployment handlers"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"SpiderSubnet-Application-Controllers","caller":"applicationinformers/replicaset_informer.go:15","msg":"Setting up ReplicaSet handlers"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"SpiderSubnet-Application-Controllers","caller":"applicationinformers/statefulset_informer.go:15","msg":"Setting up StatefulSet handlers"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"SpiderSubnet-Application-Controllers","caller":"applicationinformers/daemonset_informer.go:15","msg":"Setting up DaemonSet handlers"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"SpiderSubnet-Application-Controllers","caller":"applicationinformers/job_informer.go:15","msg":"Setting up Job informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"SpiderSubnet-Application-Controllers","caller":"applicationinformers/cronjob_informer.go:15","msg":"Setting up CronJob handlers"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"Subnet-Informer","caller":"subnetmanager/subnet_informer.go:250","msg":"Starting Subnet informer"}
{"level":"INFO","ts":"2024-02-07T00:34:13.189Z","logger":"Subnet-Informer","caller":"subnetmanager/subnet_informer.go:252","msg":"Waiting for informer caches to sync"}
I0207 00:34:13.189663       1 shared_informer.go:311] Waiting for caches to sync for SpiderCoordinator
I0207 00:34:13.189900       1 shared_informer.go:311] Waiting for caches to sync for SpiderSubnet
I0207 00:34:13.290581       1 shared_informer.go:318] Caches are synced for SpiderSubnet
I0207 00:34:13.290743       1 shared_informer.go:318] Caches are synced for SpiderCoordinator
{"level":"INFO","ts":"2024-02-07T00:34:13.290Z","logger":"Subnet-Informer","caller":"subnetmanager/subnet_informer.go:257","msg":"Starting workers"}
{"level":"INFO","ts":"2024-02-07T00:34:13.290Z","logger":"Subnet-Informer","caller":"subnetmanager/subnet_informer.go:262","msg":"Starting dynamic informer worker"}
{"level":"INFO","ts":"2024-02-07T00:34:13.290Z","logger":"Subnet-Informer","caller":"subnetmanager/subnet_informer.go:265","msg":"Started workers"}
{"level":"INFO","ts":"2024-02-07T00:34:13.291Z","logger":"Coordinator-Informer","caller":"coordinatormanager/coordinator_informer.go:352","msg":"spidercoordinator change podCIDRType from auto to none","CoordinatorName":"default","Operation":"PROCESS"}
{"level":"WARN","ts":"2024-02-07T00:34:13.291Z","logger":"Coordinator-Informer","caller":"coordinatormanager/coordinator_informer.go:363","msg":"kube-system/kubeadm-config is no found, trying to fetch the ClusterCIDR from kube-controller-manager Pod","CoordinatorName":"default","Operation":"PROCESS"}
E0207 00:34:13.299179       1 runtime.go:79] Observed a panic: runtime.boundsError{x:0, y:0, signed:true, code:0x0} (runtime error: index out of range [0] with length 0)
goroutine 326 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2228240?, 0xc0006cfc68})
        /src/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000035780?})
        /src/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
panic({0x2228240, 0xc0006cfc68})
        /usr/local/go/src/runtime/panic.go:884 +0x213
github.com/spidernet-io/spiderpool/pkg/coordinatormanager.(*CoordinatorController).fetchPodAndServerCIDR(0xc00041a000, {0x2729bb8, 0xc0007a7500}, 0xc00059a380, 0xc001586000)
        /src/pkg/coordinatormanager/coordinator_informer.go:377 +0x9c6
github.com/spidernet-io/spiderpool/pkg/coordinatormanager.(*CoordinatorController).syncHandler(0xc00041a000, {0x2729bb8, 0xc0007a7500}, {0xc000336ec6, 0x7})
        /src/pkg/coordinatormanager/coordinator_informer.go:319 +0x17b
github.com/spidernet-io/spiderpool/pkg/coordinatormanager.(*CoordinatorController).processNextWorkItem(0xc00041a000, {0x2729bb8, 0xc0006c49f0})
        /src/pkg/coordinatormanager/coordinator_informer.go:292 +0x310
github.com/spidernet-io/spiderpool/pkg/coordinatormanager.(*CoordinatorController).runWorker(...)
        /src/pkg/coordinatormanager/coordinator_informer.go:276
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
        /src/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:259 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /src/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x1ec0120?, {0x2709ea0, 0xc0007a74a0}, 0x1, 0xc0006c6060)
        /src/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0001b58f0?, 0x3b9aca00, 0x0, 0x0?, 0xc00011bef0?)
        /src/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x2729bb8, 0xc0006c49f0}, 0xc00011e950, 0xc00051f260?, 0xc00051f5f0?, 0x60?)
        /src/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:259 +0x99
k8s.io/apimachinery/pkg/util/wait.UntilWithContext({0x2729bb8?, 0xc0006c49f0?}, 0xc000457c50?, 0xc000457e60?)
        /src/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:170 +0x2b
created by github.com/spidernet-io/spiderpool/pkg/coordinatormanager.(*CoordinatorController).run
        /src/pkg/coordinatormanager/coordinator_informer.go:267 +0x170
panic: runtime error: index out of range [0] with length 0 [recovered]
        panic: runtime error: index out of range [0] with length 0

goroutine 326 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000035780?})
        /src/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xd7
panic({0x2228240, 0xc0006cfc68})
        /usr/local/go/src/runtime/panic.go:884 +0x213
github.com/spidernet-io/spiderpool/pkg/coordinatormanager.(*CoordinatorController).fetchPodAndServerCIDR(0xc00041a000, {0x2729bb8, 0xc0007a7500}, 0xc00059a380, 0xc001586000)
        /src/pkg/coordinatormanager/coordinator_informer.go:377 +0x9c6
github.com/spidernet-io/spiderpool/pkg/coordinatormanager.(*CoordinatorController).syncHandler(0xc00041a000, {0x2729bb8, 0xc0007a7500}, {0xc000336ec6, 0x7})
        /src/pkg/coordinatormanager/coordinator_informer.go:319 +0x17b
github.com/spidernet-io/spiderpool/pkg/coordinatormanager.(*CoordinatorController).processNextWorkItem(0xc00041a000, {0x2729bb8, 0xc0006c49f0})
        /src/pkg/coordinatormanager/coordinator_informer.go:292 +0x310
github.com/spidernet-io/spiderpool/pkg/coordinatormanager.(*CoordinatorController).runWorker(...)
        /src/pkg/coordinatormanager/coordinator_informer.go:276
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
        /src/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:259 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /src/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x1ec0120?, {0x2709ea0, 0xc0007a74a0}, 0x1, 0xc0006c6060)
        /src/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0001b58f0?, 0x3b9aca00, 0x0, 0x0?, 0xc00011bef0?)
        /src/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x2729bb8, 0xc0006c49f0}, 0xc00011e950, 0xc00051f260?, 0xc00051f5f0?, 0x60?)
        /src/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:259 +0x99
k8s.io/apimachinery/pkg/util/wait.UntilWithContext({0x2729bb8?, 0xc0006c49f0?}, 0xc000457c50?, 0xc000457e60?)
        /src/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:170 +0x2b
created by github.com/spidernet-io/spiderpool/pkg/coordinatormanager.(*CoordinatorController).run
        /src/pkg/coordinatormanager/coordinator_informer.go:267 +0x170

What did you expect to happen?

All spiderpool pods should be up and running without crashes

How to reproduce it (as minimally and precisely as possible)

Install k8s cluster using k0s 1.29:

curl -sSLf https://get.k0s.sh | sudo sh
sudo k0s install controller --single
sudo k0s start
sudo k0s status

sudo k0s kubeconfig admin > ~/.kube/config

Install spiderpool

helm repo add spiderpool https://spidernet-io.github.io/spiderpool
helm repo update spiderpool
helm install spiderpool spiderpool/spiderpool --namespace kube-system --set ipam.enableStatefulSet=true

Additional Context

No response

The text was updated successfully, but these errors were encountered:

cyclinder · 2024-02-07T03:27:35Z

Could you also show the logs of the spiderpool-init？ I think it is a bug, I will fix it later.

infinitydon · 2024-02-07T12:46:59Z

Looks like the spiderpool-init also depends on the controller been up:

kubectl -n kube-system logs spiderpool-init
{"level":"INFO","ts":"2024-02-07T00:27:31.072Z","logger":"Spiderpool-Init","caller":"cmd/config.go:211","msg":"Ignore creating default IPv4 Subnet or IPPool"}
{"level":"INFO","ts":"2024-02-07T00:27:31.072Z","logger":"Spiderpool-Init","caller":"cmd/config.go:250","msg":"Ignore creating default IPv6 Subnet or IPPool"}
{"level":"INFO","ts":"2024-02-07T00:27:31.072Z","logger":"Spiderpool-Init","caller":"cmd/config.go:301","msg":"Init default config: {Namespace:kube-system ControllerName:spiderpool-controller AgentName:spiderpool-agent CoordinatorName:default CoordinatorMode:auto CoordinatorPodCIDRType:auto CoordinatorPodDefaultRouteNic: CoordinatorPodMACPrefix: CoordinatorDetectGateway:false CoordinatorDetectIPConflict:false CoordinatorTunePodRoutes:true CoordinatorHijackCIDR:[169.254.0.0/16] V4SubnetName: V4IPPoolName: V4CIDR: V4IPRanges:[] V4Gateway: V6SubnetName: V6IPPoolName: V6CIDR: V6IPRanges:[] V6Gateway: enableMultusConfig:true installMultusCNI:true DefaultCNIDir:/etc/cni/net.d DefaultCNIName: DefaultCNINamespace:kube-system MultusConfigMap:spiderpool-multus ReadinessFile:/etc/spiderpool/ready}"}
{"level":"INFO","ts":"2024-02-07T00:27:31.169Z","logger":"Spiderpool-Init","caller":"cmd/root.go:38","msg":"Try to create default Coordinator default"}
{"level":"INFO","ts":"2024-02-07T00:27:32.182Z","caller":"cmd/client.go:68","msg":"Failed to create default Coordinator default, recreate in 2s: Internal error occurred: failed calling webhook \"spidercoordinator.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidercoordinator?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:27:35.190Z","caller":"cmd/client.go:68","msg":"Failed to create default Coordinator default, recreate in 2s: Internal error occurred: failed calling webhook \"spidercoordinator.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidercoordinator?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:27:38.198Z","caller":"cmd/client.go:68","msg":"Failed to create default Coordinator default, recreate in 2s: Internal error occurred: failed calling webhook \"spidercoordinator.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidercoordinator?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:27:41.206Z","caller":"cmd/client.go:68","msg":"Failed to create default Coordinator default, recreate in 2s: Internal error occurred: failed calling webhook \"spidercoordinator.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidercoordinator?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:27:44.214Z","caller":"cmd/client.go:68","msg":"Failed to create default Coordinator default, recreate in 2s: Internal error occurred: failed calling webhook \"spidercoordinator.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidercoordinator?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:27:47.226Z","caller":"cmd/client.go:68","msg":"Failed to create default Coordinator default, recreate in 2s: Internal error occurred: failed calling webhook \"spidercoordinator.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidercoordinator?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:27:50.262Z","caller":"cmd/client.go:68","msg":"Failed to create default Coordinator default, recreate in 2s: Internal error occurred: failed calling webhook \"spidercoordinator.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidercoordinator?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:27:53.270Z","caller":"cmd/client.go:68","msg":"Failed to create default Coordinator default, recreate in 2s: Internal error occurred: failed calling webhook \"spidercoordinator.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidercoordinator?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:27:56.278Z","caller":"cmd/client.go:68","msg":"Failed to create default Coordinator default, recreate in 2s: Internal error occurred: failed calling webhook \"spidercoordinator.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidercoordinator?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:27:59.286Z","caller":"cmd/client.go:68","msg":"Failed to create default Coordinator default, recreate in 2s: Internal error occurred: failed calling webhook \"spidercoordinator.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidercoordinator?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:02.294Z","caller":"cmd/client.go:68","msg":"Failed to create default Coordinator default, recreate in 2s: Internal error occurred: failed calling webhook \"spidercoordinator.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidercoordinator?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:05.302Z","caller":"cmd/client.go:68","msg":"Failed to create default Coordinator default, recreate in 2s: Internal error occurred: failed calling webhook \"spidercoordinator.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidercoordinator?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:08.310Z","caller":"cmd/client.go:68","msg":"Failed to create default Coordinator default, recreate in 2s: Internal error occurred: failed calling webhook \"spidercoordinator.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidercoordinator?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:11.318Z","caller":"cmd/client.go:68","msg":"Failed to create default Coordinator default, recreate in 2s: Internal error occurred: failed calling webhook \"spidercoordinator.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidercoordinator?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:14.326Z","caller":"cmd/client.go:68","msg":"Failed to create default Coordinator default, recreate in 2s: Internal error occurred: failed calling webhook \"spidercoordinator.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidercoordinator?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:16.344Z","caller":"cmd/client.go:54","msg":"Succeed to create default Coordinator: {TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:default GenerateName: Namespace: SelfLink: UID:caa98b8d-4e4f-45cb-bf43-ca9d5e27d296 ResourceVersion:1248 Generation:1 CreationTimestamp:2024-02-07 00:28:16 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[] Finalizers:[spiderpool.spidernet.io] ManagedFields:[{Manager:spiderpool-init Operation:Update APIVersion:spiderpool.spidernet.io/v2beta1 Time:2024-02-07 00:28:16 +0000 UTC FieldsType:FieldsV1 FieldsV1:{\"f:spec\":{\".\":{},\"f:detectGateway\":{},\"f:detectIPConflict\":{},\"f:hijackCIDR\":{},\"f:hostRPFilter\":{},\"f:hostRuleTable\":{},\"f:mode\":{},\"f:podCIDRType\":{},\"f:podDefaultRouteNIC\":{},\"f:podMACPrefix\":{},\"f:tunePodRoutes\":{},\"f:txQueueLen\":{}}} Subresource:}]} Spec:{Mode:0xc000199a40 PodCIDRType:0xc000199a50 HijackCIDR:[169.254.0.0/16] PodMACPrefix:0xc000199a70 TunePodRoutes:0xc0002a8f14 PodDefaultRouteNIC:0xc000199a60 HostRuleTable:0xc0002a8f00 HostRPFilter:0xc0002a8ee8 TxQueueLen:0xc0002a8f18 DetectIPConflict:0xc0002a8ee5 DetectGateway:0xc0002a8ee4} Status:{Phase: OverlayPodCIDR:[] ServiceCIDR:[]}}"}
{"level":"INFO","ts":"2024-02-07T00:28:16.344Z","logger":"Spiderpool-Init","caller":"cmd/config.go:314","msg":"the first cni config file is /etc/cni/net.d/10-kuberouter.conflist in /etc/cni/net.d"}
[controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
Detected at:
        >  goroutine 1 [running]:
        >  runtime/debug.Stack()
        >       /usr/local/go/src/runtime/debug/stack.go:24 +0x65
        >  sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
        >       /src/vendor/sigs.k8s.io/controller-runtime/pkg/log/log.go:60 +0xcd
        >  sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Enabled(0xc000135080, 0x4a00000000000000?)
        >       /src/vendor/sigs.k8s.io/controller-runtime/pkg/log/deleg.go:111 +0x3d
        >  github.com/go-logr/logr.Logger.Info({{0x1b23d48?, 0xc000135080?}, 0xc0003a2300?}, {0xc0003a2300, 0x93}, {0x0, 0x0, 0x0})
        >       /src/vendor/github.com/go-logr/logr/logr.go:280 +0x78
        >  sigs.k8s.io/controller-runtime/pkg/log.(*KubeAPIWarningLogger).HandleWarningHeader(0xc0005152f0?, 0x1?, {0x18af996?, 0x7?}, {0xc0003a2300?, 0x93?})
        >       /src/vendor/sigs.k8s.io/controller-runtime/pkg/log/warning_handler.go:65 +0x170
        >  k8s.io/client-go/rest.handleWarnings(0xc0003f6ea0?, {0x1b0b800?, 0xc0005152f0?})
        >       /src/vendor/k8s.io/client-go/rest/warnings.go:144 +0xfc
        >  k8s.io/client-go/rest.(*Request).transformResponse(0xc0001c2480, 0xc0001f6bd0, 0xc0001f4800)
        >       /src/vendor/k8s.io/client-go/rest/request.go:1179 +0x5cf
        >  k8s.io/client-go/rest.(*Request).Do.func1(0xc0002d09c0?, 0x0?)
        >       /src/vendor/k8s.io/client-go/rest/request.go:1064 +0x45
        >  k8s.io/client-go/rest.(*Request).request.func3.1(...)
        >       /src/vendor/k8s.io/client-go/rest/request.go:1039
        >  k8s.io/client-go/rest.(*Request).request.func3(0xc0001f6bd0, 0xc00056b2e0, {0x1b219b8?, 0xc0002d09c0?}, 0x0?, 0x0?, 0xc00056b278?, {0x0?, 0x0?}, 0x19aa508)
        >       /src/vendor/k8s.io/client-go/rest/request.go:1046 +0xde
        >  k8s.io/client-go/rest.(*Request).request(0xc0001c2480, {0x1b21440, 0xc00032b0e0}, 0x7?)
        >       /src/vendor/k8s.io/client-go/rest/request.go:1048 +0x4e5
        >  k8s.io/client-go/rest.(*Request).Do(0xc0001c2480, {0x1b21440, 0xc00032b0e0})
        >       /src/vendor/k8s.io/client-go/rest/request.go:1063 +0xc9
        >  sigs.k8s.io/controller-runtime/pkg/client.(*typedClient).Create(0xc0004b2900, {0x1b21440, 0xc00032b0e0}, {0x1b31c80?, 0xc00033fa40}, {0x0, 0x0, 0xc0003f4500?})
        >       /src/vendor/sigs.k8s.io/controller-runtime/pkg/client/typed_client.go:48 +0x366
        >  sigs.k8s.io/controller-runtime/pkg/client.(*client).Create(0xc0004b2900?, {0x1b21440?, 0xc00032b0e0?}, {0x1b31c80?, 0xc00033fa40?}, {0x0?, 0x0?, 0x0?})
        >       /src/vendor/sigs.k8s.io/controller-runtime/pkg/client/client.go:288 +0xf0
        >  github.com/spidernet-io/spiderpool/cmd/spiderpool-init/cmd.(*CoreClient).WaitForCoordinatorCreated(0xc0003eee20, {0x1b21440, 0xc00032b0e0}, 0xc00033fa40)
        >       /src/cmd/spiderpool-init/cmd/client.go:52 +0xef
        >  github.com/spidernet-io/spiderpool/cmd/spiderpool-init/cmd.Execute()
        >       /src/cmd/spiderpool-init/cmd/root.go:55 +0x446
        >  main.main()
        >       /src/cmd/spiderpool-init/main.go:11 +0x17
{"level":"INFO","ts":"2024-02-07T00:28:16.357Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": EOF"}
{"level":"INFO","ts":"2024-02-07T00:28:18.360Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:21.366Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:24.374Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:27.382Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:30.391Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:33.398Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:36.406Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:39.414Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:42.422Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:45.430Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:48.439Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:51.446Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:54.458Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:28:57.495Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:00.503Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:03.510Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:06.518Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:09.526Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:12.534Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:15.542Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:18.550Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:21.558Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:24.566Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:27.574Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:30.582Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:33.590Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:36.598Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:39.606Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:42.614Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:45.622Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:48.630Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:51.638Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:54.646Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:29:57.654Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:00.662Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:03.670Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:06.678Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:09.686Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:12.694Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:15.702Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:18.710Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:21.718Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:24.727Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:27.734Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:30.742Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:33.750Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:36.758Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:39.770Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:42.806Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:45.814Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:48.822Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:51.830Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:54.838Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:30:57.846Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:00.854Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:03.862Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:06.870Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:09.878Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:12.886Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:15.894Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:18.902Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:21.910Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:24.918Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:27.926Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:30.934Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:33.946Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:36.950Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:39.958Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:42.966Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:45.974Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:48.982Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:51.990Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:54.998Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:31:58.006Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:32:01.014Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:32:04.022Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:32:07.030Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:32:10.038Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:32:13.046Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:32:16.054Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:32:19.066Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:32:22.070Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:32:25.078Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"INFO","ts":"2024-02-07T00:32:28.086Z","caller":"cmd/client.go:178","msg":"Failed to create multuscniconfig kube-system/mynet, recreate in 2s: Internal error occurred: failed calling webhook \"spidermultusconfig.spiderpool.spidernet.io\": failed to call webhook: Post \"https://spiderpool-controller.kube-system.svc:5722/mutate-spiderpool-spidernet-io-v2beta1-spidermultusconfig?timeout=10s\": dial tcp 10.99.41.184:5722: connect: connection refused"}
{"level":"FATAL","ts":"2024-02-07T00:32:31.073Z","logger":"Spiderpool-Init","caller":"cmd/root.go:152","msg":"context deadline exceeded"}

infinitydon · 2024-03-07T15:53:51Z

Hi using the latest release, the above error is not occurring but stateful pods are not coming up, following is the error from kubectl describe pod:

QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Normal   Scheduled               3m33s                default-scheduler  Successfully assigned default/test-sts-0 to k0s-spiderpool-test
  Warning  FailedCreatePodSandBox  3m33s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d0b1b5d83ae644ee6b7d3cf841be8d3e86428c539c17250316af880c8243b662": plugin type="multus" name="multus-cni-network" failed (add): [default/test-sts-0/e6bdbf46-31b6-41e8-b2ef-04242d5cb212:macvlan-ens19]: error adding container to network "macvlan-ens19": failed to GetCoordinatorConfig: [GET /coordinator/config][500] getCoordinatorConfigFailure  spidercoordinator: default no ready
  Normal   SandboxChanged          6s (x17 over 3m32s)  kubelet            Pod sandbox changed, it will be killed and re-created.

The spider controller and agent pods are not giving any errors in the logs

Below is the manifest I used:

MACVLAN_MASTER_INTERFACE="ens19"
MACVLAN_MULTUS_NAME="macvlan-$MACVLAN_MASTER_INTERFACE"

cat <<EOF | kubectl apply -f -
apiVersion: spiderpool.spidernet.io/v2beta1
kind: SpiderMultusConfig
metadata:
  name: ${MACVLAN_MULTUS_NAME}
  namespace: kube-system
spec:
  cniType: macvlan
  enableCoordinator: true
  macvlan:
    master:
    - ${MACVLAN_MASTER_INTERFACE}
EOF


cat <<EOF | kubectl apply -f -
apiVersion: spiderpool.spidernet.io/v2beta1
kind: SpiderIPPool
metadata:
  name: test-ippool
spec:
  subnet: 10.6.0.0/16
  ips:
    - 10.6.168.101-10.6.168.110
EOF


cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test-sts
spec:
  replicas: 2
  selector:
    matchLabels:
      app: test-sts
  template:
    metadata:
      annotations:
        ipam.spidernet.io/ippool: |-
            {      
              "ipv4": ["test-ippool"]
            }
        v1.multus-cni.io/default-network: kube-system/macvlan-ens19
      labels:
        app: test-sts
    spec:
      containers:
        - name: test-sts
          image: nginx
          imagePullPolicy: IfNotPresent
EOF

cyclinder · 2024-03-08T01:51:05Z

Hi @infinitydon, Thanks for the report. Could you show some debug information using the following commands?

kubectl get spidercoordinator default -o yaml
kubectl get cm -n kube-system kubeadm-config -o yaml
kubectl get po -A -l component=kube-controller-manager

And what is the version of spiderpool you used?

The spider controller and agent pods are not giving any errors in the logs

could you show the logs of spiderpool-controller？ Maybe the replicas of spiderpool-controller is greater than 1 in your cluster and you need to look at the logs of the other replicas.

infinitydon · 2024-03-08T02:03:30Z

@cyclinder - I am currently using k0s and not kubeadm cluster, so kubectl get po -A -l component=kube-controller-manager and
kubectl get cm -n kube-system kubeadm-config -o yaml will not work for my cluster.

Is spider-pool made to work only in a kubeadm based cluster?

kubectl get spidercoordinator default -o yaml

apiVersion: spiderpool.spidernet.io/v2beta1
kind: SpiderCoordinator
metadata:
  creationTimestamp: "2024-03-07T15:44:21Z"
  finalizers:
  - spiderpool.spidernet.io
  generation: 1
  name: default
  resourceVersion: "662"
  uid: f86489c3-42a8-4b2c-afcd-b6cdfc239d11
spec:
  detectGateway: false
  detectIPConflict: false
  hijackCIDR:
  - 169.254.0.0/16
  hostRPFilter: 0
  hostRuleTable: 500
  mode: auto
  podCIDRType: auto
  podDefaultRouteNIC: ""
  podMACPrefix: ""
  tunePodRoutes: true
  txQueueLen: 0
status:
  overlayPodCIDR: []
  phase: NotReady
  reason: No kube-controller-manager pod found, unable to get clusterCIDR
  serviceCIDR: []

Spiderpool-controller, this is just one replica:

kubectl -n kube-system get po
NAME                                    READY   STATUS      RESTARTS   AGE
kube-proxy-k5ptd                        1/1     Running     0          10h
kube-router-xx77q                       1/1     Running     0          10h
metrics-server-7556957bb7-dnjmr         1/1     Running     0          10h
coredns-6cd46fb86c-clh5d                1/1     Running     0          10h
spiderpool-controller-7c6f4fcdd-8ptlx   1/1     Running     0          10h
spiderpool-agent-zt6bp                  1/1     Running     0          10h
spiderpool-init                         0/1     Completed   0          10h

spiderpool-controller Logs attached below

spiderpool-logs-march-7-2024.txt

Spiderpool version is v0.9.2

kubectl -n kube-system get deploy,ds -o wide
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS              IMAGES                                                         SELECTOR
deployment.apps/spiderpool-controller   1/1     1            1           10h   spiderpool-controller   ghcr.io/spidernet-io/spiderpool/spiderpool-controller:v0.9.2   app.kubernetes.io/component=spiderpool-controller,app.kubernetes.io/instance=spiderpool,app.kubernetes.io/name=spiderpool

NAME                              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE   CONTAINERS         IMAGES                                                    SELECTOR
daemonset.apps/spiderpool-agent   1         1         1       1            1           kubernetes.io/os=linux   10h   spiderpool-agent   ghcr.io/spidernet-io/spiderpool/spiderpool-agent:v0.9.2   app.kubernetes.io/component=spiderpool-agent,app.kubernetes.io/instance=spiderpool,app.kubernetes.io/name=spiderpool

cyclinder · 2024-03-08T02:12:12Z

Is spider-pool made to work only in a kubeadm based cluster?

Currently, we synchronize the cluster CIDR from kubeadm-config configMap and KCM pod, I don't know much about k0s, can you tell me where I can get the cluster CIDR in a k0s based cluster?

And you can solve this by creating a configMap manually, please refer to https://spidernet-io.github.io/spiderpool/v0.9/usage/install/overlay/get-started-calico/#prerequisites.

infinitydon · 2024-03-08T14:51:38Z

Works now after creating the configmap.

For k0s, I am using the default settings, so the parameters can be retrieved from the k0s command k0s config create | grep -i cidr:

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  creationTimestamp: null
  name: k0s
spec:
  api:
    address: 192.168.100.61
    k0sApiPort: 9443
    port: 6443
    sans:
    - 192.168.100.61
    - 10.244.0.1
    - fe80::be24:11ff:fefb:7299
    - fe80::be24:11ff:fec3:394e
    - fe80::1005:ffff:fea5:39af
    - fe80::9416:34ff:fec4:6200
    - fe80::dccc:91ff:fe1e:e1d3
    - fe80::489b:6bff:fe1b:7417
    - fe80::3cb7:68ff:fe1e:72cb
    - fe80::3838:53ff:fe64:2c7a
  controllerManager: {}
  extensions:
    helm:
      charts: null
      concurrencyLevel: 5
      repositories: null
    storage:
      create_default_storage_class: false
      type: external_storage
  installConfig:
    users:
      etcdUser: etcd
      kineUser: kube-apiserver
      konnectivityUser: konnectivity-server
      kubeAPIserverUser: kube-apiserver
      kubeSchedulerUser: kube-scheduler
  konnectivity:
    adminPort: 8133
    agentPort: 8132
  network:
    calico: null
    clusterDomain: cluster.local
    dualStack: {}
    kubeProxy:
      iptables:
        minSyncPeriod: 0s
        syncPeriod: 0s
      ipvs:
        minSyncPeriod: 0s
        syncPeriod: 0s
        tcpFinTimeout: 0s
        tcpTimeout: 0s
        udpTimeout: 0s
      metricsBindAddress: 0.0.0.0:10249
      mode: iptables
    kuberouter:
      autoMTU: true
      hairpin: Enabled
      ipMasq: false
      metricsPort: 8080
      mtu: 0
      peerRouterASNs: ""
      peerRouterIPs: ""
    nodeLocalLoadBalancing:
      envoyProxy:
        apiServerBindPort: 7443
        konnectivityServerBindPort: 7132
      type: EnvoyProxy
    podCIDR: 10.244.0.0/16
    provider: kuberouter
    serviceCIDR: 10.96.0.0/12
  scheduler: {}
  storage:
    etcd:
      externalCluster: null
      peerAddress: 192.168.100.61
    type: etcd
  telemetry:
    enabled: true

Since spiderpool should be able to work on any k8s distro, I will suggest a different name for the configmap since not all the clusters will be based on kubeadm, maybe give it a generic name, what do you think?

cyclinder · 2024-03-11T02:16:00Z

Maybe we need a new ENV for spiderpool-controller, something like ClusterType. Its value can be kubeadm,k3s,k0s,systemd, etc. spiderpool-controller reads the value of this ENV at startup, and reads the cluster's CIDR from the relevant resources.

Can you read the ClusterConfig by using kubectl? like kubectl get clusterconfig -o yaml

infinitydon · 2024-03-12T18:21:28Z

kubectl get clusterconfig -o yaml does not work in the k0s:

ubuntu@k0s-spiderpool-test:~$ kubectl get clusterconfig -o yaml - clusterconfig
error: the server doesn't have a resource type "clusterconfig"

I agree with the suggestion of using ENV values so each user will be responsible for providing the values since there could be multiple ways of getting the values depending on the k8s distro.

cyclinder · 2024-03-15T05:49:39Z

Sadly, I went through on k0s quickly. Did not find a way to get cluster cidr by command or reading any resources, maybe we need to ask for help from the k0s community.

infinitydon added the kind/bug label Feb 7, 2024

infinitydon assigned cyclinder and Icarus9913 Feb 7, 2024

infinitydon changed the title ~~SpiderPool Controller Crash~~ SpiderPool Controller Crash On k8s 1.29 (k0s installer) Feb 7, 2024

ty-dc mentioned this issue Mar 25, 2024

Spider-agent Pod crashes on a cluster deployed with rke2 #3209

Closed

Icarus9913 removed their assignment Apr 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SpiderPool Controller Crash On k8s 1.29 (k0s installer) #3204

SpiderPool Controller Crash On k8s 1.29 (k0s installer) #3204

infinitydon commented Feb 7, 2024 •

edited

cyclinder commented Feb 7, 2024

infinitydon commented Feb 7, 2024

infinitydon commented Mar 7, 2024 •

edited

cyclinder commented Mar 8, 2024 •

edited

infinitydon commented Mar 8, 2024 •

edited

cyclinder commented Mar 8, 2024

infinitydon commented Mar 8, 2024

cyclinder commented Mar 11, 2024

infinitydon commented Mar 12, 2024

cyclinder commented Mar 15, 2024

SpiderPool Controller Crash On k8s 1.29 (k0s installer) #3204

SpiderPool Controller Crash On k8s 1.29 (k0s installer) #3204

Comments

infinitydon commented Feb 7, 2024 • edited

Spiderpool Version

Bug Type

Main CNI

What happened?

What did you expect to happen?

How to reproduce it (as minimally and precisely as possible)

Additional Context

cyclinder commented Feb 7, 2024

infinitydon commented Feb 7, 2024

infinitydon commented Mar 7, 2024 • edited

cyclinder commented Mar 8, 2024 • edited

infinitydon commented Mar 8, 2024 • edited

cyclinder commented Mar 8, 2024

infinitydon commented Mar 8, 2024

cyclinder commented Mar 11, 2024

infinitydon commented Mar 12, 2024

cyclinder commented Mar 15, 2024

infinitydon commented Feb 7, 2024 •

edited

infinitydon commented Mar 7, 2024 •

edited

cyclinder commented Mar 8, 2024 •

edited

infinitydon commented Mar 8, 2024 •

edited