[install error] koordlet panic

[googs1025]

I followed the installation instructions in docs here, but I encountered a panic issue. Could it be related to the read-only port of kubelet?

root@VM-0-15-ubuntu:/home/ubuntu# helm repo add koordinator-sh https://koordinator-sh.github.io/charts/
"koordinator-sh" has been added to your repositories
root@VM-0-15-ubuntu:/home/ubuntu# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "koordinator-sh" chart repository
Update Complete. ⎈Happy Helming!⎈
root@VM-0-15-ubuntu:/home/ubuntu# helm install koordinator koordinator-sh/koordinator --version 1.4.1
NAME: koordinator
LAST DEPLOYED: Wed May  1 12:56:26 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
root@VM-0-15-ubuntu:/home/ubuntu# kubectl get pods -nkoordinator-system
NAME                                 READY   STATUS             RESTARTS       AGE
koord-descheduler-5d9854998c-5nldm   1/1     Running            0              5m23s
koord-descheduler-5d9854998c-g9vsp   1/1     Running            0              5m23s
koord-manager-59c5b5698f-cvqt5       1/1     Running            0              5m23s
koord-manager-59c5b5698f-mdvhc       1/1     Running            0              5m23s
koord-scheduler-75699f7c69-hwx4t     1/1     Running            0              5m23s
koord-scheduler-75699f7c69-nvdf4     1/1     Running            0              5m23s
koordlet-mdng5                       0/1     CrashLoopBackOff   5 (97s ago)    5m23s
koordlet-nth2s                       0/1     CrashLoopBackOff   5 (89s ago)    5m23s
koordlet-x75rb                       0/1     CrashLoopBackOff   5 (110s ago)   5m23s

root@VM-0-15-ubuntu:/home/ubuntu# kubectl logs koordlet-x75rb -nkoordinator-system
E0501 04:59:59.963543  495704 pleg.go:84] failed to create pod watcherinotify_init: too many open files
F0501 04:59:59.963600  495704 states_pods.go:65] failed to create PLEG, inotify_init: too many open files
goroutine 1 [running, locked to thread]:
k8s.io/klog/v2.stacks(0x1)
	/go/pkg/mod/k8s.io/klog/v2@v2.60.1/klog.go:860 +0x8a
k8s.io/klog/v2.(*loggingT).output(0x3c123a0, 0x3, 0x0, 0xc00085ea10, 0x1, {0x2fcb44a?, 0x1?}, 0x3c137a0?, 0x0)
	/go/pkg/mod/k8s.io/klog/v2@v2.60.1/klog.go:825 +0x686
k8s.io/klog/v2.(*loggingT).printfDepth(0x3c123a0, 0x6b3a48?, 0x0, {0x0, 0x0}, 0x40?, {0x2487303, 0x19}, {0xc000858af0, 0x1, ...})
	/go/pkg/mod/k8s.io/klog/v2@v2.60.1/klog.go:630 +0x1f2
k8s.io/klog/v2.(*loggingT).printf(...)
	/go/pkg/mod/k8s.io/klog/v2@v2.60.1/klog.go:612
k8s.io/klog/v2.Fatalf(...)
	/go/pkg/mod/k8s.io/klog/v2@v2.60.1/klog.go:1516
github.com/koordinator-sh/koordinator/pkg/koordlet/statesinformer/impl.NewPodsInformer()
	/go/src/github.com/koordinator-sh/koordinator/pkg/koordlet/statesinformer/impl/states_pods.go:65 +0xe7
github.com/koordinator-sh/koordinator/pkg/koordlet/statesinformer/impl.init()
	/go/src/github.com/koordinator-sh/koordinator/pkg/koordlet/statesinformer/impl/registry.go:26 +0x1bb

goroutine 10 [sleep]:
time.Sleep(0x6fc23ac00)
	/usr/local/go/src/runtime/time.go:194 +0x12e
sigs.k8s.io/controller-runtime/pkg/log.init.0.func1()
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/log/log.go:63 +0x38
created by sigs.k8s.io/controller-runtime/pkg/log.init.0
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/log/log.go:62 +0x25

goroutine 22 [select]:
github.com/koordinator-sh/koordinator/pkg/util/metrics.(*metricGC).run(0xc0003aa6c0)
	/go/src/github.com/koordinator-sh/koordinator/pkg/util/metrics/expire_metric.go:232 +0xfc
created by github.com/koordinator-sh/koordinator/pkg/util/metrics.(*metricGC).Run
	/go/src/github.com/koordinator-sh/koordinator/pkg/util/metrics/expire_metric.go:225 +0x56

goroutine 70 [select]:
github.com/patrickmn/go-cache.(*janitor).Run(0xc000530cb0, 0x0?)
	/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1079 +0x85
created by github.com/patrickmn/go-cache.runJanitor
	/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1099 +0xed

goroutine 71 [select]:
github.com/patrickmn/go-cache.(*janitor).Run(0xc000530cc0, 0x0?)
	/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1079 +0x85
created by github.com/patrickmn/go-cache.runJanitor
	/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1099 +0xed

Environment:
App version: 1.4.1
Kubernetes version (use kubectl version): 1.28
OS version: Ubuntu 22.04
Kernal version:
Cgroup driver: cgroupfs/systemd

[googs1025]

/kind bug

[saintube]

@googs1025 The panic message indicates that the koordlet fails to initialize the inotify for host cgroups. It may be due to exceeding the host inotify limit, for which you can check fs.inotify.max_user_instances and fs.inotify.max_user_instances in the host /etc/sysctl.conf.