metrics and objects deployments generating tons of zombie processes and using up cluster node process limits

[gvoden]

What happened:
Deploying metrics, metrics aggregator and kube-objects (all images tagged with 1.2.1) seems to cause lots of zombie processes to be created on the cluster node where the deployment is and eventually cluster node is overwhelmed and crashes (Amazon EKS 1.22)

What you expected to happen:
Metrics and object collections should function normally.

How to reproduce it (as minimally and precisely as possible):
Deploy Splunk Connect with below YAML

global:
logLevel: info
splunk:
hec:
host: http-inputs-hoopp.splunkcloud.com
insecureSSL: false
port: 443
protocol: https
token:
splunk-kubernetes-logging:
enabled: true
journalLogPath: /var/log/journal
logs:
isg-containers:
logFormatType: cri
from:
container: isg-
pod: '*'
multiline:
firstline: /^\d{4}-\d{2}-\d{2} \d{1,2}:\d{1,2}:\d{1,2}.\d{3}/
sourcetype: kube:container
timestampExtraction:
format: '%Y-%m-%d %H:%M:%S.%NZ'
regexp: time="(?\d{4}-\d{2}-\d{2}T[0-2]\d:[0-5]\d:[0-5]\d.\d{9}Z)"
image:
registry: docker.io
name: splunk/fluentd-hec
tag: 1.3.1
pullPolicy: Always
resources:
limits:
memory: 1.5Gi
splunk:
hec:
indexName: eks_logs
splunk-kubernetes-metrics:
image:
registry: docker.io
name: splunk/k8s-metrics
tag: 1.2.1
pullPolicy: Always
imageAgg:
registry: docker.io
name: splunk/k8s-metrics-aggr
tag: 1.2.1
pullPolicy: Always
rbac:
create: true
serviceAccount:
create: true
name: splunk-kubernetes-metrics
splunk:
hec:
indexName: eks_metrics
splunk-kubernetes-objects:
image:
registry: docker.io
name: splunk/kube-objects
tag: 1.2.1
pullPolicy: Always
kubernetes:
insecureSSL: true
objects:
apps:
v1:
- interval: 30s
name: deployments
- interval: 30s
name: daemon_sets
- interval: 30s
name: replica_sets
- interval: 30s
name: stateful_sets
core:
v1:
- interval: 30s
name: pods
- interval: 30s
name: namespaces
- interval: 30s
name: nodes
- interval: 30s
name: services
- interval: 30s
name: config_maps
- interval: 30s
name: secrets
- interval: 30s
name: persistent_volumes
- interval: 30s
name: service_accounts
- interval: 30s
name: persistent_volume_claims
- interval: 30s
name: resource_quotas
- interval: 30s
name: component_statuses
- mode: watch
name: events
rbac:
create: true
serviceAccount:
create: true
name: splunk-kubernetes-objects
splunk:
hec:
indexName: eks_meta
Anything else we need to know?:

Scaling down the deployment for metrics and objects to 0 makes the zombie processes disappear immediately
Environment:

• Kubernetes version (use kubectl version): EKS 1.22

• Ruby version (use ruby --version):

• OS (e.g: cat /etc/os-release):

• NAME="Amazon Linux"
  VERSION="2"
  ID="amzn"
  ID_LIKE="centos rhel fedora"
  VERSION_ID="2"
  PRETTY_NAME="Amazon Linux 2"
  ANSI_COLOR="0;33"
  CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
  HOME_URL="https://amazonlinux.com/"

• Splunk version: - check YAML above

• Splunk Connect for Kubernetes helm chart version: 1.4.3

• Others:

[gvoden]

We found the following in our logs:
2023-04-27 18:06:26 +0000 [error]: #0 unexpected error error_class=Kubeclient::HttpError error="HTTP status code 403, v1 is forbidden: User "system:serviceaccount:splunk-connect-k8s:splunk-kubernetes-objects" cannot list resource "v1" in API group "" at the cluster scope for GET https://10.100.0.1/api/apps/v1"

The service account was missing permissions to list the v1 resource. After updating permissions in our clusterrole we no longer see this error and the zombie process creation is cleared and issue is resolved.
Question, why does the pod need access to this v1 endpoint? And did it require access to it in prior versions?