High memory usage with watch-kube-secrets enable

[twingate-blee]

We're running certificate-exporter on 4 environments. The memory usage on all other envs are below 50m. For some reason in our dev environment the memory spikes to about 500m when it reads the secrets. I have increased the default max-cache-duration to 20 mins and see the spike now every 20 mins.

In each env we are reading in the same number of certificates

The container image is:
docker.io/enix/x509-certificate-exporter:3.3.0

Args are:

    Args:
      --watch-kube-secrets
      --secret-type=kubernetes.io/tls:tls.crt
      --max-cache-duration=300s
      --listen-address=:9793

Log with cert paths redacted

time="2023-01-05T19:57:30Z" level=info msg="fetching configuration from within the cluster"
time="2023-01-05T19:57:30Z" level=info msg="loaded configuration, API server is at https://10.27.128.1:443"
time="2023-01-05T19:57:30Z" level=info msg="fetching API server version"
time="2023-01-05T19:57:30Z" level=info msg="kubernetes server version is v1.22.15-gke.100"
time="2023-01-05T19:57:30Z" level=info msg="starting x509-certificate-exporter version 3.3.0"
time="2023-01-05T19:57:51Z" level=info msg="3 valid certificate(s) found in \"k8s/*****\""
time="2023-01-05T19:57:51Z" level=info msg="3 valid certificate(s) found in \"k8s/*****\""
time="2023-01-05T19:57:51Z" level=info msg="3 valid certificate(s) found in \"k8s/*****\""
time="2023-01-05T19:57:51Z" level=info msg="3 valid certificate(s) found in \"k8s/*****\""
time="2023-01-05T19:57:51Z" level=info msg="3 valid certificate(s) found in \"k8s/*****\""
time="2023-01-05T19:57:51Z" level=info msg="parsed 15 certificates (0 read failures)"
time="2023-01-05T19:57:51Z" level=info msg="listening on :9793"

High mem in dev

What it typical memory usage looks like in other environments.

Do you know what would be causing the higher memory usage?

[pcecot]

I am seeing the same problem with only 4 certificates.