Huge memory usage increase in v0.19.1+

[dmitriishaburov]

Which component:
Controller v0.19.1+
(didn't checked v0.19.0 since it doesn't have published chart

Describe the bug
After update from any previous version memory usage from ~7mb to ~50+ mb (slightly drops after some time)

To Reproduce
Steps to reproduce the behavior:

1. Update from v0.18.5 to v0.19.1
2. Check graph on memory usage of container

Expected behavior
Memory usage not increase so drastically

Version of Kubernetes:

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:40:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25+", GitVersion:"v1.25.12-eks-2d98532", GitCommit:"0aa16cf4fac4da27b9e9e9ba570b990867f6a3d8", GitTreeState:"clean", BuildDate:"2023-07-28T16:52:04Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}

Additional context
Basically we're running controllers with 20MB limits/requests in the kubernetes.
It was OK for a long time, but new version went to OOMKilled and only runs now with at least 60MB or more

While that's not a big numbers at all, it's still strange that there's so big memory usage spike without getting some cool additional features.

Feel free to close the issue though since it's mostly just a rant 😄

[alvneiayu]

hi @dmitriishaburov

Thanks to let us know that. We are going to investigate it because it is pretty curious. I think that during this release version, we included the Secret informer to regenerate the secrets managing by Sealed Secrets when they are removed manually by the users.

Let me investigate it, again thanks a lot for letting us know.

Álvaro

[dmitriishaburov]

Tested, up to ~10x higher memory usage is still there, but now possible to migitage by skipRecreate: true

On graph: old version / two pods on latest version / latest version with skipRecreate

Probably issue can be closed if there's no plans for reviewing mem usage.