bug: increase of buffer pvc size results in reconcile error

[applike-ss]

Describe the issue

When increasing the pvc buffer size the operator logs errors during reconcilation, as the STS can not be adjusted.

To Reproduce

• Have a buffer PVC
• increase the size from 1Gi to 2Gi
• observe error

Expected behavior

storage increase should be detected and fulfilled

Your Environment

- Fluent Operator version: 2.4.0
- Container Runtime: containerd://1.6.23+bottlerocket
- Operating system: Bottlerocket OS 1.15.0 (aws-k8s-1.28)
- Kernel version: 6.1.49

How did you install fluent operator?

helm chart https://fluent.github.io/helm-charts

Additional context

No response

[wenchajun]

Please provide more detailed information, such as whether it is the PVC (Persistent Volume Claim) for Fluentd or Fluent Bit, and how the adjustments were made, along with any error logs.

[applike-ss]

I didn't assume there would be a different handling when it comes to PVCs for Fluentd or Fluent Bit.

We are using Fluentd in this case and have updated our Fluentd CRs .spec.buffer.pvc.spec.resources.requests.storage property.

These are the errors we would get:

ERROR	Reconciler error	{"controller": "fluentd", "controllerGroup": "fluentd.fluent.io", "controllerKind": "Fluentd", "Fluentd": {"name":"fluentd","namespace":"fluentd"}, "namespace": "fluentd", "name": "fluentd", "reconcileID": "53e93bcd-e9c9-4fd7-b9ad-b02e0785bafd", "error": "StatefulSet.apps \"fluentd\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden"}

So basically statefulset update error due to the template changing, which is not allowed.

[icy]

A good reference implementation by ECK (ElasticSearch Operator): elastic/cloud-on-k8s#3752. They delete the stateful set (but keep the pods).