Longhorn api-server PUT request rate

[krohrsb]

I guess my question comes down to checking the chattiness of longhorn PUT requests to the apiserver.

Specifically PUT "/apis/longhorn.io/v1beta2/namespaces/longhorn-system/replicas/<some-pvc-id>"

I have 52 volumes, each with 2 replicas, across 5 nodes. So a total of 104 replicas.

Prometheus denotes roughly 34 PUT requests per second by the longhorn group.

Compared to the next highest caller (2.5r/s), longhorn is about 1200% (1.2k%) higher :)

So I am just wanting to confirm if this is expected, and if anything can be done to mitigate it without repercussions.

I realize its probably doing so for a reason, it just seemed high. But perhaps I just don't know better :)

[derekbit]

@krohrsb What's your Longhorn version?

[krohrsb]

1.5.4 - I've sent a bundle via longhorn-support-bundle@rancher.com

[derekbit]

@krohrsb Please send it to longhorn-support-bundle@suse.com. Thanks.

[krohrsb]

Ah ok, done.

[krohrsb]

I run k3s and so things like etcd/api-server are bundled in, unsure if that impedes the bundle investigation.

Here are some metrics of api-server calls by group, resource (30s resolution) across 3 hours if it helps visualize things.

[krohrsb]

I suppose this correlates to logs found generated by longhorn-manager-* pods. tailing the longhorn-system namespace shows many of these types of logs associated with UPDATE operations.

longhorn-system/longhorn-manager-cnrpz[longhorn-manager]: time="2024-03-07T18:48:31Z" level=info msg="Request (user: system:serviceaccount:longhorn-system:longhorn-service-account, longhorn.io/v1beta2, Kind=Replica, namespace: longhorn-system, name: pvc-604f78fe-9dc1-49dc-83f1-af176a91609a-r-fb064246, operation: UPDATE) patchOps: [{\"op\": \"replace\", \"path\": \"/metadata/labels\", \"value\": {\"longhorn.io/backing-image\":\"\",\"longhorndiskuuid\":\"c136add7-dc6c-477b-ab73-dbe139680c91\",\"longhornnode\":\"node-2\",\"longhornvolume\":\"pvc-604f78fe-9dc1-49dc-83f1-af176a91609a\"}}]" func="admission.(*Handler).admit" file="admission.go:115" service=admissionWebhook

Though perhaps this is visible in the support bundle, as well.

[derekbit]

@krohrsb Let's discuss the issue #8114. Thank you.