MinPinnedVersionId not increase when there's no batch queries running

[xxchan]

background:
https://risingwave-labs.slack.com/archives/C034TRN6A49/p1714528348327289

During oncall, we found MinPinnedVersionIdNotIncrease alert keeps firing for a cluster. The version id increase only every 3 hours.

Grafana indicates that there were no batch queries running.
So it's weird that the version id has been pinned for such an extended period.

Not sure if this is a bug. If it's not, maybe we should remove this alert.

[xxchan]

@zwang28 @Little-Wallace randomly assigned to you. Feel free to find someone else to check this issue

[hzxa21]

Take a glance at grafana:

• Barrier latency and barrier number increases at 05/01 01:30 UTC+8:
  

• At around the same time, the delay in MinPinnedVersionId happened:
  

I suspect the high barrier latency is the trigger of the issue.

[hzxa21]

Recovery triggered at the same time:

I suspect that one CN stops receiving version update after recovery.

[zwang28]

The version id increase only every 3 hours.

unpin every 3 hours is force by max_version_pinning_duration_sec = 10800

[fuyufjh]

Note down some observations of today's case: Name: MinPinnedVersionIdNotIncrease Sev: [warning] Cluster: [prod-aws-euno1-eks-a] at 10:04 AM.

Grafana URL

Min-pinned version ID was stuck:

Meanwhile, pinned epoch IDs were normal:

Barrier was abnormal, but it's after the 1st min-epoch stuck. Thus, it's likely to be a result instead of the cause.

There was no heavy batch queries.

Now I am wondering what happens at 7:58 AM i.e. the 1st stuck

The stuck at 7:58 AM, which I consider as the root cause, was caused by a sink error:

Grafana Logs URL

s3 error: service error: NoSuchKey: The specified key does not exist
and
prefetch meet error when read 13368429..13451152 from sst-39302557 (63181742)

[zwang28]

The hummock version is pinned by log store. It's consistently held due to the large volume of historical logs to consume, until forcefully unpinned by max_version_pinning_duration_sec.

The log input rate of sink exceeds the consumption rate, so the situation will deteriorate.

https://risingwave-labs.slack.com/archives/C034TRN6A49/p1716526705213879?thread_ts=1716516350.612169&cid=C034TRN6A49