Investigate test_basebackup_with_high_slru_count failure

[jcsp]

No description provided.

[koivunej]

Just now: https://neon-github-public-dev.s3.amazonaws.com/reports/main/8937443657/index.html#suites/c62b105f3a4f00dd6be4ad88810e0e02/7ddb54b21e1dcce3

[VladLazar]

I looked at the failure Joonas linked. It's a failure to start-up after a SIGQUIT stop for both the ps and the sk (we do that as part of the test setup).

Upon starting both the ps and sk seem to be stuck waiting for their PID file flocks (based on logs). The fd for the pidfile should have been dropped and, in any case, we shouldn't block on the lock since we use LOCK_NB. Must be missing something.

[VladLazar]

Also seen in https://neon-github-public-dev.s3.amazonaws.com/reports/main/8938399293/index.html#suites/c62b105f3a4f00dd6be4ad88810e0e02/a0635cfcff8697b9/ - same symptom

[jcsp]

This week:

• eyeball code again
• add more logs

[jcsp]

Seen in this benchmark run: https://neon-github-public-dev.s3.amazonaws.com/reports/main/9204710983/index.html

There are two possibly-related failures:

• test_basebackup_with_high_slru_count is timing out on pageserver startup, where the pageserver log prints the version and then nothing else: this may mean it is stuck on its flock call for a pidfile.
• test_download_churn is timing out on pageserver shutdown, where the pageserver prints that it has handled the signal, but neon_local does not see the process disappear.