[DB-26-3] Open scavenged chunks quickly. Build midpoints later on demand

[timothycoleman]

Changed: Open the database faster by building the midpoints for scavenged chunks later on demand

Otherwise opening a database with thousands scavenged chunks can become too slow

[linear]

DB-733 Open scavenged chunks more quickly

[sakno]

I have no questions about functional aspects of the proposed changes. From non-functional point of view, there are two improvements possible:

• If lock contention is rare (this is mentioned in the comments inside of the code), the same can be done in lock-free fashion
• RequestCaching might start calculation of midpoints in the background task for each chunk (minimal changes are required) and synchronize at its result only on demand.

[shaan1337]

nice catch 👍 changes look good to me, i've just added a couple of minor comments (and
i also need to do a quick test)

[shaan1337]

On my machine, basic performance tests show at least a 10x improvement in the opening speed of scavenged chunks at startup.
Previously, it was taking roughly 40ms, now it takes around 1-4ms !

[github-actions]

🚨 @timothycoleman Failed to create cherry Pick PR due to error:

RequestError [HttpError]: Merge conflict
   at /home/runner/work/_actions/EventStore/Automations/master/cherry-pick-pr-for-label/node_modules/@octokit/request/dist-node/index.js:66:23
   at processTicksAndRejections (node:internal/process/task_queues:96:5) {
 status: 409,
 headers: {
   'access-control-allow-origin': '*',
   'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset',
   connection: 'close',
   'content-length': '112',
   'content-security-policy': "default-src 'none'",
   'content-type': 'application/json; charset=utf-8',
   date: 'Mon, 20 May 2024 14:29:36 GMT',
   'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
   server: 'GitHub.com',
   'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
   vary: 'Accept-Encoding, Accept, X-Requested-With',
   'x-accepted-github-permissions': 'contents=write',
   'x-content-type-options': 'nosniff',
   'x-frame-options': 'deny',
   'x-github-api-version-selected': '2022-11-28',
   'x-github-media-type': 'github.v3; format=json',
   'x-github-request-id': '3389:253A59:3A967:7399E:664B5E4F',
   'x-ratelimit-limit': '5000',
   'x-ratelimit-remaining': '4990',
   'x-ratelimit-reset': '1716218972',
   'x-ratelimit-resource': 'core',
   'x-ratelimit-used': '10',
   'x-xss-protection': '0'
 },
 request: {
   method: 'POST',
   url: 'https://api.github.com/repos/EventStore/EventStore/merges',
   headers: {
     accept: 'application/vnd.github.v3+json',
     'user-agent': 'octokit-core.js/3.3.2 Node.js/16.20.2 (linux; x64)',
     authorization: 'token [REDACTED]',
     'content-type': 'application/json; charset=utf-8'
   },
   body: '{"base":"cherry-pick-cherry-pick/4214/timothycoleman/open-scavenged-chunks-faster-release/oss-v22.10-df43d468-7c94-4015-bdc9-4957b622ae4b","commit_message":"Merge 05508d14a0fbf8dc76a22b6a3bfec1691e6e7baf into cherry-pick-cherry-pick/4214/timothycoleman/open-scavenged-chunks-faster-release/oss-v22.10-df43d468-7c94-4015-bdc9-4957b622ae4b [skip ci]\\n\\n\\nskip-checks: true\\n","head":"05508d14a0fbf8dc76a22b6a3bfec1691e6e7baf"}',
   request: { agent: [Agent], hook: [Function: bound bound register] }
 },
 documentation_url: 'https://docs.github.com/rest/branches/branches#merge-a-branch'
}

🚨👉 Check https://github.com/EventStore/EventStore/actions/runs/9160288944

[github-actions]

@timothycoleman 👉 Created pull request targeting release/oss-v23.10: #4260

[github-actions]

@timothycoleman 👉 Created pull request targeting release/oss-v24.2: #4261

[github-actions]

@timothycoleman 👉 Created pull request targeting release/oss-v24.4: #4262