Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement transactional large blob cache writes #9726

Merged
merged 10 commits into from
May 24, 2024

Conversation

yamadapc
Copy link
Contributor

@yamadapc yamadapc commented May 20, 2024

This PR adds unit-tests for FS/LMDB caches and implements transactional writes. This prevents a large blob write from failing partially or from broken entries from being read.

The implementation of this is in LMDBCache.js. Large blobs are stored through an entity LargeBlobEntry, which contains a randomly generated key for the FSCache large blob entries used. Since FSCache large blob keys are generated through this mechanism, it's not possible for a partially written entry to be read. Furthermore, it's not possible for the entry to become corrupted.

In addition to this, changes were made to the RequestTracker cache writing.

Previously, parcel-query worked by listing directory files under the cache directory and finding the latest large blob keys on disk. This was very coupled with the cache implementation in use. To improve this and fix breakage caused by my changes, a new RequestCacheInfo entry is written with the latest request graph cache key.

parcel-query is changed to read this entry, then scan the request graph nodes to find the asset and bundle graph cache entries.

Some parcel-query tests were patched as it was found that they only previously worked due to stale .parcel-cache directories being present (e.g.: on v2 on a clean workspace, running only the parcel-query integration tests will fail; they only pass in CI because other tests populate a directory).

@yamadapc
Copy link
Contributor Author

This breaks parcel-query because it directly reads the cache filesystem and assumes it's in a certain state.

@yamadapc yamadapc added this pull request to the merge queue May 24, 2024
@yamadapc yamadapc removed this pull request from the merge queue due to a manual request May 24, 2024
@yamadapc yamadapc added this pull request to the merge queue May 24, 2024
Merged via the queue into v2 with commit d988abe May 24, 2024
16 of 17 checks passed
@mischnic mischnic deleted the issue/improve-fs-cache-reliability branch May 28, 2024 11:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants