[card-cache] optimizations to avoid deadlocks #420
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
valayDave
force-pushed
the
cards/cache-optimizations-2
branch
2 times, most recently
from
5e44d02
to
7776685
Compare
- deadlocks happened when cleanups coincided with heavy load
- tests simulating a cleanup (disk + shared-objects) and heavy load together were successfully able to reproduce the deadlock situation
- Locking at the `context` level:
- Each time we want to clean up we lock to create a new context
- All directories/processes are written within that new context.
- Switching this context which ensure that all new processes get created differently and the cleanup process can safely remove everything.
- Context also sets the read/write directory for the cache object used in the API endpoint.
- All locking on the API side now always time-bound. The code time's out if it can't acquire a lock.
- The method ensure that all operations won't in-definately hold the lock.
- Changed defaults for minimum amount to time to wait for cards in the cache process to 20 seconds (helps make things snappier)
- Added `timings` dict in card cache to optimize loading cycles (Ensured that it is set based on a per-card basis)
valayDave
force-pushed
the
cards/cache-optimizations-2
branch
from
7776685
to
ac44069
Compare
valayDave
commented
Apr 17, 2024
| if cards is None: | ||
| return web.Response( | ||
| content_type="text/html", | ||
| status=404, | ||
| body="Card not found for task. Possibly still being processed. Please refresh page to check again.", | ||
| body=html_reload_script, |
There was a problem hiding this comment.
Change to UI coming soon!.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[card-cache] optimizations to avoid deadlocks
contextlevel:timingsdict in card cache to optimize loading cycles (Ensured that it is set based on a per-card basis)