Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Race condition in remote filesystem sync #539

Open
ml-evs opened this issue Jan 18, 2024 · 0 comments · May be fixed by #562
Open

Race condition in remote filesystem sync #539

ml-evs opened this issue Jan 18, 2024 · 0 comments · May be fixed by #562
Assignees
@ml-evs
Labels
bug Something isn't working

Comments

@ml-evs
Copy link
Member

ml-evs commented Jan 18, 2024 •
edited

Currently multiple workers will happily trigger a remote fs sync; we need to add a lock in the database for ongoing syncs, and then an automatic retry from the processes that did not acquire the lock.

The only bad side effect here is the unnecessary extra syncs, as opposed to any data corruption.
@ml-evs ml-evs added the bug Something isn't working label Jan 18, 2024
@ml-evs ml-evs self-assigned this Jan 18, 2024
@ml-evs ml-evs linked a pull request Feb 4, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ml-evs ml-evs

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add process lock for remote filesystem scraper the-grey-group/datalab
1 participant
@ml-evs