Run most file locking through functions that handle Ceph EIO #4924
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…o issues/4874-lock-io-error
|
@DailyDreaming This should be ready for review. |
DailyDreaming
approved these changes
May 18, 2024
| else: | ||
| # Something else went wrong | ||
| os.close(fd) | ||
| raise |
There was a problem hiding this comment.
Should we close the file in either case? i.e.
except OSError as e:
os.close(fd)
if e.errno in (errno.EACCES, errno.EAGAIN):
# File is still locked by someone else.
# Look at the next file instead
continue
else:
# Something else went wrong
raise
There was a problem hiding this comment.
Yeah, we do need a close before continuing.
| else: | ||
| # Something went wrong | ||
| os.close(dirFD) | ||
| raise |
There was a problem hiding this comment.
Suggestion:
except OSError as e:
os.close(dirFD)
if not e.errno in (errno.EACCES, errno.EAGAIN):
# Not a locked file error. Something went wrong
raise
| pass | ||
| else: | ||
| raise | ||
| os.close(fd) |
There was a problem hiding this comment.
Suggestion:
try:
fcntl.lockf(fd, fcntl.LOCK_UN)
except OSError as e:
if e.errno != errno.EIO:
# Sometimes Ceph produces errno.EIO . We don't need to retry because
# we're going to close the FD and after that the file can't remain
# locked by us.
raise
os.close(fd)
There was a problem hiding this comment.
This doesn't leave the comment really in the right place relative to the code.
|
I had to change some This probably opens new ways to hang a Toil workflow if you have read access to its temp directories. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This will fix #4874.
But it also makes us sit forever retrying locks if fcntl gives an EIO. Since officially the syscall isn't meant to be giving EIO, maybe this is fine?
Changelog Entry
To be copied to the draft changelog by merger:
Reviewer Checklist
issues/XXXX-fix-the-thingin the Toil repo, or from an external repo.camelCasethat want to be insnake_case.docs/running/{cliOptions,cwl,wdl}.rstMerger Checklist