Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore ebusy when deleting a directory #11066

Merged
merged 3 commits into from
Apr 29, 2024

Conversation

lukebakken
Copy link
Collaborator

Reported in #11047

This error manifests when the quorum directory is its own mount point.

@lukebakken lukebakken self-assigned this Apr 23, 2024
@lukebakken lukebakken marked this pull request as draft April 23, 2024 16:05
Copy link
Member

@michaelklishin michaelklishin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lukebakken unlike with enoent, should ebusy lead to a (limited) number retries with sleeps in between?

Which would require a slightly larger change.

@michaelklishin
Copy link
Member

According to glibc docs, ebusy means that a file cannot be unlinked (deleted) immediately because it is used in some shape or form according to the kernel.

So looks like we must retry up to N times with a delay in between (of, say, 5ms?)

@lukebakken
Copy link
Collaborator Author

A retry won't fix the issue with /quorum being its own mountpoint, since attempting to delete it will always result in ebusy

@lukebakken lukebakken force-pushed the rabbitmq-server-11047-ignore-ebusy branch from 6ddd380 to f45a262 Compare April 24, 2024 14:53
@lukebakken lukebakken marked this pull request as ready for review April 24, 2024 14:54
@lukebakken lukebakken force-pushed the rabbitmq-server-11047-ignore-ebusy branch from f45a262 to ab909c3 Compare April 25, 2024 15:43
lukebakken and others added 3 commits April 29, 2024 11:18
Reported in #11047

This error manifests when the quorum directory is its own mount point
A folder can be a mount point and the kernel won't allow
deleting a mount point - it will return ebusy

We don't need to check for ebusy when deleting a file
msg_stores is a potential mount point and therefore might exist
@michaelklishin michaelklishin force-pushed the rabbitmq-server-11047-ignore-ebusy branch from ab909c3 to c9fcc4d Compare April 29, 2024 15:18
@michaelklishin
Copy link
Member

The force-push was a rebase.

@michaelklishin
Copy link
Member

An additional investigation suggests that when there are two overlapping directories on two different Linux mounts (at least with EBS volumes), for instance

  • /data/rabbitmq/node_data on volume A
  • /data/rabbitmq/node_data/quorum on volume B

then the node won't be able to delete /data/rabbitmq/node_data/quorum (a kernel lock will be held by volume B) but it will be able to delete its contents. So this can be a worthy improvement in certain environments.

@michaelklishin michaelklishin merged commit e158a86 into main Apr 29, 2024
16 of 18 checks passed
@michaelklishin michaelklishin deleted the rabbitmq-server-11047-ignore-ebusy branch April 29, 2024 17:15
@michaelklishin
Copy link
Member

@Mergifyio backport v3.13.x

Copy link

mergify bot commented Apr 29, 2024

backport v3.13.x

✅ Backports have been created

michaelklishin added a commit that referenced this pull request Apr 29, 2024
Ignore `ebusy` when deleting a directory (backport #11066)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants