Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Permission steps during updates against an existing instance cause major playbook execution slowdowns & high CPU usage on NFS backed storage #69

Open
thunderysteak opened this issue Oct 13, 2023 · 0 comments

Comments

@thunderysteak
Copy link
Contributor

thunderysteak commented Oct 13, 2023

When using the playbook against an existing instance (as part of updates) that is either a part of a larger relay, or has a lot of media files, the file permission tasks crawls media folders as well:
https://github.com/mastodon/mastodon-ansible/blob/main/bare/roles/web/tasks/mastodon-postflight.yml#L102

As Ansible crawls every single media file, it causes the execution to take way too long to perform as it edits every single media file permissions. In testing, it took over two hours and it appeared to be stuck.

This gets worse on NFS backed storage, as Ansible rapidly hammers it and causes the CPU to majorly spike up. Obviously this isn't really ideal.

firefox_2023-10-13_23-02-12

Mastodon with NFS backed storage Playbook for error replication:
https://github.com/thunderysteak/steaks-ansible-playbooks/tree/main/nfs-mastodon-media

Suggestions on how to solve this:

  • Omit /home/mastodon/live/public/system from having chown tasks performed against it
  • Same as above but put it behind a feature flag for backwards compatibility with the default behavior being DISABLED.

Tasks that cause issues:

  • Ensure that we have correct file permissions with owner being the user and NGINX being the group
  • Ensure that we have correct file permissions with owner being the user and www-data being the group
  • Ensure that we have correct selinux context permissions for RedHat systems
  • Restore selinux condition on RedHat systems
  • Create folder structure for Mastodon public folder
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant