Add info for systemd based distros #583

stratakis · 2020-05-15T16:34:19Z

Provide examples for managing the buildbot-worker
service through systemd unit files for systemd based
distributions.

Mariatta · 2020-05-15T16:40:17Z

Note I have just removed the Netlify integration, since we're using readthedocs preview build now.

buildworker.rst

tiran · 2020-05-15T16:41:59Z

buildworker.rst

+        Wants=network.target
+        After=network.target


Wants=network-online.target After=network-online.target local-fs.target

Will amend that

tiran · 2020-05-15T16:42:29Z

buildworker.rst

+      where the enforcing mode is enabled, usually related to the twistd.pid file.
+      If the service fails to start, you should check the output of
+      ``systemctl status buildbot-worker.service`` as well as the
+      ``/var/log/audit/audit.log`` file for potential issues and remedies.


ausearch -m AVC

I usually use sealert -a /var/log/audit/audit.log which provides messages with the warning and how to fix them, but I thought it could be too specific for the devguide.

Is this really going to be an issue? Unless you define a policy the buildbot daemon should run in unconfined_t and not be affected by SELinux policy restrictions.

It was always affected on all the buildbots I've set up (Fedora and RHEL). Basically selinux prevents systemd's open and read access for the twistd.pid file.

So after following the instructions on the buildbot worker page to set it up and creating the systemd unit files, when trying to start the service, initially this error message is show (date/time is trimmed out):

`
Following twistd.log until startup finished..
AVC avc: denied { read } for pid=1 comm="systemd" name="twistd.pid" dev="dm-0" ino=25387713 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

AVC avc: denied { read } for pid=1 comm="systemd" name="twistd.pid" dev="dm-0" ino=25387713 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

AVC avc: denied { read } for pid=1 comm="systemd" name="twistd.pid" dev="dm-0" ino=25387713 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

localhost systemd[1]: buildbot-worker.service: Can't convert PID files /home/buildbot/buildarea/twistd.pid O_PATH file descriptor to proper file descriptor: Permission denied

localhost systemd[1]: buildbot-worker.service: Can't convert PID files /home/buildbot/buildarea/twistd.pid O_PATH file descriptor to proper file descriptor: Permission denied

localhost setroubleshoot[1489084]: SELinux is preventing systemd from read access on the file twistd.pid. For complete SELinux messages run: sealert -l 57d3c874-be75-46f6-82c9-3680ac54002c

localhost python3[1489084]: SELinux is preventing systemd from read access on the file twistd.pid.#12#012***** Plugin catchall (100. confidence) suggests **************************#12#012
If you believe that systemd should be allowed read access on the twistd.pid file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#12# ausearch -c 'systemd' --raw | audit2allow -M my-systemd#012# semodule -X 300 -i my-systemd.pp#012

localhost setroubleshoot[1489084]: SELinux is preventing systemd from read access on the file twistd.pid. For complete SELinux messages run: sealert -l 57d3c874-be75-46f6-82c9-3680ac54002c

localhost python3[1489084]: SELinux is preventing systemd from read access on the file twistd.pid.#12#012***** Plugin catchall (100. confidence) suggests **************************#12#012If you believe that systemd should be allowed read access on the twistd.pid file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#12# ausearch -c 'systemd' --raw | audit2allow -M my-systemd#012# semodule -X 300 -i my-systemd.pp#012
`
After adjusting the policy, another error comes up:

`
Following twistd.log until startup finished..

localhost audit[1]: AVC avc: denied { open } for pid=1 comm="systemd" path="/home/buildbot/buildarea/twistd.pid" dev="dm-0" ino=25387713 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

localhost systemd[1]: buildbot-worker.service: Can't convert PID files /home/buildbot/buildarea/twistd.pid O_PATH file descriptor to proper file descriptor: Permission denied

localhost systemd[1]: buildbot-worker.service: Can't convert PID files /home/buildbot/buildarea/twistd.pid O_PATH file descriptor to proper file descriptor: Permission denied

localhost audit[1]: AVC avc: denied { open } for pid=1 comm="systemd" path="/home/buildbot/buildarea/twistd.pid" dev="dm-0" ino=25387713 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

localhost setroubleshoot[8714]: SELinux is preventing systemd from open access on the file /home/buildbot/buildarea/twistd.pid. For complete SELinux messages run: sealert -l 3d54c639-fea4-4a18-be47-c5fe1d57a02a

localhost python3[8714]: SELinux is preventing systemd from open access on the file /home/buildbot/buildarea/twistd.pid.#012#012***** Plugin catchall (100. confidence) suggests **************************#12#012If you believe that systemd should be allowed open access on the twistd.pid file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#12# ausearch -c 'systemd' --raw | audit2allow -M my-systemd#012# semodule -X 300 -i my-systemd.pp#012`

After allowing that as well, the service starts normally.

buildworker.rst

tiran · 2020-05-21T15:57:10Z

I see. SELinux and systemd both treat /home as a privileged and restricted area. System services are generally not allowed to access /home tos prevents daemons from stealing ssh private keys, personal emails, or your cat pictures.

I know of four options to deal with this problem

move buildbot out of /home. For example deploy buildbot code to /opt/buildbot and use systemd RuntimeDirectory, StateDirectory, and LogsDirectory (/run, /var/log, ... see man systemd.exec)
run buildbot as systemd user service. You have to put the service file to /home/buildbot/.config/systemd/user/buildbot.service, enable lingering with loginctl enable-linger buildbot, and start the service from a buildbot login shell (not su/sudo!) as systemctl --user enable --now buildbot.service.
create a custom SELinux policy, types, and file contexts for buildbot.
make the init_t SELinux type permissive semanage permissive -a init_t

stratakis · 2020-06-09T22:41:49Z

I see. SELinux and systemd both treat /home as a privileged and restricted area. System services are generally not allowed to access /home tos prevents daemons from stealing ssh private keys, personal emails, or your cat pictures.

I know of four options to deal with this problem
* move buildbot out of `/home`. For example deploy buildbot code to `/opt/buildbot` and use systemd `RuntimeDirectory`, `StateDirectory`, and `LogsDirectory` (/run, /var/log, ... see man systemd.exec)

That would go against the current guidelines of setting up a buildbot worker (or requiring a bigger overhaul) so I wouldn't go with that option.

* run buildbot as systemd user service. You have to put the service file to `/home/buildbot/.config/systemd/user/buildbot.service`, enable lingering with `loginctl enable-linger buildbot`, and start the service from a buildbot login shell (not su/sudo!) as `systemctl --user enable --now buildbot.service`.

This solution I liked the most, however the systemctl --user option is not available in RHEL7.

* create a custom SELinux policy, types, and file contexts for buildbot.

Maybe that would be the best way then. Any pointers on how to work with that?

* make the init_t SELinux type permissive `semanage permissive -a init_t`

Not sure I would like to change init_t

ezio-melotti · 2022-07-12T01:43:48Z

If this PR is still relevant, it should be updated after the devguide reorganization, reviewed, and merged.

stratakis · 2022-07-18T13:36:52Z

If this PR is still relevant, it should be updated after the devguide reorganization, reviewed, and merged.

Thanks for the reminder. Is there an ETA for the devguide reorganization?

ezio-melotti · 2022-07-19T21:28:09Z

The devguide reorganization is done already, and there is now a conflict on the PR that must be resolved.

ezio-melotti · 2022-10-07T05:04:54Z

@stratakis can you merge main into your branch and resolve the conflict?

Provide examples for managing the buildbot-worker service through systemd unit files for systemd based distributions.

stratakis · 2022-10-10T15:17:06Z

Rebased.

ezio-melotti · 2022-10-11T11:11:03Z

Thanks!
@methane / @zware, can you (re-)review?

the-knights-who-say-ni added the CLA signed label May 15, 2020

tiran requested changes May 15, 2020

View reviewed changes

willingc added the needs: PR update An update or rebase to an existing PR is needed. label Oct 19, 2020

ezio-melotti closed this Jun 5, 2022

ezio-melotti reopened this Jun 5, 2022

ezio-melotti requested review from zware and tiran June 5, 2022 11:06

Add info for systemd based distros

1b64a98

Provide examples for managing the buildbot-worker service through systemd unit files for systemd based distributions.

stratakis force-pushed the buildbot_doc_revamp branch from be6a64f to 1b64a98 Compare October 10, 2022 15:15

hugovk removed the needs: PR update An update or rebase to an existing PR is needed. label Apr 17, 2023

willingc added the needs: PR update An update or rebase to an existing PR is needed. label Oct 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add info for systemd based distros #583

Add info for systemd based distros #583

stratakis commented May 15, 2020

Mariatta commented May 15, 2020

tiran May 15, 2020

stratakis May 15, 2020

tiran May 15, 2020

stratakis May 15, 2020

tiran May 19, 2020

stratakis May 19, 2020

stratakis May 21, 2020

tiran commented May 21, 2020

stratakis commented Jun 9, 2020

ezio-melotti commented Jul 12, 2022

stratakis commented Jul 18, 2022

ezio-melotti commented Jul 19, 2022

ezio-melotti commented Oct 7, 2022 •

edited

stratakis commented Oct 10, 2022

ezio-melotti commented Oct 11, 2022

Add info for systemd based distros #583

Are you sure you want to change the base?

Add info for systemd based distros #583

Conversation

stratakis commented May 15, 2020

Mariatta commented May 15, 2020

tiran May 15, 2020

Choose a reason for hiding this comment

stratakis May 15, 2020

Choose a reason for hiding this comment

tiran May 15, 2020

Choose a reason for hiding this comment

stratakis May 15, 2020

Choose a reason for hiding this comment

tiran May 19, 2020

Choose a reason for hiding this comment

stratakis May 19, 2020

Choose a reason for hiding this comment

stratakis May 21, 2020

Choose a reason for hiding this comment

tiran commented May 21, 2020

stratakis commented Jun 9, 2020

ezio-melotti commented Jul 12, 2022

stratakis commented Jul 18, 2022

ezio-melotti commented Jul 19, 2022

ezio-melotti commented Oct 7, 2022 • edited

stratakis commented Oct 10, 2022

ezio-melotti commented Oct 11, 2022

ezio-melotti commented Oct 7, 2022 •

edited