Systemd export: provide $PS to processes, no longer use systemd templates

[RobinDaugherty]

The systemd export implementation was designed using systemd templates. When system templates are instantiated, the only piece of information that could be given to an instantiated instance is a single string coming after the "@", which as currently implemented is the port number.

Port numbers as part of the service name are not useful for humans or for scripting because they change when the process formation changes—they're assigned sequentially when each service is exported, so there's no continuity between exports if the formation is changed.

When foreman is run directly, it provides a $PS environment variable that can be used within the Procfile and within each process, to allow the process to be identified in the context of the app. This is in the form "(process_name).num" and is used standardly throughout foreman (for example when logging).

By removing the use of systemd templates, each generated systemd service gets a full service file that can have multiple pieces of information included. The $PORT and $PS environment variables are set within the file.

Syslog will continue to receive the full service name, but the service name is now "(app_name)-(process_name).(index)" instead of "(app_name)-(process_name)@(port)". These are the names used when controlling the services (e.g. systemctl stop myapp-sidekiq.1.service).

In addition, a change was made to allow all of the services to be controlled as a group using the app target. This was done by adding StopWhenUnneeded=yes to each service as documented here. So the commands systemctl start myapp.target and systemctl stop myapp.target will start and stop the entire formation.

While the app target that is exported is configured to be wanted by the multi-user target, that doesn't take effect unless the target is enabled. So to cause the app services to be started automatically on system start, the command systemctl enable myapp.target must be issued once after foreman has exported.

This is a breaking change, so anyone using systemd export will need to be aware of this. I'm not sure how to approach backward compatibility with this. One possibility is to preserve the existing systemd export method as-is and rename my new method to something else like "systemd2".

[ddollar]

Thanks!

[ddollar]

Released in 0.87.0

[RobinDaugherty]

@ddollar This was a breaking change, I really don't think it should be released with a minor version bump.

[jarthod]

Well indeed I just broke my setup without realizing it thanks to this minor update :) I also didn't found any breaking change warning in the changelog when I looked for it. I would happily write a PR to improve the changelog but I'm not even sure of what would be the instructions to migrate as I'm still trying to figuring out how to backport this (because in addition to that I've been using custom systemd template to enable file logging).

Not a big deal of course this changes are welcomed but I agree with @RobinDaugherty the release we probably a bit too quiet ;)

[RobinDaugherty]

@jarthod sorry about that. Can you elaborate on the issues you ran into? I can try to write something for the changelog. My expectation was that it would only cause problems when you're inspecting the logs for a specific job. Are there other use cases that affected you?

[jarthod]

@RobinDaugherty no problem, it's not your fault ;)

Well I believe the first one was that I used to restart my services using app-service.target but these targets do not exists any more now it's app.target. Which is great, it's simpler, but it was unexpected for me. And also I think it may be more complex now if we want to restart a single service (which has several instances) without the others? (I haven't really tried and I'm not an expert in systemctl so there may be an easy way)

After that I noticed a couple days later that the change in service names (from @port to .number) spawned a whole new set of processes without removing the previous ones which means all my background processes were duplicated and I had to stop each of them individually because they were not included in any target any more, the systemd files where gone but the process were still running like orphans.

Then I noticed that restarting app.target was not working for me because I had to customize the systemd templates (because of #717) to add file logging and thus I was missing the new StopWhenUnneeded=yes directive. Had to dive into the merge requests to find that information and add it back to my template.

[RobinDaugherty]

So it sounds like the following instructions should be added to the release notes, perhaps as post_install_instructions:

1. To start/stop/restart all background services, use systemctl stop appname or service appname stop.
2. To start/stop/restart all background services with a specific process name, use systemctl stop appname-processname or service appname-processname stop.
3. To start/stop/restart a single background services where more than one of the same process is in the formation, use systemctl stop appname-processname.number or service appname-processname.number stop.
4. If you customized the systemd templates, update them by comparing to the current versions of master.target.erb and process.service.erb.
5. When you deploy your application with the new version of foreman, stop all background services before deploying (unless your deployment process stops all services before deploying).

Does that cover all of it @jarthod?

[jarthod]

Thanks for providing these,

1. Not quite, systemctl stop/start/restart appname doesn't work because there's no such "service", it's a target, so we have to type appname.target. In my instance:

> sudo systemctl restart updown
Failed to restart updown.service: Unit updown.service not found.

2. I don't believe this is true, it was possible before with appname-service.target but these don't exists any more so I don't see any way to do this with systemctl currently.

For 3, 4, 5 that sounds good 👍