Update listmonk@.service

[MaximilianKohler]

#843 (comment)

[kmohrf]

Your commit breaks the service template unit. That %i is there for a reason. Please try to understand what you modify.

[MaximilianKohler]

Your commit breaks the service template unit. That %i is there for a reason. Please try to understand what you modify.

Are you talking about ConditionPathExists=/etc/listmonk/%i.toml ? Per the referenced issue #843 it causes an error, so this guide, discussed in that issue, recommends changing it.

I didn't try without changing those lines since that person had an issue + the guide says to change them, but it worked according to the guide, so that's why I suggested changing it.

[kmohrf]

Yes the guide you’ve linked to recommends changing some lines and in doing so breaks a very useful feature of the systemd unit: the ability to run more than one instance of listmonk with different configurations but the same service unit.

The lines you’ve removed (and replaced) in your PR are implementing a behavior that is intentional. If there is a bug in the implementation I’m all for fixing that. But you’ve changed the entire nature of the unit.

[MaximilianKohler]

Good to know. I tagged you in the issue because I assumed you know more about this than me. So please feel free to make specific suggestions/changes.

[MaximilianKohler]

Apparently my listmonk service stopped on its own:

systemctl status listmonk
● listmonk.service
   Loaded: loaded (/etc/systemd/system/listmonk.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Sat 2023-12-09 21:18:45 PST; 23h ago

Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:24] Not an absolute path, ignoring: %S/listmonk-
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:47] Unknown lvalue 'ProtectControlGroups' in s...ervice'
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:48] Unknown lvalue 'ProtectKernelTunables' in ...ervice'
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:56] Invalid device node path 'False'. Ignoring.
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:58] Unknown lvalue 'RestrictNamespaces' in sec...ervice'
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:60] Unknown lvalue 'RestrictRealtime' in secti...ervice'
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:65] Unknown lvalue 'MemoryDenyWriteExecute' in...ervice'
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:67] Unknown lvalue 'LockPersonality' in sectio...ervice'
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:71] Unknown lvalue 'ProtectKernelModules' in s...ervice'
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:74] Unknown lvalue 'PrivateUsers' in section 'Service'
Hint: Some lines were ellipsized, use -l to show in full.

Is it possible to send an email notification when that happens? I know my CSF firewall is able to email me about events. I modified these lines to try to fix the issue:
I commented out the 2nd line here:

EnvironmentFile=-/etc/default/listmonk
# EnvironmentFile=-/etc/default/listmonk-%i

I'm not sure whether to comment out these lines:

StateDirectory=listmonk-%i
Environment=HOME=%S/listmonk-%i
WorkingDirectory=%S/listmonk-%i

Or change them to:

StateDirectory=/etc/listmonk
Environment=HOME=/etc/listmonk
WorkingDirectory=/usr/bin

I was thinking to simplify it the same way bamboowonder did, but that guide doesn't cover creating a new user for User=<yourUser>. I saw another page showing that you can run it without specifying a user, so for now I've done that.

However, as noted here #843 (comment) another way to "fix this" is by running systemctl restart listmonk after systemctl daemon-reload. Is there a way to make it do that automatically?

Oh, I think it didn't restart after the server rebooted. So this needs to be changed too:
Restart=on-failure to Restart=always or something else: https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html

Summary:

This file needs major changes to be more compatible. I suggested in the issue-thread that we could have two versions or two sets of commands in the same file with one set commented out by default.

Perhaps we could leave the current file mostly the same (and/or rename it to listmonk-multi@.service) and create a new listmonk-simple@.service file containing:

[Unit]
Description=listmonk email service
ConditionPathExists=/etc/listmonk/config.toml
Wants=network.target
# The PostgreSQL database may not be on the same host but if it
# is listmonk should wait for it to start up.
After=postgresql.service

[Service]
Type=simple
PermissionsStartOnly=true
WorkingDirectory=/usr/bin
EnvironmentFile=-/etc/default/listmonk
ExecStartPre=/usr/bin/mkdir -p "/listmonk/uploads"
ExecStartPre=/usr/bin/listmonk --config /etc/listmonk/config.toml --upgrade --yes
ExecStart=/usr/bin/listmonk --config /etc/listmonk/config.toml $SYSTEMD_LISTMONK_ARGS
Restart=always

# Use systemd’s ability to disable security-sensitive features
# that listmonk does not explicitly need.
# NoNewPrivileges should be enabled by DynamicUser=yes but systemd-analyze
# still recommended to explicitly enable it.
NoNewPrivileges=True
# listmonk doesn’t need any capabilities as defined by the linux kernel
# see: https://man7.org/linux/man-pages/man7/capabilities.7.html
CapabilityBoundingSet=
# listmonk only executes native code with no need for any other ABIs.
SystemCallArchitectures=native

# Make /home/, /root/, and /run/user/ inaccessible.
# If you set ExecStartPre=/usr/bin/mkdir -p "listmonk/uploads" to a directory in /home/ or /root/ it will cause uploads to fail 
# See https://github.com/knadh/listmonk/issues/843#issuecomment-1836023524
ProtectHome=True

# Make sure files created by listmonk are only readable by itself and
# others in the listmonk system group.
UMask=0027
# listmonk only needs to support the IPv4 and IPv6 address families.
RestrictAddressFamilies=AF_INET AF_INET6

[Install]
WantedBy=multi-user.target

Maybe ExecStartPre=/usr/bin/mkdir -p "/listmonk/uploads" should be changed to /etc/listmonk/uploads.

And still missing is a static directory. https://listmonk.app/docs/templating/#system-templates

EDIT:

Changing Restart=on-failure to Restart=always didn't work. It still didn't start up after a reboot:

systemctl status listmonk -l
● listmonk.service
   Loaded: loaded (/etc/systemd/system/listmonk.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Sun 2023-12-10 21:48:59 PST; 1min 35s ago
  Process: 861 ExecStartPre=/usr/bin/listmonk --config /etc/listmonk/config.toml --upgrade --yes (code=exited, status=1/FAILURE)
  Process: 844 ExecStartPre=/usr/bin/mkdir -p /listmonk/uploads (code=exited, status=0/SUCCESS)

Dec 10 21:48:59 domain systemd[1]: listmonk.service: control process exited, code=exited status=1
Dec 10 21:48:59 domain systemd[1]: Failed to start listmonk.service.
Dec 10 21:48:59 domain systemd[1]: Unit listmonk.service entered failed state.
Dec 10 21:48:59 domain systemd[1]: listmonk.service failed.
Dec 10 21:48:59 domain systemd[1]: listmonk.service holdoff time over, scheduling restart.
Dec 10 21:48:59 domain systemd[1]: Stopped listmonk.service.
Dec 10 21:48:59 domain systemd[1]: start request repeated too quickly for listmonk.service
Dec 10 21:48:59 domain systemd[1]: Failed to start listmonk.service.
Dec 10 21:48:59 domain systemd[1]: Unit listmonk.service entered failed state.
Dec 10 21:48:59 domain systemd[1]: listmonk.service failed.

But if I restart the listmonk service it works:

systemctl restart listmonk
systemctl status listmonk -l

● listmonk.service
   Loaded: loaded (/etc/systemd/system/listmonk.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2023-12-10 21:54:25 PST; 2s ago
  Process: 2435 ExecStartPre=/usr/bin/listmonk --config /etc/listmonk/config.toml --upgrade --yes (code=exited, status=0/SUCCESS)
  Process: 2433 ExecStartPre=/usr/bin/mkdir -p /listmonk/uploads (code=exited, status=0/SUCCESS)
 Main PID: 2444 (listmonk)
   CGroup: /system.slice/listmonk.service
           └─2444 /usr/bin/listmonk --config /etc/listmonk/config.toml

Dec 10 21:54:25 domain listmonk[2435]: 2023/12/10 21:54:25 init.go:145: reading config: /etc/listmonk/config.toml
Dec 10 21:54:25 domain listmonk[2435]: 2023/12/10 21:54:25 init.go:273: connecting to db: localhost:5432/listmonk
Dec 10 21:54:25 domain listmonk[2435]: 2023/12/10 21:54:25 upgrade.go:64: no upgrades to run. Database is up to date.
Dec 10 21:54:25 domain systemd[1]: Started listmonk.service.
Dec 10 21:54:25 domain listmonk[2444]: 2023/12/10 21:54:25 main.go:102: v2.5.1 (a6a2b69 2023-08-15T15:49:28Z, linux/amd64)
Dec 10 21:54:25 domain listmonk[2444]: 2023/12/10 21:54:25 init.go:145: reading config: /etc/listmonk/config.toml
Dec 10 21:54:25 domain listmonk[2444]: 2023/12/10 21:54:25 init.go:273: connecting to db: localhost:5432/listmonk
Dec 10 21:54:25 domain listmonk[2444]: 2023/12/10 21:54:25 init.go:593: media upload provider: filesystem
Dec 10 21:54:25 domain listmonk[2444]: 2023/12/10 21:54:25 init.go:517: loaded email (SMTP) messenger: xxx@email-smtp.us-west-1.amazonaws.com
Dec 10 21:54:25 domain listmonk[2444]: ⇨ http server started on 127.0.0.1:9000

I removed that line and it still failed to start after rebooting the server:

systemctl status listmonk -l
● listmonk.service - listmonk email service
   Loaded: loaded (/etc/systemd/system/listmonk.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Sun 2023-12-10 22:15:32 PST; 8s ago
  Process: 905 ExecStart=/usr/bin/listmonk --config /etc/listmonk/config.toml (code=exited, status=1/FAILURE)
  Process: 902 ExecStartPre=/usr/bin/mkdir -p /listmonk/uploads (code=exited, status=0/SUCCESS)
 Main PID: 905 (code=exited, status=1/FAILURE)

Dec 10 22:15:32 domain systemd[1]: Unit listmonk.service entered failed state.
Dec 10 22:15:32 domain systemd[1]: listmonk.service failed.
Dec 10 22:15:32 domain systemd[1]: listmonk.service holdoff time over, scheduling restart.
Dec 10 22:15:32 domain systemd[1]: Stopped listmonk email service.
Dec 10 22:15:32 domain systemd[1]: start request repeated too quickly for listmonk.service
Dec 10 22:15:32 domain systemd[1]: Failed to start listmonk email service.
Dec 10 22:15:32 domain systemd[1]: Unit listmonk.service entered failed state.
Dec 10 22:15:32 domain systemd[1]: listmonk.service failed.

I added these lines from one of the guides:

TimeoutStopSec=10
RestartSec=5

And put it back to Restart=on-failure. And that worked.

[MaximilianKohler]

@kmohrf does everything look good now with this service and the new simple.service? The only thing I'm not sure about is the HOME in Environment=HOME=%S/listmonk-%i, and if I broke something by changing ExecStartPre=/usr/bin/mkdir -p "${HOME}/uploads".

Also, I understand that %S is a placeholder, but I'm not sure what value/location it's referencing in this script/file. I would like to add some explanation of what folders it's putting things in and reading from, but I don't fully understand how it

runs more than one instance of listmonk with different configurations but the same service unit.

[MaximilianKohler]

Logs:

I'm not sure which is best, but when reviewing https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html I saw these options and added them commented out in the simple.service https://github.com/knadh/listmonk/blob/master/listmonk-simple.service file:

#LogsDirectory=/etc/listmonk
#CacheDirectory=/etc/listmonk/cache

I assume that listmonk doesn't use a cache, so I can probably remove that? I don't know how to get listmonk to save logs with that though, and this (alternative?) may be the easiest/best way (which is working for me):

ExecStart=/bin/bash -ce "exec /usr/bin/listmonk --config /etc/listmonk/config.toml --static-dir /etc/listmonk/static >>/etc/listmonk/listmonk.log 2>&1"

So I added it as an option. I'm not sure if it conflicts with anything else.

There are other ways to do it, but they vary based on systemd version #1462 (comment).