Borg systemd service not working

[matthuszagh]

I have a systemd service that's intended to backup a BTRFS snapshot of my home directory to remote storage (borgbase). The script works exactly as expected when run manually from a shell, but sleeps indefinitely when run as part of the systemd service. The service config is:

[Unit]
Description=Backup snapshots to borgbase.

[Service]
Environment="LOCALE_ARCHIVE=/nix/store/in621vh2kj0ayqa6qc9pqnjvx6hzj5h5-glibc-locales-2.32-46/lib/locale/locale-archive"
Environment="PATH=/nix/store/1l4apnk802cbwzh3m7lympwlz0p1lpf0-borgbackup-1.1.16/bin:/nix/store/vqwk0h76mrpnckxcn30hlvansds8jvm1-gawk-5.1.0/bin:/nix/store/k1fqj7gz82nz0mdjxzgd52ggll2asnqk-openssh-8.6p1/bin:/nix/store/64zqi7kv3jd1cjv0yjg1396041p11j7c-sshpass-1.06/bin:/nix/store/a4v1akahda85rl9gfphb07zzw79z8pb1-coreutils-8.32/bin:/nix/store/1hvm45djn8wkfg64gbmlqpfj4dnjh594-findutils-4.7.0/bin:/nix/store/7n3yzh9wza4bdqc04v01xddnfhkrwk2a-gnugrep-3.6/bin:/nix/store/g34ldykl1cal5b9ir3xinnq70m52fcnq-gnused-4.8/bin:/nix/store/iriz2f82dq1sbkzjkjy6cggbb7wnrpz4-systemd-247.6/bin:/nix/store/1l4apnk802cbwzh3m7lympwlz0p1lpf0-borgbackup-1.1.16/sbin:/nix/store/vqwk0h76mrpnckxcn30hlvansds8jvm1-gawk-5.1.0/sbin:/nix/store/k1fqj7gz82nz0mdjxzgd52ggll2asnqk-openssh-8.6p1/sbin:/nix/store/64zqi7kv3jd1cjv0yjg1396041p11j7c-sshpass-1.06/sbin:/nix/store/a4v1akahda85rl9gfphb07zzw79z8pb1-coreutils-8.32/sbin:/nix/store/1hvm45djn8wkfg64gbmlqpfj4dnjh594-findutils-4.7.0/sbin:/nix/store/7n3yzh9wza4bdqc04v01xddnfhkrwk2a-gnugrep-3.6/sbin:/nix/store/g34ldykl1cal5b9ir3xinnq70m52fcnq-gnused-4.8/sbin:/nix/store/iriz2f82dq1sbkzjkjy6cggbb7wnrpz4-systemd-247.6/sbin"
Environment="TZDIR=/nix/store/y4j4k0l6w941wriprxz13dhvz896lw3m-tzdata-2020f/share/zoneinfo"



ExecStart=/nix/store/j2qhiw66fyrf0z2zg3i7i2l2adllgh9j-source/hosts/profiles/borg-backup/borg-backup.sh
User=matt

The backup script is

#!/bin/sh

# get the latest snapshot
SNAPSHOT_DATETIME=$(find /.snapshots/home/* -maxdepth 0 -name "home\.*" | tail -n 1 | awk '{print substr ($0, 23, 42)}')
SNAPSHOT_FILE="/.snapshots/home/home.${SNAPSHOT_DATETIME}"

export BORG_REPO=fs828tyj@fs828tyj.repo.borgbase.com:repo
export BORG_PASSCOMMAND="cat /home/matt/src/nixos/secrets/borg-backup-password"
export BORG_RSH="sshpass -f /home/matt/src/nixos/secrets/ssh-password-matt ssh -i /home/matt/.ssh/id_rsa"

info() { printf "\n%s %s\n\n" "$( date )" "$*" >&2; }
trap 'echo $( date ) Backup interrupted >&2; exit 2' INT TERM

info "Starting backup"

# backup
borg create                             \
    --verbose                           \
    --filter AME                        \
    --list                              \
    --stats                             \
    --show-rc                           \
    --compression zstd,2                \
    --exclude-caches                    \
    ::"{hostname}-${SNAPSHOT_DATETIME}" \
    ${SNAPSHOT_FILE}

backup_exit=$?

exit ${backup_exit}

Using strace on the borg create ... process showed a bunch of lines like

select(8, [5 7], [], [4 5 7], {tv_sec=1, tv_usec=0}) = 0 (Timeout)
select(8, [5 7], [], [4 5 7], {tv_sec=1, tv_usec=0}) = 0 (Timeout)
select(8, [5 7], [], [4 5 7], {tv_sec=1, tv_usec=0}) = 0 (Timeout)
select(8, [5 7], [], [4 5 7], {tv_sec=1, tv_usec=0}) = 0 (Timeout)

The file descriptors associated with the process are:

total 0
lr-x------ 1 matt users 64 Feb 21 08:40 0 -> /dev/null
lrwx------ 1 matt users 64 Feb 21 08:40 1 -> 'socket:[78715895]'
lrwx------ 1 matt users 64 Feb 21 08:40 2 -> 'socket:[78715895]'
l-wx------ 1 matt users 64 Feb 21 08:40 4 -> 'pipe:[78935047]'
lr-x------ 1 matt users 64 Feb 21 08:40 5 -> 'pipe:[78935048]'
lr-x------ 1 matt users 64 Feb 21 08:40 7 -> 'pipe:[78935049]'

What am I doing wrong here?

Full disclosure, I've cross-posted this from here, but I think maybe this is a better location for it.

[infectormp]

Did you read the guide with systemd example from borg docs? https://borgbackup.readthedocs.io/en/stable/deployment/automated-local.html

[matthuszagh]

Yeah I saw that but I don't really see why that works and mine doesn't. It's still oneshot service with an execstart that runs a bash script with borg create in it. Maybe I'm missing some salient difference, but it's not clear to me.

[RonnyPfannschmidt]

Additionally, your service doesn't seem to set the one-shot type

[matthuszagh]

Sorry, I should have clarified. It's not, but I tried one-shot too and got the same result.

[matthuszagh]

What should I set PYTHON_UNBUFFERED to? 1? What does this do? I'm having trouble finding info about it.

[matthuszagh]

Is that supposed to be PYTHONUNBUFFERED?

[matthuszagh]

I tried PYTHONBUFFERED=1. Not seeing any difference.

[infectormp]

I see you are using some %-based variable. Maybe it's unrelaited but there are also small note

When invoking borg from unit files, pay particular attention to escaping, especially when using the now/utcnow placeholders, since systemd performs its own %-based variable replacement even in quoted text. To avoid interference from systemd, double all percent signs ({hostname}-{now:%Y-%m-%d_%H:%M:%S} becomes {hostname}-{now:%%Y-%%m-%%d_%%H:%%M:%%S}).

[matthuszagh]

Oh yeah I saw that too, but I don't think it applies here as that info function works ok.