Automate backups & exports hourly

[Mag1cByt3s]

I just had an issue with the database after upgrading.

I had no success in fixing the errors.

Thankfully i still had a backup but that was a few hours old already.

Since my documents are very mission-critical i would like to automatically backup everything every hour.

So far I only know the export function which I can trigger manually.

I would like to automate that and save it to my storage server. I could easily mount a remote nfs path to save the backups but i have no idea how i can automatically request and save the export.

Any help with that would be appreciated.

Thanks.

[tommoor]

The best way to backup on such a regular basis would be to take a snapshot of the server disk, this is a process that would be done completely outside of the Outline software. Depending on your host it may have such capability built into the server management.

[Mag1cByt3s]

I am currently taking full snapshots of the RAID array every 24 hours.

Also i am taking backups of the outltine folder including database every hour via an rsync cronjob.

Still the database was corrupted even after replacing the backup it could only be restored with the full backup i had on TrueNAS. But that was 24 hours old.

[tommoor]

It would be interesting to get more details about the database corruption. That sounds like a pretty rare scenario..

[Mag1cByt3s]

I don't have logs about these errors anymore.

But it complained about columns in the database being already existend as it tried to run the migration script on startup after the update.

I tried to edit the db manually by deleting these columns but that just caused more issues.

[Mag1cByt3s]

So small update to this: My database got just nuked again. I just opened outline and my data was completely gone.
I have no clue what happened i only saw authorization errors in the logs.
After copying over a postgres backup my documents are back but I assume that some data is lost.
The problem is that i have so many notes that i am not sure what is lost.

I need a reliable way to backup everything.

But at least backing up the postgres folder seems to be enough if the db version did not change.

[tommoor]

My database got just nuked again

It sounds like your postgres container storage is not setup correctly, you can get behavior similar to this if using Docker without mapping the data directory to a volume.

[Mag1cByt3s]

Here is my current docker-compose stack: https://privatebin.net/?e3e5fb0b2b3836f4#BXN89x76FsTxF8c1qUoG6crLHeyZXpp8i2sU6AEEceJD

maybe someone can have a look.

The mount point /mnt/nas/outline is mounted via NFS from TrueNAS.

[Mag1cByt3s]

I am using TrueNAS to store the DB on a RAID1 system. I wonder if i can snapshot the postgres folder every 30 minutes or so.

I am also now considering taking off-site backups.

Otherwise i might have to look into using the API to export my collections: https://www.getoutline.com/developers#tag/Collections/paths/~1collections.export/post

It would be interesting to see how other people handle the backup strategy here.

My current backup strategy are just a few cronjobs:

# create an hourly backup of outline
0 * * * * /usr/bin/rsync -a /mnt/pve/truenas/outline/ /mnt/pve/truenas/Backups/outline.hourly >/dev/null 2>&1
0 * * * * /usr/bin/rsync -a /mnt/pve/truenas/outline/ /papa/storage/Backups/Pascal/outline.hourly >/dev/null 2>&1

# create a half-hourly backup of outline
*/30 * * * * /usr/bin/rsync -a /mnt/pve/truenas/outline/ /mnt/pve/truenas/Backups/outline.halfhourly >/dev/null 2>&1

TrueNAS is snapshotting and backuping the whole RAID every 24 hours to a backup drive.

[Mag1cByt3s]

While looking through the issues here on Github i stumbled upon this: https://github.com/captn3m0/outliner
That seems to be exactly what i need.

[tommoor]

I would just warn that it looks like that library hasn't been updated in 4 years, the API is mostly compatible over that time but there have also been many changes as you can imagine.

[Mag1cByt3s]

True. I also noticed that. I was hoping it would still be compatible. I may have to build something on my own that talks to the API and downloads an export.