borg-k8s-volume-backup

Backing up kubernetes volumes on a node using borg backup

This repo will soon contain a docker image and kubernetes resource YAMLs for backing up kubernetes volumes in a cluster

Basic concept

• Kubelet mounts pod volumes as /var/lib/kubelet/pods/<pod_uuid>/volumes/<volume_type>/<volume_name>
  • NOTE! volume_name here refers to the "VOLUME" attribute listed in kubectl get pvc output, NOT the "NAME" attribute
• By iterating over /var/lib/kubelet/pods/*/volumes/*/*, we can run operations on each mounted volume on a k8s node
  • For example, by iterating over /var/lib/kubelet/pods/*/volumes/kubernetes.io~iscsi/*, we can back up each mounted iSCSI volume.

Short-term Goals

• Done, see k8s-volume-backup.py - requires python3
  • Generic script that uses env variables for volume type, borg repo location, and borg passphrase
  • Uses kubectl get pvc in backup script to get all PVCs, and build a volume name to PVC name dictionary
  • Uses this dictionary to save backups using the PVC name rather than the volume name
    • The PVC name is typically specified by the user in a PVC resource YAML
    • The volume name may be automatically generated by a dynamic volume provisioner
    • Goal is to have archives in borg repo named something like sonarr-2020-03-01 rather than pvc-b1397318-8fa4-4216-aa6b-6568d1394e89-2020-03-01
    • Archive contents will be a 1:1 reproduction of the PVC storage contents at time of archive creation
• TBD:
  • Generic script to restore a borg backup to a k8s volume

End goals

• Create a docker image that can do backup/restory operations
  • Mount /var/lib/kubelet/pods/ using hostPath
  • Run backup script in container to back up all mounted PVCs on the kubernetes node
• Create a DaemonSet, so that one can easily back up all PVCs in a cluster by running the container on each node
  • Maybe even a DaemonJob, if that gets merged.
• Create a docker container that can do restore operations
• Create a Job template file that can be used to easily restore a borg backup archive to a PVC