Routine backups of PV/PVC objects

[rkevin-arch]

This is a #showerthoughts moment, but our galaxy cluster is designed so we can bring it back up easily from scratch. Even throughout the development process, we often teared the entire cluster apart and rebuilt from scratch using protogalaxy. In our migration procedure, we backed up the PV/PVC objects so we can import them to the new cluster, but backing up PV/PVC objects might be a good thing to do in general. In that case, we can rebuild our cluster as often as we like and not lose any user data (assuming hen is alive, and we back up PV/PVCs regularly). This might be a good thing to add to regular cluster maintenance.

[moorepants]

Why can't this information be retrieved from the backups of hen? Or do you want it in a more readily accessible form?

[rkevin-arch]

The way JupyterHub works is it creates PVCs, which creates PVs, which map to an underlying NFS directory. If a PVC gets deleted, the PV is also deleted, but the user data stays on NFS. However, if a new PVC with the same name is created, a new PV with a different autogenerated name gets created, which uses a different underlying NFS directory. The user data is still there but inaccessible in that case. By importing old PV/PVC objects, our new JupyterHub can start using the existing PVCs rather than create new ones.

[celine168]

Hm...maybe we could put the commands that are used to back up the YAML file into a script and run a cron job saving that file somewhere every so often?

[rkevin-arch]

Probably a good idea. We can bring it up in the next meeting and see what everyone else thinks.