WIP: backup and restore scripts #72
Conversation
|
cc @mserranom @gh2k |
| local mongo_tmp_dir="$tmp_dir/backup/mongo" | ||
| mkdir "$mongo_tmp_dir" | ||
|
|
||
| "$TOOLKIT_ROOT/bin/docker-compose" up -d mongo |
There was a problem hiding this comment.
Can we handle the case where there is an external Mongo DB here? I think there will be people running without the internal DBs.
There was a problem hiding this comment.
Yeah, would be useful to have a --skip-mongo option for both backup/restore.
There was a problem hiding this comment.
In addition, if we go with external redis, it could also have a --skip-redis
There was a problem hiding this comment.
I was thinking that we could look at the config in some way and back up the external mongo/redis using the same mechanism. Given that we've already checked the prerequisites for making a backup (the service is stopped so nothing will be in-flight) it makes sense to roll up the backup from the external DB at the same time. WDYT?
| "$TOOLKIT_ROOT/bin/docker-compose" stop 2>/dev/null | ||
|
|
||
| ## Dump mongo | ||
| echo "Dumping mongo..." |
There was a problem hiding this comment.
Do we also need to back up redis?
There was a problem hiding this comment.
I think we have gone back and forth on this, the docs used to say it wasn't needed but were changed earlier this year: https://github.com/overleaf/overleaf/wiki/Backup-of-Data/_compare/dd55f820bf868464db70cbf8f584a608b6425c1b...f07e059281545c18907f9971e0fd18d9492b0127
I believe it's the case that if the editor is closed, users are disconnected, and then everything is shut down cleanly (in that order), nothing left in Redis is absolutely essential. But since we don't currently have a way to close the editor and disconnect users from a script, it seems safer to back up both, given various problems that admins have run into on support. (I think scripting the close editor and disconnect users actions was being discussed elsewhere – if we add that to the backup script then maybe Redis backup is indeed not needed.)
| echo '[mongo is ready]' | ||
| } | ||
|
|
||
| function create-tmp-dir () { |
There was a problem hiding this comment.
Could mktemp be used here?
| rm -rf /tmp/dump;' | ||
| } | ||
|
|
||
| function copy-data-files () { |
There was a problem hiding this comment.
Are data files really needed for the backup? Most of them would be intermediate compile results right?
There was a problem hiding this comment.
I assumed that there would be e.g. filestore data here?
|
I think the The reason is that, according to docs, we need to gracefully close all editor sessions before attempting a backup. This should be done separately, maybe as part of |
Given that users are likely to want to run this on a cron, is there a way to send a disconnectAllUsers, prevent reconnection and wait for them to all disconnect, and then make a backup? If the only mechanism to do this is manual, then it's tricky to get a backup done on a sensible schedule. |
There isn't unfortunately, see https://github.com/overleaf/internal/issues/4951 |
|
How do we feel about resurrecting this work? @gh2k |
|
I guess this falls under @cmcisaac6's remit... I like the idea of having a canonical way to perform a backup though. |
|
Could we create a ticket to cover this so that it's tracked on the main board, please? |
Description
Work in progress. Do not merge.
This needs a lot more work to be usable in practice, but... the backup and restore process works, just like in our main monorepo.
Related issues / Pull Requests
Contributor Agreement