Data and Backups

Data Storage

Overleaf Community Edition and Overleaf Server Pro store their data in 3 separate places:

• Mongo Database: user and project data lives here.
• Redis: used as high-performant cache for in-flight data, it mainly stores data from project edition and collaboration.
• Filesystem: stores project files (such as images) and it's also used as a temporary disk cache during project compilations.
  • Note: this might be ~/sharelatex_data or ~/overleaf_data, depending on when you're instance was setup.

See section "Folders in detail" for details on the folder layout on disk.

Performing a backup

There are 3 things which need to be backed up

• Mongodb
• Overleaf Filesystem data
• Redis

In order to produce a consistent backup it is mandatory to stop users from producing new data while the backup process is running.

We advise setting up a maintenance window during which users are unable to edit their projects.

For Server Pro 3.4.0 and earlier:

• The admin interface https://overleaf.example.com/admin (Close Editor tab) provides buttons for closing the editor and disconnecting users. Closing the editor stops users from opening new editor sessions, but keeps existing sessions open. Disconnecting users is happening with a 10 seconds grace period before the page reloads.
• You will need to close the editor first, then disconnect all users.

Starting with Server Pro 3.5.0 the shutdown down process automates the closing of the site and user disconnections.

Stop the sharelatex container and start the backup process.

Once you are done with the backup you will need to restart the sharelatex container to restore editing capability.

Backup Tips

Backups need to be on a separate server to the one Overleaf running on, ideally in a different location entirely. Running multiple instances of mongodb is also not a backup as it will not prevent against corruption.

Mongodb

mongodb comes with a tool called mongodump which will export the dataset in a safe backup format.

Filesystem data

The path where your files are stored is specified in your settings file. This might be: /var/lib/overleaf (/var/lib/overleaf for versions 4.x and earlier), copying this directory recursively is needed. rsync is a good tool for the job.

Redis

Redis stores user sessions and pending document updates before they are flushed to Mongodb. To backup redis you can copy the RDB file to a secure location.

Migrating data between servers

At best you do not have any valuable data in the new instance yet. We do not have a process for merging the data of instances.

Assuming the new instance has not data yet, here are some steps you could follow. On a high level we produce a tar-ball of the mongo, redis and sharelatex volumes, copy it over to the new server and inflate it there again.

With the default docker-compose file that would be:

# Gracefully shutdown the old instance
# (See https://github.com/overleaf/overleaf/wiki/Data-and-Backups#performing-a-backup)
old-server$ docker stop sharelatex
old-server$ docker stop mongo redis

# Create the tar-ball
# Note that ~/sharelatex_data can also be ~/overleaf_data
old-server$ tar --create --file backup-old-server.tar ~/sharelatex_data ~/mongo_data ~/redis_data

# Copy the backup-old-server.tar file from the old-server to the new-server using any method that fits

# Gracefully shutdown new instance (if started yet)
new-server$ docker stop sharelatex
new-server$ docker stop mongo redis

# Move new data, you can delete it too
new-server$ mkdir backup-new-server
# Note that ~/sharelatex_data can also be ~/overleaf_data
new-server$ mv ~/sharelatex_data ~/mongo_data ~/redis_data backup-new-server/

# Populate data dirs again
new-server$ tar --extract --file backup-old-server.tar

# Start containers
new-server$ docker start mongo redis
new-server$ docker start sharelatex

Depending on your docker-compose config, you may need to adjust the paths of the mongo/redis/sharelatex volume.

With a toolkit setup that would be:

# Gracefully shutdown the old instance
# (See https://github.com/overleaf/overleaf/wiki/Data-and-Backups#performing-a-backup)
old-server$ bin/stop

# Create the tar-ball
old-server$ tar --create --file backup-old-server.tar config/ data/

# Copy the backup-old-server.tar file from the old-server to the new-server using any method that fits

# Gracefully shutdown new instance (if started yet)
new-server$ bin/stop

# Move new data, you can delete it too
new-server$ mkdir backup-new-server
new-server$ mv config/ data/ backup-new-server/

# Populate config/data dir again
new-server$ tar --extract --file backup-old-server.tar

# Start containers
new-server$ bin/up

Folders in detail

1. ~/mongo_data (b)
   • mongodb datadir
2. ~/redis_data (b)
   • redis db datadir
3. ~/sharelatex_data (or ~/overleaf_data)
   1. bin
      1. synctex (d)
         • unused in latest release, previously a custom synctex binary was used (synctex is used for source mapping between .tex files and the pdf)
   2. data
      1. cache (e)
         • binary file cache for compiles
      2. compiles (e)
         • latex compilation happens here
      3. db.sqlite (d)
         • unused in latest release, previously stored clsi cache details (either moved to simple in-memory maps or we scan the disk)
      4. db.sqlite-wal (d)
         • unused in latest release, see db.sqlite
      5. output (e)
         • latex compilation output storage for serving to client
      6. template_files (b)
         • image previews of template system (Server Pro only)
      7. user_files (b)
         • binary files of projects
      8. history (b)
         • full project history files
   3. tmp
      1. dumpFolder (e)
         • temporary files from handling zip files
      2. uploads (e)
         • buffering of file uploads (binary file/new-project-from-zip upload)
      3. projectHistories (e)
         • temporary files for full project history migrations

The folders have additional hints:

• (b) include in backups, best when the instance is stopped to ensure consistency
• (d) can be deleted
• (e) ephemeral files, can be deleted when the instance is stopped