S3

S3 Setup

This document covers the setup of S3 in Overleaf Community Edition and Server Pro. A separate document can be found on migrating existing data to S3.

S3 compatible object storage options

Here are the most popular options for S3 compatible object storage:

• AWS S3, managed, we suggest picking AWS S3 when running Server CE/Pro on AWS
• Min.io, self-hosted
• Ceph.io, self-hosted
• Other hosting providers also have some kind of managed S3 compatible object storage, you may want to use these instead of running your own when already running Server CE/Pro at such a provider.

Latency considerations when picking a S3 compatible object storage

The latency between the Server CE/Pro instance and your S3 compatible object storage is a big contributor to the time it takes to complete the migration. The latency also impacts the file-upload performance in Server CE/Pro and slow file-downloads can have a big impact on PDF compile times as well. We suggest minimizing the geo-graphical distance between your Server CE/Pro instance and the S3 compatible object storage. In a managed environment, this would mean provisioning a bucket in the same region, and for an on-premise solution, running the two on the same campus.

S3 setup

We need four "buckets" and two restricted user accounts.

The buckets should not be publicly accessible.

Note: The history related items are needed from Server CE/Pro v3.5 onwards.

Bucket	Usage	Service	Previously in /var/lib/sharelatex/data
overleaf-user-files	project user files	filestore	user_files
overleaf-template-files	template files	filestore	template_files
overleaf-project-blobs	project history blobs	history	history/overleaf-project-blobs
overleaf-chunks	history chunks	history	history/overleaf-chunks

You may want/need to pick a different name, be sure to use the custom buckets in all the commands.

The following will use placeholders for actual credentials:

• SHARELATEX_FILESTORE_S3_ACCESS_KEY_ID This is the access key/username of the restricted user of the filestore service.
• SHARELATEX_FILESTORE_S3_SECRET_ACCESS_KEY This is the secret key/password of the restricted user of the filestore service.
• SHARELATEX_HISTORY_S3_ACCESS_KEY_ID This is the access key/username of the restricted user of the history service.
• SHARELATEX_HISTORY_S3_SECRET_ACCESS_KEY This is the secret key/password of the restricted user of the history service.

Server CE/Pro only needs a small set of permissions on each bucket:

• create object
• get object
• delete object
• list bucket

Access Policies

Here is how a policy for the filestore user could look like:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::overleaf-user-files"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::overleaf-user-files/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::overleaf-template-files"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::overleaf-template-files/*"
    }
  ]
}

Here is how a policy for the history user could look like:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::overleaf-project-blobs"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::overleaf-project-blobs/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::overleaf-chunks"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::overleaf-chunks/*"
    }
  ]
}

Overview of variables

When using AWS S3

# Enable S3 backend for filestore
SHARELATEX_FILESTORE_BACKEND=s3

# Bucket name for project files
SHARELATEX_FILESTORE_USER_FILES_BUCKET_NAME=overleaf-user-files

# Bucket name for template files
SHARELATEX_FILESTORE_TEMPLATE_FILES_BUCKET_NAME=overleaf-template-files

# Key for filestore user
SHARELATEX_FILESTORE_S3_ACCESS_KEY_ID=...

# Secret for filestore user
SHARELATEX_FILESTORE_S3_SECRET_ACCESS_KEY=...

# Bucket region you picked when creating the buckets.
SHARELATEX_FILESTORE_S3_REGION=""


# Enable S3 backend for history
SHARELATEX_HISTORY_BACKEND=s3

# Bucket name for project history blobs
SHARELATEX_HISTORY_PROJECT_BLOBS_BUCKET=overleaf-project-blobs

# Bucket name for history chunks
SHARELATEX_HISTORY_CHUNKS_BUCKET=overleaf-chunks

# Key for history user
SHARELATEX_HISTORY_S3_ACCESS_KEY_ID=...

# Secret for history user
SHARELATEX_HISTORY_S3_SECRET_ACCESS_KEY=...

# Bucket region you picked when creating the buckets.
SHARELATEX_HISTORY_S3_REGION=""

When using a self-hosted option

# Enable S3 backend for filestore
SHARELATEX_FILESTORE_BACKEND=s3

# Bucket name for project files
SHARELATEX_FILESTORE_USER_FILES_BUCKET_NAME=overleaf-user-files

# Bucket name for template files
SHARELATEX_FILESTORE_TEMPLATE_FILES_BUCKET_NAME=overleaf-template-files

# Key for filestore user
SHARELATEX_FILESTORE_S3_ACCESS_KEY_ID=...

# Secret for filestore user
SHARELATEX_FILESTORE_S3_SECRET_ACCESS_KEY=...

# S3 provider endpoint
SHARELATEX_FILESTORE_S3_ENDPOINT=http://10.10.10.10:9000

# Path style addressing of buckets. Most likely you need to set this to "true".
SHARELATEX_FILESTORE_S3_PATH_STYLE="true"

# Bucket region. Most likely you do not need to configure this.
SHARELATEX_FILESTORE_S3_REGION=""


# Enable S3 backend for history
SHARELATEX_HISTORY_BACKEND=s3

# Bucket name for project history blobs
SHARELATEX_HISTORY_PROJECT_BLOBS_BUCKET=overleaf-project-blobs

# Bucket name for history chunks
SHARELATEX_HISTORY_CHUNKS_BUCKET=overleaf-chunks

# Key for history user
SHARELATEX_HISTORY_S3_ACCESS_KEY_ID=...

# Secret for history user
SHARELATEX_HISTORY_S3_SECRET_ACCESS_KEY=...

# S3 provider endpoint
SHARELATEX_HISTORY_S3_ENDPOINT=http://10.10.10.10:9000

# Path style addressing of buckets. Most likely you need to set this to "true".
SHARELATEX_HISTORY_S3_PATH_STYLE="true"

# Bucket region. Most likely you do not need to configure this.
SHARELATEX_HISTORY_S3_REGION=""

minio setup

Note: MINIO_ROOT_USER and MINIO_ROOT_PASSWORD are the root credentials of the minio instance.

Please follow the official documentation for obtaining a copy of mc.

mc alias set s3 http://10.10.10.10:9000 MINIO_ROOT_USER MINIO_ROOT_PASSWORD

# Put the contents of the policies from the previous section in the
#  respective json file policy-filestore.json and policy-history.json.
# Reminder: Replace the bucket names and credentials accordingly.

# filestore buckets, user and policy
mc mb --ignore-existing s3/overleaf-user-files
mc mb --ignore-existing s3/overleaf-template-files
mc admin user add s3 \
  SHARELATEX_FILESTORE_S3_ACCESS_KEY_ID \
  SHARELATEX_FILESTORE_S3_SECRET_ACCESS_KEY
mc admin policy add s3 overleaf-filestore policy-filestore.json
mc admin policy set s3 overleaf-filestore \
  user=SHARELATEX_FILESTORE_S3_ACCESS_KEY_ID

# history buckets, user and policy
mc mb --ignore-existing s3/overleaf-project-blobs
mc mb --ignore-existing s3/overleaf-chunks
mc admin user add s3 \
  SHARELATEX_HISTORY_S3_ACCESS_KEY_ID \
  SHARELATEX_HISTORY_S3_SECRET_ACCESS_KEY
mc admin policy add s3 overleaf-history policy-history.json
mc admin policy set s3 overleaf-history \
  user=SHARELATEX_HISTORY_S3_ACCESS_KEY_ID