Overleaf toolkit nginx returns 502 bad gateway after OS update

[sc01194]

Steps to Reproduce

1. Instance of overleaf toolkit community edition running properly
2. OS update on the vm leading to a docker update from 25.0. to 26.0.0
3. Then the url returns 502 Bad Gateway

Expected Behaviour

URL answer with logging page

Observed Behaviour

I tried the following:

1. bin/stop then bin/up -d
2. bin/stop then docker container prune then bin/up -d
3. reboot server
   no improvements

Context

As I am not a container specialist I usually stick to the doc given for an application. If you have steps to inspects nginx container to find more relevant logs let me know.

doctor information

[root@tlsvsl0005:/opt/overleaf-toolkit]$ bin/doctor 
====== Overleaf Doctor ======
- Host Information
    - Linux
    - lsb_release not found !
- Dependencies
    - bash
        - status: present
        - version info: 5.1.8(1)-release
    - docker
        - status: present
        - version info: Docker version 26.0.0, build 2ae903e
    - realpath
        - status: present
        - version info: realpath (GNU coreutils) 8.32
    - perl
        - status: present
        - version info: 5.032001
    - awk
        - status: present
        - version info: GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0-p9, GNU MP 6.2.0)
    - openssl
        - status: present
        - version info: OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022)
    - docker compose
        - status: present
        - version info: Docker Compose version v2.25.0
- Docker Daemon
    - status: up
====== Configuration ======
- config/version
    - status: present
    - version: 4.2.3
- config/overleaf.rc
    - status: present
    - values
        - SHARELATEX_DATA_PATH: data/sharelatex
        - SERVER_PRO: false
        - SHARELATEX_LISTEN_IP: 127.0.0.1
        - SHARELATEX_PORT: 80
        - MONGO_ENABLED: true
        - MONGO_IMAGE: mongo:4.4
        - MONGO_DATA_PATH: data/mongo
        - REDIS_ENABLED: true
        - REDIS_IMAGE: redis:6.2
        - REDIS_DATA_PATH: data/redis
        - NGINX_ENABLED: true
        - NGINX_CONFIG_PATH: config/nginx/nginx.conf
        - TLS_PRIVATE_KEY_PATH: config/nginx/certs/overleaf.intra.waltr.fr.key
        - TLS_CERTIFICATE_PATH: config/nginx/certs/overleaf.intra.waltr.fr.crt
        - NGINX_HTTP_PORT: 80
        - NGINX_TLS_LISTEN_IP: 10.178.127.16
        - TLS_PORT: 443
        - GIT_BRIDGE_ENABLED: false
- config/variables.env
    - status: present
    - values
/opt/overleaf-toolkit/config/variables.env: line 24: Overleaf: command not found

Technical Info

I use a sharelatex container modified as recommended in the docs to have texlive extensions

[root@tlsvsl0005:/opt/overleaf-toolkit]$ docker ps
CONTAINER ID   IMAGE                                     COMMAND                  CREATED          STATUS                    PORTS                                              NAMES
983a4ee0aeab   nginx:1.19-alpine                         "/docker-entrypoint.…"   51 minutes ago   Up 23 minutes             127.0.1.1:80->80/tcp, 10.178.127.16:443->443/tcp   nginx
63e4479d1578   sharelatex/sharelatex:with-texlive-full   "/sbin/my_init"          51 minutes ago   Up 23 minutes             127.0.0.1:80->80/tcp                               sharelatex
cf548872527e   redis:6.2                                 "docker-entrypoint.s…"   51 minutes ago   Up 23 minutes             6379/tcp                                           redis
5417a6c15694   mongo:4.4                                 "docker-entrypoint.s…"   51 minutes ago   Up 23 minutes (healthy)   27017/tcp                                          mongo

Analysis

I can only see the docker update as a modification of the running environment. Nothing else was changed that I know of.

Since this update when doing bin/up -d I have a warning not present before but as this is just a warning this might not be relevant:

[root@tlsvsl0005:/opt/overleaf-toolkit]$ bin/up -d
Initiating Mongo replica set...
WARN[0000] /opt/overleaf-toolkit/lib/docker-compose.base.yml: `version` is obsolete 
WARN[0000] /opt/overleaf-toolkit/lib/docker-compose.redis.yml: `version` is obsolete 
WARN[0000] /opt/overleaf-toolkit/lib/docker-compose.mongo.yml: `version` is obsolete 
WARN[0000] /opt/overleaf-toolkit/lib/docker-compose.nginx.yml: `version` is obsolete 
WARN[0000] /opt/overleaf-toolkit/config/docker-compose.override.yml: `version` is obsolete 
[+] Running 1/0
 ✔ Container mongo  Running                                                                                                                         0.0s 
WARN[0000] /opt/overleaf-toolkit/lib/docker-compose.base.yml: `version` is obsolete 
WARN[0000] /opt/overleaf-toolkit/lib/docker-compose.redis.yml: `version` is obsolete 
WARN[0000] /opt/overleaf-toolkit/lib/docker-compose.mongo.yml: `version` is obsolete 
WARN[0000] /opt/overleaf-toolkit/lib/docker-compose.nginx.yml: `version` is obsolete 
WARN[0000] /opt/overleaf-toolkit/config/docker-compose.override.yml: `version` is obsolete 
[+] Running 4/4
 ✔ Container redis       Running                                                                                                                    0.0s 
 ✔ Container mongo       Healthy                                                                                                                    0.0s 
 ✔ Container sharelatex  Running                                                                                                                    0.0s 
 ✔ Container nginx       Running  

[sc01194]

I did a roll back to docker 25.0.4 and the web site is up again.

[ThexXTURBOXx]

Most likely related to #229

[bshor]

I'm getting the same "Bad Gateway" error. Here is my bin/doctor output. I have only changed SHARELATEX_LISTEN_IP to 0.0.0.0 and SHARELATEX_PORT to 8099.

====== Overleaf Doctor ======
- Host Information
    - Linux
    - Output of 'lsb_release -a':
            Distributor ID:	Debian
            Description:	Debian GNU/Linux 12 (bookworm)
            Release:	12
            Codename:	bookworm
- Dependencies
    - bash
        - status: present
        - version info: 5.2.15(1)-release
    - docker
        - status: present
        - version info: Docker version 26.0.0, build 2ae903e
    - realpath
        - status: present
        - version info: realpath (GNU coreutils) 9.1
    - perl
        - status: present
        - version info: 5.036000
    - awk
        - status: present
        - version info: mawk 1.3.4 20200120
    - openssl
        - status: present
        - version info: OpenSSL 3.0.11 19 Sep 2023 (Library: OpenSSL 3.0.11 19 Sep 2023)
    - docker compose
        - status: present
        - version info: Docker Compose version v2.22.0
- Docker Daemon
    - status: up
====== Configuration ======
- config/version
    - status: present
    - version: 4.2.3
- config/overleaf.rc
    - status: present
    - values
        - SHARELATEX_DATA_PATH: data/sharelatex
        - SERVER_PRO: false
        - SHARELATEX_LISTEN_IP: 0.0.0.0
        - SHARELATEX_PORT: 8099
        - MONGO_ENABLED: true
        - MONGO_IMAGE: mongo:5.0
        - MONGO_DATA_PATH: data/mongo
        - REDIS_ENABLED: true
        - REDIS_IMAGE: redis:6.2
        - REDIS_DATA_PATH: data/redis
        - NGINX_ENABLED: false
        - NGINX_CONFIG_PATH: config/nginx/nginx.conf
        - TLS_PRIVATE_KEY_PATH: config/nginx/certs/overleaf_key.pem
        - TLS_CERTIFICATE_PATH: config/nginx/certs/overleaf_certificate.pem
        - NGINX_HTTP_LISTEN_IP: 127.0.1.1
        - NGINX_HTTP_PORT: 80
        - NGINX_TLS_LISTEN_IP: 127.0.1.1
        - TLS_PORT: 443
        - GIT_BRIDGE_ENABLED: false
- config/variables.env
    - status: present
    - values
        - SHARELATEX_FILESTORE_BACKEND: fs
        - SHARELATEX_HISTORY_BACKEND: fs
====== Warnings ======
- None, all good
====== End ======

I'm not sure either how to roll docker back to 25, or how to disable IPv6 as in the comment by @ThexXTURBOXx.

[sc01194]

@bshor
Migth be out of overleaf scope but to roll back docker version on a redhat based OS:
dnf downgrade docker-ce-3:25.0.4-1.el9.x86_64 docker-ce-cli-1:25.0.4-1.el9.x86_64 docker-ce-rootless-extras-25.0.4-1.el9.x86_64 docker-compose-plugin-2.24.7-1.el9.x86_64 docker-buildx-plugin-0.13.0-1.el9.x86_64 containerd.io-1.6.28-3.1.el9.x86_64
Adjust package version depending on your OS, following commands might help:
dnf list --showduplicate docker-ce* dnf history

[bshor]

Thanks!

I'm running Debian.

Worried that downgrading has other consequences I can't foresee -- and isn't it strange that this is the only server app that the latest public Docker doesn't work for?

[drescherflo]

Had the same issue and fixed it by disabling IPv6 for the sharelatex container like in my answer here.

Just add the following contents to config/docker-compose.override.yml:

---
version: '2.2'
services:
  sharelatex:
    # ...
    sysctls:
      - net.ipv6.conf.all.disable_ipv6=1

[bshor]

Thanks! But my config doesn't have docker-compose.override.yml, only overleaf.rc.

[drescherflo]

Thanks! But my config doesn't have docker-compose.override.yml, only overleaf.rc.

In this case you just create one with the contents above.
After the initial setup of overleaf you probably want to install TeX Live packages. Depending on how you do it, you'll need to create one anyway.

[bshor]

I think there might be some version differences -- in /lib I saw a bunch of docker-compose yml files.

But before I started making these edits, I tried to upgrade Overleaf -- and there's a new 5.0 version available! I went through the upgrade process. And in the new version, Overleaf runs fine without any config changes!

[Egolas]

I think there might be some version differences -- in /lib I saw a bunch of docker-compose yml files.

But before I started making these edits, I tried to upgrade Overleaf -- and there's a new 5.0 version available! I went through the upgrade process. And in the new version, Overleaf runs fine without any config changes!

Can you share your upgrade details? I'm facing the same issue, and wondering how to follow your experience.

[bshor]

I think I just used the upgrade script in /bin.

[das7pad]

The latest releases 4.2.4 and 5.0.3 include changes for switching all internal traffic to IPv4 on the application level.

You can pick up 4.2.4 by manually editing config/version and running bin/up afterwards.
You can pick up 5.0.3 by running bin/upgrade.

We recommend removing any manual config/docker-compose.override.yml changes for disabling IPv6 after upgrading.