Listed projects are "not authorized": Opening any project redirects to login page, after upgrade

[dvmc]

I tried to upgrade Overleaf via the toolkit from 3.0.1 to the current version 4.0.4. That did not work because of a compatibility error message in MongoDB (which has since been lost), so I have tried to downgrade back to 3.0.1 and then up to 3.1.0. Then there was no compatibility error message any more, but when I log in and click on a project, the loading animation appears, and then I am redirected to the login page with a redirection query (e.g. /login?redir=%2Fproject%2Fabcdef1234567890abcdef12). However, opening the Overleaf home URL, I am still logged in.

I have already tried to roll back to the last working version by (1) changing config/version back to 3.0.1 and (2) restoring the data folder, but to no avail. What else could I try and which logs should I look into? I would like to avoid any data loss.

Steps to Reproduce

1. Have an Overleaf version 3.0.1 running that was installed via the toolkit (ideally, have at least one project for your user).
2. Make a backup of the data directory
3. Upgrade to the latest version (4.0.4) by running bin/upgrade
4. Try to run with bin/up -d, container will repeatedly throw an error and restart
5. Downgrade back to 3.0.1 by adjusting config/version
6. Restore the data folder, run bin/up -d.
7. Try to log in and select any project. You will be redirected to a login page after a while, although you are still logged in.

Expected Behaviour

The project should have been opened after trying to open it.

Observed Behaviour

The loading animation starts and eventually I am being redirected to the login page (e.g. /login?redir=%2Fproject%2Fabcdef1234567890abcdef12), although I am logged in.

Context

Technical Info

• URL: private
• Browser Name and version: Firefox 115.0.2
• Operating System and version (desktop or mobile): Docker
• Signed in as: Admin
• Project and/or file:

Analysis

The only meaningful message I could find on client side was "not authorized" from socket.io. On the server side, only the component "real-time" had a few insights. The following is the log from "real-time" after I clicked on a project that is mine and was listed on the project page:

{"name":"real-time","hostname":"687f382947b4","pid":156,"level":30,"session":{"cookie":{"originalMaxAge":432000000,"expires":"2023-07-23T21:41:51.667Z","secure":true,"httpOnly":true,"path":"/","sameSite":"lax"},"validationToken":"v1:b_Dm","justLoggedIn":true,"passport":{"user":{"_id":"5f8706265c09bc00853932f8","first_name":"david.michaelis","last_name":"","isAdmin":true,"staffAccess":{"publisherMetrics":false,"publisherManagement":false,"institutionMetrics":false,"institutionManagement":false,"groupMetrics":false,"groupManagement":false,"adminMetrics":false},"email":"david.michaelis@example.com","referal_id":"df33716b","session_created":"2023-07-18T21:41:14.382Z","ip_address":"172.29.248.85","must_reconfirm":false}},"csrfSecret":"ML0pIHSoKZhviOgcxOLgghBC","postLoginRedirect":"/project/5f918531bc86ea00729c7c61"},"client_id":"c-i2-t9xaudZ6Bt0jB97","msg":"client connected","time":"2023-07-18T22:27:03.632Z","v":0}
{"name":"real-time","hostname":"687f382947b4","pid":156,"level":30,"user_id":"5f8706265c09bc00853932f8","project_id":"604a4269fb055e007a543f58","client_id":"c-i2-t9xaudZ6Bt0jB97","msg":"user joining project","time":"2023-07-18T22:27:03.847Z","v":0}
{"name":"real-time","hostname":"687f382947b4","pid":156,"level":30,"project_id":"604a4269fb055e007a543f58","user_id":"5f8706265c09bc00853932f8","msg":"sending join project request to web","time":"2023-07-18T22:27:03.848Z","v":0}
{"name":"real-time","hostname":"687f382947b4","pid":156,"level":40,"project_id":"604a4269fb055e007a543f58","user_id":"5f8706265c09bc00853932f8","client_id":"c-i2-t9xaudZ6Bt0jB97","err":{"message":"not authorized","name":"NotAuthorizedError","stack":"NotAuthorizedError: not authorized\n    at Request._callback (/var/www/sharelatex/real-time/app/js/WebApiManager.js:60:20)\n    at Request.self.callback (/var/www/sharelatex/real-time/node_modules/request/request.js:185:22)\n    at Request.emit (events.js:314:20)\n    at Request.<anonymous> (/var/www/sharelatex/real-time/node_modules/request/request.js:1154:10)\n    at Request.emit (events.js:314:20)\n    at IncomingMessage.<anonymous> (/var/www/sharelatex/real-time/node_modules/request/request.js:1076:12)\n    at Object.onceWrapper (events.js:420:28)\n    at IncomingMessage.emit (events.js:326:22)\n    at endReadableNT (_stream_readable.js:1241:12)\n    at processTicksAndRejections (internal/process/task_queues.js:84:21)","info":{}},"method":"joinProject","msg":"not authorized","time":"2023-07-18T22:27:03.890Z","v":0}

[dvmc]

At this point, I'm just trying to get it running as it used to. This is what I have tried:

• Backed up the data folder from when everything was working and the Overleaf (v 3.0.1) container was stopped
• Cloned the toolkit and checked out the version that I had for 3.0.1 (af5a859)
• Ran bin/init
• Adjusted the configuration
• Ran bin/up -d and bin/stop
• Restored the data folder
• Ran bin/up -d

And the same problem from above persists ("not authorized"). How is that possible? I have the same toolkit version and Overleaf version, so restoring the data folder should have worked, shouldn't it?

These are a few differences that I found during this procedure:

• In variables.env, ENABLED_LINKED_FILE_TYPES was after the checkout for version 3.0.1 project_file,project_output_file, but used to be url,project_file in my original instance (maybe because I updated from a 2.x.x to 3.0.1? that update went smoothly). I haven't found any documentation about what this setting means, but I have tried any combination of url, project_file and project_output_file, always with the same outcome.
• In overleaf.rc, I used to have SHARELATEX_PORT=<IP>:<PORT>. Now it is split into SHARELATEX_LISTEN_IP and SHARELATEX_PORT. Both seems to have the same effect.
• Every time I run bin/up, the message "Building with native build." appears. I haven't seen it when everything was working and I have not updated the docker runtime.

I have also tried to dig into the source code to see where the authentication fails. According to the stack trace from above (real-time logs), this fails at /var/www/sharelatex/real-time/app/js/WebApiManager.js, which in turn connects to the URL /project/${project_id}/join. In fact, when I try to cURL there with a POST and my credentials (Basic HTTP Auth), I get "unauthorized", even when it was a project_id that I have created. I couldn't get any further, because I couldn't find a routing directive anywhere for the /join path. Could someone give me a pointer about how authentication on that endpoint works, so I can debug this further?

[dvmc]

Problem found: A proxy was configured in the environment variables of the container, and Overleaf was using that to communicate with the docstore host. I removed the proxy from ~/.docker/config.json, then re-created the images, and now it's working again.

It is unclear to me whether the internal docker network communication is ever supposed to go through a configured proxy or not, so I am leaving this open.