All CSS / JS Not Being Passed Client Side

[un-fuf-a-doo]

I literally over the past week have setup 3 mastodon servers. The installations are identical, same OS, VM build templates, everything. For some reason on the 3rd installation when I launch the site in any browser I see a site without CSS or JavaScript. Viewing the developers console shows that several files are coming up 404's and some are being blocked due to a mismatch in type. I can assure you the files all exist, have content, permissions are the same.

GEThttps://mstdn.hackdefendr.com/packs/css/common-6632dedd.css
[HTTP/2 404 Not Found 256ms]

GEThttps://mstdn.hackdefendr.com/packs/css/default-4fb6b0ab.chunk.css
[HTTP/2 404 Not Found 304ms]

GEThttps://mstdn.hackdefendr.com/packs/js/locale_en-b2cbaebad015671e1ba8.chunk.js
[HTTP/2 404 Not Found 296ms]

GEThttps://mstdn.hackdefendr.com/packs/js/common-3869dfcbde1ff39e38cf.js
[HTTP/2 404 Not Found 285ms]

GEThttps://mstdn.hackdefendr.com/inert.css
[HTTP/2 404 Not Found 306ms]

GEThttps://mstdn.hackdefendr.com/packs/js/public-197b4c6c11335d189f0d.chunk.js
[HTTP/2 404 Not Found 390ms]

GEThttps://mstdn.hackdefendr.com/avatars/original/missing.png
[HTTP/2 404 Not Found 263ms]

GEThttps://mstdn.hackdefendr.com/packs/media/images/preview-5df98290371ead9a70bc3cd4733bbfa7.jpg
[HTTP/2 404 Not Found 269ms]

The resource from “https://mstdn.hackdefendr.com/packs/css/common-6632dedd.css” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff).
about
The resource from “https://mstdn.hackdefendr.com/packs/js/common-3869dfcbde1ff39e38cf.js” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff).
about
None of the “sha256” hashes in the integrity attribute match the content of the subresource. about
The resource from “https://mstdn.hackdefendr.com/packs/js/locale_en-b2cbaebad015671e1ba8.chunk.js” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff).
about
None of the “sha256” hashes in the integrity attribute match the content of the subresource. about
The resource from “https://mstdn.hackdefendr.com/packs/css/default-4fb6b0ab.chunk.css” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff).
about
The resource from “https://mstdn.hackdefendr.com/inert.css” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff).
about
The resource from “https://mstdn.hackdefendr.com/packs/js/public-197b4c6c11335d189f0d.chunk.js” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff).
about
None of the “sha256” hashes in the integrity attribute match the content of the subresource. about
GEThttps://mstdn.hackdefendr.com/packs/media/images/preview-5df98290371ead9a70bc3cd4733bbfa7.jpg
[HTTP/2 404 Not Found 200ms]

GEThttps://mstdn.hackdefendr.com/avatars/original/missing.png
[HTTP/2 404 Not Found 202ms]

GEThttps://mstdn.hackdefendr.com/apple-touch-icon.png
[HTTP/2 404 Not Found 201ms]

GEThttps://mstdn.hackdefendr.com/favicon.ico
[HTTP/2 404 Not Found 208ms]

The resource from “https://mstdn.hackdefendr.com/packs/css/common-6632dedd.css” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff).
about
The resource from “https://mstdn.hackdefendr.com/inert.css” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff).
about
The resource from “https://mstdn.hackdefendr.com/packs/css/default-4fb6b0ab.chunk.css” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff).
about

This is what I see:
​

Did something get pushed recently that might have caused this?

Regards,
Jeff

[nightpool]

Navigating to https://mstdn.hackdefendr.com/packs/js/common-3869dfcbde1ff39e38cf.js in the browser shows nothing but a blank 404 page. There's nothing in Mastodon that could cause that to happen—either nginx finds that file or it doesn't. Did you maybe forget to compile assets? you need to run rake assets:precompile every time you deploy a new source version of mastodon

…

On Sun, Jan 2, 2022 at 3:34 PM Jeff ***@***.***> wrote: I literally over the past week have setup 3 mastodon servers. The installations are identical, same OS, VM build templates, everything. For some reason on the 3rd installation when I launch the site in any browser I see a site without CSS or JavaScript. Viewing the developers console shows that several files are coming up 404's and some are being blocked due to a mismatch in type. I can assure you the files all exist, have content, permissions are the same. GEThttps://mstdn.hackdefendr.com/packs/css/common-6632dedd.css [HTTP/2 404 Not Found 256ms] GEThttps://mstdn.hackdefendr.com/packs/css/default-4fb6b0ab.chunk.css [HTTP/2 404 Not Found 304ms] GEThttps://mstdn.hackdefendr.com/packs/js/locale_en-b2cbaebad015671e1ba8.chunk.js [HTTP/2 404 Not Found 296ms] GEThttps://mstdn.hackdefendr.com/packs/js/common-3869dfcbde1ff39e38cf.js [HTTP/2 404 Not Found 285ms] GEThttps://mstdn.hackdefendr.com/inert.css [HTTP/2 404 Not Found 306ms] GEThttps://mstdn.hackdefendr.com/packs/js/public-197b4c6c11335d189f0d.chunk.js [HTTP/2 404 Not Found 390ms] GEThttps://mstdn.hackdefendr.com/avatars/original/missing.png [HTTP/2 404 Not Found 263ms] GEThttps://mstdn.hackdefendr.com/packs/media/images/preview-5df98290371ead9a70bc3cd4733bbfa7.jpg [HTTP/2 404 Not Found 269ms] The resource from “https://mstdn.hackdefendr.com/packs/css/common-6632dedd.css” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). about The resource from “https://mstdn.hackdefendr.com/packs/js/common-3869dfcbde1ff39e38cf.js” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). about None of the “sha256” hashes in the integrity attribute match the content of the subresource. about The resource from “https://mstdn.hackdefendr.com/packs/js/locale_en-b2cbaebad015671e1ba8.chunk.js” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). about None of the “sha256” hashes in the integrity attribute match the content of the subresource. about The resource from “https://mstdn.hackdefendr.com/packs/css/default-4fb6b0ab.chunk.css” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). about The resource from “https://mstdn.hackdefendr.com/inert.css” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). about The resource from “https://mstdn.hackdefendr.com/packs/js/public-197b4c6c11335d189f0d.chunk.js” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). about None of the “sha256” hashes in the integrity attribute match the content of the subresource. about GEThttps://mstdn.hackdefendr.com/packs/media/images/preview-5df98290371ead9a70bc3cd4733bbfa7.jpg [HTTP/2 404 Not Found 200ms] GEThttps://mstdn.hackdefendr.com/avatars/original/missing.png [HTTP/2 404 Not Found 202ms] GEThttps://mstdn.hackdefendr.com/apple-touch-icon.png [HTTP/2 404 Not Found 201ms] GEThttps://mstdn.hackdefendr.com/favicon.ico [HTTP/2 404 Not Found 208ms] The resource from “https://mstdn.hackdefendr.com/packs/css/common-6632dedd.css” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). about The resource from “https://mstdn.hackdefendr.com/inert.css” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). about The resource from “https://mstdn.hackdefendr.com/packs/css/default-4fb6b0ab.chunk.css” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). about This is what I see: ​ [image: Screen Shot 2022-01-02 at 3 31 08 PM] <https://user-images.githubusercontent.com/67521579/147889991-9f161964-b6c5-4c69-b86c-b1f211045ea1.png> Did something get pushed recently that might have caused this? Regards, Jeff — Reply to this email directly, view it on GitHub <#17221>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABZCV3BUL4KGIUMWX7NCMTUUDAGFANCNFSM5LEC53CQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[un-fuf-a-doo]

As I said, the files are all there on disk. I followed the same instructions starting here (https://docs.joinmastodon.org/admin/prerequisites/). I'm using the same nginx config that is provided in the git repo. No step was missed, nothing skipped. When I run the setup program, each line is read to make sure the value is correct.

mastodon@mstdn:~/live$ ll public/packs/css/
total 3432
drwxrwxr-x 2 mastodon mastodon   4096 Jan  2 13:30 ./
drwxrwxr-x 6 mastodon mastodon   4096 Jan  2 13:30 ../
-rw-rw-r-- 1 mastodon mastodon  29335 Jan  2 15:15 common-6632dedd.css
-rw-rw-r-- 1 mastodon mastodon  39355 Jan  2 15:15 common-6632dedd.css.map
-rw-rw-r-- 1 mastodon mastodon 326788 Jan  2 15:15 contrast-b21f3056.chunk.css
-rw-rw-r-- 1 mastodon mastodon 741496 Jan  2 15:15 contrast-b21f3056.chunk.css.map
-rw-rw-r-- 1 mastodon mastodon 323309 Jan  2 15:15 default-4fb6b0ab.chunk.css
-rw-rw-r-- 1 mastodon mastodon 736008 Jan  2 15:15 default-4fb6b0ab.chunk.css.map
-rw-rw-r-- 1 mastodon mastodon 155377 Jan  2 15:15 .gz
-rw-rw-r-- 1 mastodon mastodon   8056 Jan  2 15:15 mailer-c0e0ad29.chunk.css
-rw-rw-r-- 1 mastodon mastodon  14750 Jan  2 15:15 mailer-c0e0ad29.chunk.css.map
-rw-rw-r-- 1 mastodon mastodon 339997 Jan  2 15:15 mastodon-light-e831b6ee.chunk.css
-rw-rw-r-- 1 mastodon mastodon 773936 Jan  2 15:15 mastodon-light-e831b6ee.chunk.css.map

I compared files from one of the other hosts I built out that is working properly.

mastodon@mst01-lax:~/live$ ll public/packs/css/
total 3348
drwxrwxr-x 2 mastodon mastodon   4096 Dec 11 14:54 ./
drwxrwxr-x 6 mastodon mastodon   4096 Dec 11 14:54 ../
-rw-rw-r-- 1 mastodon mastodon  66636 Dec 11 14:54 .gz
-rw-rw-r-- 1 mastodon mastodon  29335 Dec 11 14:54 common-6632dedd.css
-rw-rw-r-- 1 mastodon mastodon  39355 Dec 11 14:54 common-6632dedd.css.map
-rw-rw-r-- 1 mastodon mastodon 326788 Dec 11 14:54 contrast-b21f3056.chunk.css
-rw-rw-r-- 1 mastodon mastodon 741496 Dec 11 14:54 contrast-b21f3056.chunk.css.map
-rw-rw-r-- 1 mastodon mastodon 323309 Dec 11 14:54 default-4fb6b0ab.chunk.css
-rw-rw-r-- 1 mastodon mastodon 736008 Dec 11 14:54 default-4fb6b0ab.chunk.css.map
-rw-rw-r-- 1 mastodon mastodon   8056 Dec 11 14:54 mailer-c0e0ad29.chunk.css
-rw-rw-r-- 1 mastodon mastodon  14750 Dec 11 14:54 mailer-c0e0ad29.chunk.css.map
-rw-rw-r-- 1 mastodon mastodon 339997 Dec 11 14:54 mastodon-light-e831b6ee.chunk.css
-rw-rw-r-- 1 mastodon mastodon 773936 Dec 11 14:54 mastodon-light-e831b6ee.chunk.css.map

Regards,
Jeff

[vekee]

@un-fuf-a-doo

After refrenced your issue, I find the solution at below helped me
chmod +x /home/mastodon

3584

[nightpool]

Well, then there must be something wrong with your nginx configuration, because you can clearly see that it's not serving the files that are there: https://mstdn.hackdefendr.com/packs/css/common-6632dedd.css

…

On Sun, Jan 2, 2022 at 4:13 PM Jeff ***@***.***> wrote: As I said, the files are all there on disk. I followed the same instructions starting here ( https://docs.joinmastodon.org/admin/prerequisites/). I'm using the same nginx config that is provided in the git repo. No step was missed, nothing skipped. When I run the setup program, each line is read to make sure the value is correct. ***@***.***:~/live$ ll public/packs/css/ total 3432 drwxrwxr-x 2 mastodon mastodon 4096 Jan 2 13:30 ./ drwxrwxr-x 6 mastodon mastodon 4096 Jan 2 13:30 ../ -rw-rw-r-- 1 mastodon mastodon 29335 Jan 2 15:15 common-6632dedd.css -rw-rw-r-- 1 mastodon mastodon 39355 Jan 2 15:15 common-6632dedd.css.map -rw-rw-r-- 1 mastodon mastodon 326788 Jan 2 15:15 contrast-b21f3056.chunk.css -rw-rw-r-- 1 mastodon mastodon 741496 Jan 2 15:15 contrast-b21f3056.chunk.css.map -rw-rw-r-- 1 mastodon mastodon 323309 Jan 2 15:15 default-4fb6b0ab.chunk.css -rw-rw-r-- 1 mastodon mastodon 736008 Jan 2 15:15 default-4fb6b0ab.chunk.css.map -rw-rw-r-- 1 mastodon mastodon 155377 Jan 2 15:15 .gz -rw-rw-r-- 1 mastodon mastodon 8056 Jan 2 15:15 mailer-c0e0ad29.chunk.css -rw-rw-r-- 1 mastodon mastodon 14750 Jan 2 15:15 mailer-c0e0ad29.chunk.css.map -rw-rw-r-- 1 mastodon mastodon 339997 Jan 2 15:15 mastodon-light-e831b6ee.chunk.css -rw-rw-r-- 1 mastodon mastodon 773936 Jan 2 15:15 mastodon-light-e831b6ee.chunk.css.map I compared files from one of the other hosts I built out that is working properly. ***@***.***:~/live$ ll public/packs/css/ total 3348 drwxrwxr-x 2 mastodon mastodon 4096 Dec 11 14:54 ./ drwxrwxr-x 6 mastodon mastodon 4096 Dec 11 14:54 ../ -rw-rw-r-- 1 mastodon mastodon 66636 Dec 11 14:54 .gz -rw-rw-r-- 1 mastodon mastodon 29335 Dec 11 14:54 common-6632dedd.css -rw-rw-r-- 1 mastodon mastodon 39355 Dec 11 14:54 common-6632dedd.css.map -rw-rw-r-- 1 mastodon mastodon 326788 Dec 11 14:54 contrast-b21f3056.chunk.css -rw-rw-r-- 1 mastodon mastodon 741496 Dec 11 14:54 contrast-b21f3056.chunk.css.map -rw-rw-r-- 1 mastodon mastodon 323309 Dec 11 14:54 default-4fb6b0ab.chunk.css -rw-rw-r-- 1 mastodon mastodon 736008 Dec 11 14:54 default-4fb6b0ab.chunk.css.map -rw-rw-r-- 1 mastodon mastodon 8056 Dec 11 14:54 mailer-c0e0ad29.chunk.css -rw-rw-r-- 1 mastodon mastodon 14750 Dec 11 14:54 mailer-c0e0ad29.chunk.css.map -rw-rw-r-- 1 mastodon mastodon 339997 Dec 11 14:54 mastodon-light-e831b6ee.chunk.css -rw-rw-r-- 1 mastodon mastodon 773936 Dec 11 14:54 mastodon-light-e831b6ee.chunk.css.map Regards, Jeff — Reply to this email directly, view it on GitHub <#17221 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABZCVYSDEG634CMOC2W6XTUUDEZ7ANCNFSM5LEC53CQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[un-fuf-a-doo]

Well, then there is something wrong with the nginx config supplied in the git repository. Because that is the config I am using, with the only lines edited were the server name and the ssl certificate/key.

Here is the nginx config:

map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

upstream backend {
    server 127.0.0.1:3000 fail_timeout=0;
}

upstream streaming {
    server 127.0.0.1:4000 fail_timeout=0;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=CACHE:10m inactive=7d max_size=1g;

server {
  listen 80;
  listen [::]:80;
  server_name mstdn.hackdefendr.com;
  root /home/mastodon/live/public;
  location /.well-known/acme-challenge/ { allow all; }
  location / { return 301 https://$host$request_uri; }
}

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name mstdn.hackdefendr.com;

  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;

  ssl_certificate     /etc/nginx/ssl/hackdefendr.com.pem;
  ssl_certificate_key /etc/nginx/ssl/hackdefendr.com.key;

  keepalive_timeout    70;
  sendfile             on;
  client_max_body_size 80m;

  root /home/mastodon/live/public;

  gzip on;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_proxied any;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

  add_header Strict-Transport-Security "max-age=31536000";

  location / {
    try_files $uri @proxy;
  }

  location ~ ^/(emoji|packs|system/accounts/avatars|system/media_attachments/files) {
    add_header Cache-Control "public, max-age=31536000, immutable";
    add_header Strict-Transport-Security "max-age=31536000";
    try_files $uri @proxy;
  }

  location /sw.js {
    add_header Cache-Control "public, max-age=0";
    add_header Strict-Transport-Security "max-age=31536000";
    try_files $uri @proxy;
  }

  location @proxy {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Proxy "";
    proxy_pass_header Server;

    proxy_pass http://backend;
    proxy_buffering on;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    proxy_cache CACHE;
    proxy_cache_valid 200 7d;
    proxy_cache_valid 410 24h;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    add_header X-Cached $upstream_cache_status;
    add_header Strict-Transport-Security "max-age=31536000";

    tcp_nodelay on;
  }

  location /api/v1/streaming {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Proxy "";

    proxy_pass http://streaming;
    proxy_buffering off;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    tcp_nodelay on;
  }

  error_page 500 501 502 503 504 /500.html;
}

Do you see any issues with it?

[ClearlyClaire]

It seems Rails itself is called on those files, which should not happen.
Are you sure whatever nginx is running can actually access the files? Check /home/mastodon and /home/mastodon/live's permissions.

[Gameink]

Hey I have the exact same problem even after using chmod o+x... do you have any other Idea what could cause this problem?
Thank you in advance!

[ClearlyClaire]

What do you mean by “exact same problem”? Merely CSS and javascript not being loaded? Or Rails getting the requests? What do nginx logs show?

[Gameink]

Yes the CSS and javascript did not load rightly. Got an combination of 404 and 502 Proxy error. But I used Apache2. Which permissions do I have to give to the mastodon dir?

[ClearlyClaire]

Mastodon does not provide an Apache2 configuration file, and we can't help you without seeing yours.

[nightpool]

@Gameink please open a new thread, your issue (with Apache) has nothing to do with the original issue (using Nginx)

[mickenordin]

I had this exact issue with css and js not being loaded, when trying to upgrade to 4.1.1. I am running nginx in front of the official docker container. Do I need to change any nginx configuration?

[DejavuMoe]

I had this exact issue with css and js not being loaded, when trying to upgrade to 4.1.1. I am running nginx in front of the official docker container. Do I need to change any nginx configuration?

Same to me🤕

[H3arn]

I came across the same problem and the following steps worked for me. You can try that too.

Check this

mastodon/dist/nginx.conf

Lines 59 to 60 in f3fca78

	# If Docker is used for deployment and Rails serves static files,
	# then needed must replace line `try_files $uri =404;` with `try_files $uri @proxy;`.

and add this environmental variable to your docker config as described here

Check the value of RAILS_SERVE_STATIC_FILES, it has previously been interpreted as truthy if it was set to any non-empty value, while it is now only interpreted as true if it's literally set to true.

If you are using Docker, it is likely that your setup does not have nginx serving the files directly, but puma doing so.

[mickenordin]

For me the issue seems to be that puma replies with 403 for all paths. This seems to be unrelated to nginx and just applies to all paths that puma should serve on port 3000, as I get the same result when using curl directly against the web instance on port 3000. I am not sure why mastodon image for version 4.1.1. started behaving like this, but I do not think this is an issue with permissions, as changing permissions to folders changes nothing of pumas behaviour. the mastodon user owns all files in /opt/mastodon and the mastodon user is running puma. Is there a new version of puma in this image that requires config changes or migrationsteps to be run?

[ClearlyClaire]

Check the value of RAILS_SERVE_STATIC_FILES, it has previously been interpreted as truthy if it was set to any non-empty value, while it is now only interpreted as true if it's literally set to true.

If you are using Docker, it is likely that your setup does not have nginx serving the files directly, but puma doing so.

[mickenordin]

That was definately the problem, it was set to false. Setting the environment variable to true solves the issue for me.

[CrimsonEdgeHope]

I was trying out mastodon v4.2.7, have had it deployed by following instructions from https://docs.joinmastodon.org/admin/install/ ,literally, and encountered the exact issue. I make sure no Permission Denied error occurring after chmodding directories, and inspecting Nginx error.log, however some defects remained. According to #17221 (reply in thread) and #17221 (comment) I set RAILS_SERVE_STATIC_FILES to true in .env.production, but incorrect mime/type value was still being given, cause of css and js not being properly loaded.

I took at look at these comments:

# If Docker is used for deployment and Rails serves static files,
# then needed must replace line `try_files $uri =404;` with `try_files $uri @proxy;`.

I was NOT using docker for my Mastodon deployment at all, including ruby, postgresql, redis, sidekiq. My nginx has access to filesystem with 100% certainty. The comment:

# If Docker is used for deployment and Rails serves static files,

is somehow misleading, I think, or I just simply got it wrong. I replaced all the try_files $uri =404; appearances with try_files $uri @proxy; and problem solved. BOOM.

In addition, with the puma backend listening to localhost:3000, I saw 403 Forbidden curling against it, no additional param, even at root path: