Failed to load resource: the server responded with a status of 404

[hexsix]

I've just migrating to a new machine.

Actually, I switched from nginx-proxy-manager to native nginx, and now I don't know how to configure it.

I'm using mastodon v4.1.15 with docker and s3.

DevTools Log

  /packs/js/features/compose-52fb55ff58c7b28418de.chunk.js:1 
        
        
       GET https://mastodon.instance.tld/packs/js/features/compose-52fb55ff58c7b28418de.chunk.js net::ERR_ABORTED 404 (Not Found)
/packs/js/features/home_timeline-2c26368fff116a932b7e.chunk.js:1 
        
        
       GET https://mastodon.instance.tld/packs/js/features/home_timeline-2c26368fff116a932b7e.chunk.js net::ERR_ABORTED 404 (Not Found)
/packs/js/features/notifications-eb3251f99273468a3871.chunk.js:1 
        
        
       GET https://mastodon.instance.tld/packs/js/features/notifications-eb3251f99273468a3871.chunk.js net::ERR_ABORTED 404 (Not Found)
mastodon.instance.tld/:1 Refused to apply style from 'https://mastodon.instance.tld/packs/css/common-a844dc34.css' because its MIME type ('text/html') is not a supported stylesheet MIME type, and strict MIME checking is enabled.
mastodon.instance.tld/:52 
        
        
       GET https://mastodon.instance.tld/packs/js/application-a332f173c40993abfc23.chunk.js net::ERR_ABORTED 404 (Not Found)
mastodon.instance.tld/:29 
        
        
       GET https://mastodon.instance.tld/packs/js/common-0481525dd8e129bb4a67.js net::ERR_ABORTED 404 (Not Found)
mastodon.instance.tld/:1 Refused to execute script from 'https://mastodon.instance.tld/packs/js/common-0481525dd8e129bb4a67.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.
mastodon.instance.tld/:1 Refused to apply style from 'https://mastodon.instance.tld/packs/css/dark-pink-a9777bfe.chunk.css' because its MIME type ('text/html') is not a supported stylesheet MIME type, and strict MIME checking is enabled.
mastodon.instance.tld/:30 
        
        
       GET https://mastodon.instance.tld/packs/js/locale_zh-CN-bf8fe85aeb8ceb0190a7.chunk.js net::ERR_ABORTED 404 (Not Found)
mastodon.instance.tld/:1 Refused to execute script from 'https://mastodon.instance.tld/packs/js/locale_zh-CN-bf8fe85aeb8ceb0190a7.chunk.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.
mastodon.instance.tld/:1 Refused to execute script from 'https://mastodon.instance.tld/packs/js/application-a332f173c40993abfc23.chunk.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.
android-chrome-144x144-ff3110f7772743bdd0c1c47fb7b2d4e0.png:1 
        
        
       GET https://mastodon.instance.tld/packs/media/icons/android-chrome-144x144-ff3110f7772743bdd0c1c47fb7b2d4e0.png 404 (Not Found)
mastodon.instance.tld/:1 Error while trying to use the following icon from the Manifest: https://mastodon.instance.tld/packs/media/icons/android-chrome-144x144-ff3110f7772743bdd0c1c47fb7b2d4e0.png (Download error or resource isn't a valid image)
mastodon.instance.tld/:1 The resource https://mastodon.instance.tld/packs/js/features/home_timeline-2c26368fff116a932b7e.chunk.js was preloaded using link preload but not used within a few seconds from the window's load event. Please make sure it has an appropriate `as` value and it is preloaded intentionally.
mastodon.instance.tld/:1 The resource https://mastodon.instance.tld/packs/js/features/compose-52fb55ff58c7b28418de.chunk.js was preloaded using link preload but not used within a few seconds from the window's load event. Please make sure it has an appropriate `as` value and it is preloaded intentionally.
mastodon.instance.tld/:1 The resource https://mastodon.instance.tld/packs/js/features/notifications-eb3251f99273468a3871.chunk.js was preloaded using link preload but not used within a few seconds from the window's load event. Please make sure it has an appropriate `as` value and it is preloaded intentionally.

Here's my new nginx.conf

mastodon.conf

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
    }

    upstream backend {
        server 127.0.0.1:3000 fail_timeout=0;
    }

    upstream streaming {
        # Instruct nginx to send connections to the server with the least number of connections
        # to ensure load is distributed evenly.
        least_conn;

        server 127.0.0.1:4000 fail_timeout=0;
        # Uncomment these lines for load-balancing multiple instances of streaming for scaling,
        # this assumes your running the streaming server on ports 4000, 4001, and 4002:
        # server 127.0.0.1:4001 fail_timeout=0;
        # server 127.0.0.1:4002 fail_timeout=0;
    }

    proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=CACHE:10m inactive=7d max_size=1g;

    server {
    listen 80;
    listen [::]:80;
    server_name mastodon.instance.tld;
    root /home/mastodon/live/public;
    location /.well-known/acme-challenge/ { allow all; }
    location / { return 301 https://$host$request_uri; }
    }

    server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name mastodon.instance.tld;

    ssl_protocols TLSv1.2 TLSv1.3;

    # You can use https://ssl-config.mozilla.org/ to generate your cipher set.
    # We recommend their "Intermediate" level.
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305;

    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;

    # Uncomment these lines once you acquire a certificate:
    ssl_certificate     /etc/letsencrypt/live/mastodon.instance.tld/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mastodon.instance.tld/privkey.pem;

    keepalive_timeout    70;
    sendfile             on;
    client_max_body_size 99m;

    root /home/mastodon/live/public;

    gzip on;
    gzip_disable "msie6";
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_buffers 16 8k;
    gzip_http_version 1.1;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript image/svg+xml image/x-icon;

    location / {
        try_files $uri @proxy;
    }

    # If Docker is used for deployment and Rails serves static files,
    # then needed must replace line `try_files $uri =404;` with `try_files $uri @proxy;`.
    location = /sw.js {
        add_header Cache-Control "public, max-age=604800, must-revalidate";
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
        try_files $uri @proxy;
    }

    location ~ ^/assets/ {
        add_header Cache-Control "public, max-age=2419200, must-revalidate";
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
        try_files $uri @proxy;
    }

    location ~ ^/avatars/ {
        add_header Cache-Control "public, max-age=2419200, must-revalidate";
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
        try_files $uri @proxy;
    }

    location ~ ^/emoji/ {
        add_header Cache-Control "public, max-age=2419200, must-revalidate";
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
        try_files $uri @proxy;
    }

    location ~ ^/headers/ {
        add_header Cache-Control "public, max-age=2419200, must-revalidate";
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
        try_files $uri @proxy;
    }

    location ~ ^/packs/ {
        add_header Cache-Control "public, max-age=2419200, must-revalidate";
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
        try_files $uri @proxy;
    }

    location ~ ^/shortcuts/ {
        add_header Cache-Control "public, max-age=2419200, must-revalidate";
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
        try_files $uri @proxy;
    }

    location ~ ^/sounds/ {
        add_header Cache-Control "public, max-age=2419200, must-revalidate";
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
        try_files $uri @proxy;
    }

    location ~ ^/system/ {
        add_header Cache-Control "public, max-age=2419200, immutable";
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
        add_header X-Content-Type-Options nosniff;
        add_header Content-Security-Policy "default-src 'none'; form-action 'none'";
        try_files $uri @proxy;
    }

    location ^~ /api/v1/streaming {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Proxy "";

        proxy_pass http://streaming;
        proxy_buffering off;
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";

        tcp_nodelay on;
    }

    location @proxy {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Proxy "";
        proxy_pass_header Server;

        proxy_pass http://backend;
        proxy_buffering on;
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        proxy_cache CACHE;
        proxy_cache_valid 200 7d;
        proxy_cache_valid 410 24h;
        proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
        add_header X-Cached $upstream_cache_status;

        tcp_nodelay on;
    }

    error_page 404 500 501 502 503 504 /500.html;
    }

media.conf

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name media.instance.tld;
  root /var/www/html;

  ssl_certificate     /etc/letsencrypt/live/media.instance.tld/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/media.instance.tld/privkey.pem;

  keepalive_timeout 30;

  location = / {
    index index.html;
  }

  location / {
    try_files $uri @s3;
  }

  set $s3_backend 'https://media-instance-tld.s3.us-west-1.amazonaws.com';

  location @s3 {
    limit_except GET {
      deny all;
    }

    resolver 8.8.8.8;
    proxy_set_header Host media-instance-tld.s3.us-west-1.amazonaws.com;
    proxy_set_header Connection '';
    proxy_set_header Authorization '';
    proxy_hide_header Set-Cookie;
    proxy_hide_header 'Access-Control-Allow-Origin';
    proxy_hide_header 'Access-Control-Allow-Methods';
    proxy_hide_header 'Access-Control-Allow-Headers';
    proxy_hide_header x-amz-id-2;
    proxy_hide_header x-amz-request-id;
    proxy_hide_header x-amz-meta-server-side-encryption;
    proxy_hide_header x-amz-server-side-encryption;
    proxy_hide_header x-amz-bucket-region;
    proxy_hide_header x-amzn-requestid;
    proxy_ignore_headers Set-Cookie;
    proxy_pass $s3_backend$uri;
    proxy_intercept_errors off;

    proxy_cache CACHE;
    proxy_cache_valid 200 48h;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_lock on;

    expires 1y;
    add_header Cache-Control public;
    add_header 'Access-Control-Allow-Origin' '*';
    add_header X-Cache-Status $upstream_cache_status;
    add_header X-Content-Type-Options nosniff;
    add_header Content-Security-Policy "default-src 'none'; form-action 'none'";
  }
}

And my .env.production

.env.production

# Generated with mastodon:setup on 2022-04-03 13:46:38 UTC

# Some variables in this file will be interpreted differently whether you are
# using docker-compose or not.

DEFAULT_LOCALE=zh-CN
GITHUB_REPOSITORY=hexsix/mastodon
LOCAL_DOMAIN=mastodon.instance.tld
SINGLE_USER_MODE=false
SECRET_KEY_BASE=xxx
OTP_SECRET=xxx
VAPID_PRIVATE_KEY=xxx
VAPID_PUBLIC_KEY=xxx

DB_HOST=mastodon_db_1
DB_PORT=5432
DB_NAME=mastodon_production
DB_USER=mastodon
DB_PASS=xxx

REDIS_HOST=mastodon_redis_1
REDIS_PORT=6379
REDIS_PASSWORD=

S3_ENABLED=true
S3_PROTOCOL=https
S3_BUCKET=media-instance-tld
S3_REGION=us-west-1
S3_HOSTNAME=s3.us-west-1.amazonaws.com
#S3_ENDPOINT=https://media-instance-tld.s3.us-west-1.amazonaws.com/
AWS_ACCESS_KEY_ID=xxx
AWS_SECRET_ACCESS_KEY=xxx
S3_ALIAS_HOST=media.instance.tld

[hexsix]

For example, nginx failed to get /packs/js/locale_zh-CN-bf8fe85aeb8ceb0190a7.chunk.js

nginx/access.log

/var/log/nginx/access.log:x.x.x.x - - [22/Mar/2024:01:27:50 +0800] "GET /packs/js/locale_zh-CN-bf8fe85aeb8ceb0190a7.chunk.js HTTP/2.0" 404 481 "https://mastodon.instance.tld/home" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"

I confirm that locale_zh-CN-bf8fe85aeb8ceb0190a7.chunk.js is under /packs/js/

ls public/packs/js/

$ docker exec -it mastodon_web_1 bash
mastodon@3592ca660055:~$ ls public/packs/js/ | grep 'locale_zh-CN-bf8fe85aeb8ceb0190a7.chunk.js'
locale_zh-CN-bf8fe85aeb8ceb0190a7.chunk.js
locale_zh-CN-bf8fe85aeb8ceb0190a7.chunk.js.br
locale_zh-CN-bf8fe85aeb8ceb0190a7.chunk.js.gz
locale_zh-CN-bf8fe85aeb8ceb0190a7.chunk.js.map
locale_zh-CN-bf8fe85aeb8ceb0190a7.chunk.js.map.br
locale_zh-CN-bf8fe85aeb8ceb0190a7.chunk.js.map.gz

And here's my web container configuration if anyone need it

docker-compose.yml

  web:
    container_name: mastodon_web_1
    image: ${image}
    restart: unless-stopped
    env_file: .env.production
    command: bash -c "rm -f /mastodon/tmp/pids/server.pid; bundle exec rails s -p 3000"
    networks:
      - external_network
      - internal_network
    healthcheck:
      # prettier-ignore
      test: ['CMD-SHELL', 'wget -q --spider --proxy=off localhost:3000/health || exit 1']
    ports:
      - '127.0.0.1:3000:3000'
    depends_on:
      - db
      - redis
      # - es
    volumes:
      - ./public/system:/mastodon/public/system

[hexsix]

Go to cloudflare caching configuration and purge everything fix my problem.

I found that Cf-Cache-Status in the header is HIT, which can be found in DevTools Network section.

Yes, I used try_files $uri =404; for a couple of days. But cloudflare cache make my static files still work.