Critical issue on nginx worker exited on signal 4

[hieuhtr]

Issue: nginx worker continuously exited on signal 4

My brotli configuration

brotli on;
brotli_comp_level 9;
brotli_static on;
brotli_min_length 1024;
brotli_types    application/atom+xml application/javascript application/json application/rss+xml
                    application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype
                    application/x-font-ttf application/x-javascript application/xhtml+xml application/xml
                    font/eot font/opentype font/otf font/truetype image/svg+xml image/vnd.microsoft.icon
                    image/x-icon image/x-win-bitmap text/css text/javascript text/plain text/xml application/x-ndjson application/vnd.spring-boot.actuator.v2+json;

Nginx error logs

2024/02/29 08:06:22 [notice] 1#1: signal 29 (SIGIO) received
2024/02/29 08:06:27 [notice] 1#1: signal 17 (SIGCHLD) received from 11110
2024/02/29 08:06:27 [alert] 1#1: worker process 11110 exited on signal 4
2024/02/29 08:06:27 [notice] 1#1: start worker process 11112
2024/02/29 08:06:27 [notice] 1#1: signal 29 (SIGIO) received
2024/02/29 08:06:27 [notice] 1#1: signal 17 (SIGCHLD) received from 11111
2024/02/29 08:06:27 [alert] 1#1: worker process 11111 exited on signal 4
2024/02/29 08:06:27 [notice] 1#1: start worker process 11113
2024/02/29 08:06:27 [notice] 1#1: signal 29 (SIGIO) received
2024/02/29 08:06:32 [info] 11112#11112: *64972 client 127.0.0.1 closed keepalive connection
2024/02/29 08:06:37 [notice] 1#1: signal 17 (SIGCHLD) received from 11112
2024/02/29 08:06:37 [alert] 1#1: worker process 11112 exited on signal 4
2024/02/29 08:06:37 [notice] 1#1: start worker process 11123
2024/02/29 08:06:37 [notice] 1#1: signal 29 (SIGIO) received

• I'm trying nginx:1.23.3 and nginx:1.24.0, using dynamic load brotli but it persist.
• content-length below 1024 B is fine as well (non compress)

Try to switch back gzip and everything is OK

As I remember, my previous nginx & brotli compiling on 2023-08-30 still fine, I just updated recently on Feb 2024

[simneu]

hi @hieuhtr,

I'm facing similar issues with a recent (but new, we didn't use brotli before) release. Did you find a resolution yet?

On my side:

• exit on signal 4 is happening every time we have brotli on in our configuration and nginx tries to serve a br compressed page. Removing brotli on from config fixes the issue.
• setup is using nginx:1.18.0 and ngx_brotli module was compiled and then packed into an AWS AMI, than we then are trying to use on our final instances. The instance building the AWS AMI and the actual end instance using the AMI are both amd64 (if that can hint to a particular direction, cause I thought this could be a module compilation issue.
• I tried debugging (with limited knowledge) what causes the signal 4, but couldn't find any hints yet.

Thanks.

[NicolasCARPi]

Had the same issue (elabftw/elabftw#4600). Had to revert to using commit 6e975bc because newer version cause this crash. Hope it can be adressed, or we'll all be stuck to this working commit ;)

It's with nginx 1.24. Full Dockerfile is available here: https://github.com/elabftw/elabimg/blob/master/Dockerfile

[simneu]

Awesome @NicolasCARPi, thanks for taking the time to help. 👍

I'll try commit you suggested when I get few minutes, and will update here with news.

[hieuhtr]

Hi guys, sorry for late update. I tried several ways but there is no hope on latest codebase.

I reverted build step by this commit hash 6e975bcb015f62e1f303054897783355e2a877dc thanks @NicolasCARPi for recommendation.

Here is my simple solution.

FROM debian:bullseye-slim AS builder

ARG NGINX_VERSION=1.24.0
ARG MOD_SECURITY_VERSION=3.0.8

# https://github.com/google/ngx_brotli/issues/166
ENV NGX_BROTLI_COMMIT_HASH=6e975bcb015f62e1f303054897783355e2a877dc

WORKDIR /root/

# Download nginx source
RUN wget http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz \
    && tar zxf nginx-${NGINX_VERSION}.tar.gz

# Install & Compile dynamic module Brotli
RUN git clone https://github.com/google/ngx_brotli \
    && cd ngx_brotli && git reset --hard $NGX_BROTLI_COMMIT_HASH \
    && git submodule update --init \
    && cd ../nginx-${NGINX_VERSION} \
    && ./configure --with-compat --add-dynamic-module=../ngx_brotli \
    && make modules

...