dockergen.1: invalid value for containers which should be ignored

[QuAzI]

I am playing with Prometheus and I found that nginx-proxy always flood when it starts or stops

nginx-proxy                | dockergen.1 | 2023/02/09 10:38:55 Received event die for container dd6fa4697fa9
nginx-proxy                | dockergen.1 | 2023/02/09 10:38:55 Received event die for container e046a6256eec                                                                                  nginx-proxy                | dockergen.1 | 2023/02/09 10:38:55 Received event stop for container dd6fa4697fa9
nginx-proxy                | dockergen.1 | 2023/02/09 10:38:55 Received event stop for container e046a6256eec                                                                                 nginx-proxy                | dockergen.1 | 2023/02/09 10:38:56 invalid value
nginx-proxy                | dockergen.1 | 2023/02/09 10:38:56 invalid value
nginx-proxy                | dockergen.1 | 2023/02/09 10:38:56 invalid value
nginx-proxy                | dockergen.1 | 2023/02/09 10:38:56 invalid value
nginx-proxy                | dockergen.1 | 2023/02/09 10:38:56 invalid value
nginx-proxy                | dockergen.1 | 2023/02/09 10:38:56 invalid value
nginx-proxy                | dockergen.1 | 2023/02/09 10:38:56 invalid value
nginx-proxy                | dockergen.1 | 2023/02/09 10:38:56 invalid value
nginx-proxy                | dockergen.1 | 2023/02/09 10:38:56 invalid value
...
nginx-proxy                | dockergen.1 | 2023/02/09 10:38:56 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'

and if Prometheus started earlier, nginx-proxy crashes on start

Problematic docker-compose.yml for Prometheus

version: '3.9'

services:
  prometheus:
    image: prom/prometheus:v2.42.0
    container_name: prometheus
    restart: always
    volumes:
      - ./prometheus/:/etc/prometheus/
      - prometheus_data:/prometheus
    ports:
      - 9090:9090
    command: --web.enable-lifecycle --config.file=/etc/prometheus/prometheus.yml
  alertmanager:
    image: prom/alertmanager:v0.25.0
    container_name: alertmanager
    restart: always
    ports:
      - 9093:9093
    volumes:
      - ./alertmanager:/config
      - alertmanager-data:/data
    command: --config.file=/config/alertmanager.yml --log.level=debug
    depends_on:
      - prometheus
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    restart: always
    ports:
      - 8080:8080
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    depends_on:
      - redis
      - prometheus
  redis:
    image: redis:latest
    container_name: redis
    restart: always
    ports:
      - 6379:6379

volumes:
  prometheus_data:
  alertmanager-data:

[rhansen]

I am playing with Prometheus and I found that nginx-proxy always flood

Does nginx-proxy still work?

when it starts or stops

I assume "it" in your sentence refers to Prometheus, not nginx-proxy. Right?

and if Prometheus started earlier, nginx-proxy crashes on start

What do you mean by "crashes"? What is the relevant log output?

Problematic docker-compose.yml for Prometheus

I see no VIRTUAL_HOST environment variable set for any of the containers. Are you expecting them to be proxied by nginx-proxy?

I also don't see your nginx-proxy container here. I assume it is in a different compose file. Can you provide that?

What are the contents of /etc/nginx/conf.d/default.conf in your nginx-proxy container? (run docker exec $nginx_proxy_container_name exec nginx -T)

[QuAzI]

1. nginx-proxy still work if Prometheus env starts after nginx-proxy. If Prometheus env starts first - nginx-proxy can't start.
2. yes
3. I will post it next time when Prometheus starts faster than nginx-proxy. It happens not all time) But still a lot of errors in log
   nginx-proxy-works-but-with-errors.zip
4. I am expecting only that Prometheus doesn't affect nginx-proxy. Looks like some conflict with cadvisor volumes.
   nginx-container configured in different place of course and worked fine before for years)
   nginx-conf-dump-20230209.txt

[rhansen]

But still a lot of errors in log

The "invalid value" messages are probably benign, but I can't be certain without examining the contents of /etc/nginx/conf.d/default.conf.

I am expecting only that Prometheus doesn't affect nginx-proxy.

nginx-proxy works by querying Docker for available containers on startup, and by watching for changes to containers. Any time any container starts or exits, nginx-proxy wakes up and reprocess the containers. So Prometheus does affect nginx-proxy in that sense, but nginx-proxy should ignore the Prometheus container if the VIRTUAL_HOST environment variable is not set on the Prometheus container. You can use docker inspect on the Prometheus container to examine the environment variables that nginx-proxy sees. Please copy and paste those environment variables so that we can see if there's an undesired interaction.

Looks like some conflict with cadvisor volumes.

What makes you say that?

nginx-container configured in different place of course and worked fine before for years)

I don't understand. Are you saying that on another machine nginx-proxy is working as expected and generates the config you attached? Or that it worked for years but something broke recently? If the latter, what have you changed?

[bmccorm2]

I'm having the same issue so i tried to distill it down as much as possible. I'm not running any other containers to proxy for now.

docker-compose.yml

version: "3.6"

services:
  proxy:
    image: nginxproxy/nginx-proxy
    container_name: proxy
    restart: always
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro

Not sure if this is an error or has any other ill effects but the log:

proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 15#15: using the "epoll" event method
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 15#15: nginx/1.23.3
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 15#15: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 15#15: OS: Linux 5.10.0-21-amd64
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 15#15: getrlimit(RLIMIT_NOFILE): 1048576:1048576
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 15#15: start worker processes
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 15#15: start worker process 19
proxy    | dockergen.1 | 2023/02/11 05:05:29 invalid value
proxy    | dockergen.1 | 2023/02/11 05:05:29 Generated '/etc/nginx/conf.d/default.conf' from 1 containers
proxy    | dockergen.1 | 2023/02/11 05:05:29 Running 'nginx -s reload'
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 15#15: signal 1 (SIGHUP) received from 22, reconfiguring
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 15#15: reconfiguring
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 15#15: using the "epoll" event method
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 15#15: start worker processes
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 15#15: start worker process 23
proxy    | dockergen.1 | 2023/02/11 05:05:29 Watching docker events
proxy    | dockergen.1 | 2023/02/11 05:05:29 invalid value
proxy    | dockergen.1 | 2023/02/11 05:05:29 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 19#19: gracefully shutting down
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 19#19: exiting
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 19#19: exit
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 15#15: signal 17 (SIGCHLD) received from 19
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 15#15: worker process 19 exited with code 0
proxy    | nginx.1     | 2023/02/11 05:05:29 [notice] 15#15: signal 29 (SIGIO) received

[rhansen]

It turns out the "invalid value" message is a debug message that I accidentally left in when making some changes. It can be ignored. See nginx-proxy/docker-gen#494