Shutdown takes 10s even without any activity

[snicoll]

I am evaluating Tempo using docker compose version=main-23f9d00, branch=main, revision=23f9d002f.

I have postgres, grafana, tempo, and prometheus defined with what looks like vanilla configuration for Tempo:

server:
  http_listen_port: 3200
distributor:
  receivers:
    zipkin:
storage:
  trace:
    backend: local
    local:
      path: /tmp/tempo/blocks
metrics_generator:
  registry:
    external_labels:
      source: tempo
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
      - url: http://host.docker.internal:9090/api/v1/write
        send_exemplars: true
overrides:
  metrics_generator_processors: [ service-graphs, span-metrics ]

What I notice is that docker compose down is taking around 10s consistently, because Tempo takes that amount of time to shutdown. The three other services take around 0.1s to stop.

I've tried to look at the ingester configuration and played with it a little bit but that 10sec shutdown "delay" still applies. Does anybody know where that could come from? Thanks!

[joe-elliott]

possibly due to this poor choice here:

https://github.com/grafana/tempo/blob/main/modules/distributor/receiver/shim.go#L312

it has just never bothered us enough to fix it. at some point a community member was working a PR to make it configurable and drastically reduce the default. at this point we could probably just remove it since we added shutdown-delay here:

#3395

[snicoll]

at this point we could probably just remove it since we added shutdown-delay

Thanks for the feedback. Should I create an issue for this?

[joe-elliott]

Sure. Are you interested in submitting a PR to remove it?

[snicoll]

Yes! I'll add that to my list, thanks for the opportunity

[joe-elliott]

awesome! thank you :)