Pusher failed to consume trace data err="DoBatch: InstancesCount <= 0

[chazapp]

I'm trying to implement OpenTelemetry tracing in my Django DRF Application, using OTEL Django Auto Instrumentation. I ultimately want to push my traces into Tempo to visualize them into Grafana later on. I'm using OTLPSpanExporter to push my traces via gRPC on port 4317. I'm doing this locally with Tempo running in a Docker container. When the traces are pushed, I get this error Pusher failed to consume trace data err="DoBatch: InstancesCount <= 0 in Tempo logs. What am I doing wrong ? Here's the full context:

Incidents-API Settings.py

# settings.py
if os.getenv("TRACING_BACKEND_HOST"):
    DjangoInstrumentor().instrument()
    trace.set_tracer_provider(
        TracerProvider(
                resource=Resource.create({SERVICE_NAME: "incidents-api"})
        )
    )
    tracer = trace.get_tracer(__name__)
    otlp_exporter = OTLPSpanExporter(
        endpoint=os.getenv("TRACING_BACKEND_HOST"), # localhost:4317
    )

    # Create a BatchSpanProcessor and add the exporter to it
    span_processor = BatchSpanProcessor(otlp_exporter)

    # add to the tracer
    trace.get_tracer_provider().add_span_processor(span_processor)

# docker-compose.yml
version: "3"
services:
  tempo:
    image: grafana/tempo:latest
    command: "-target=distributor -config.file=/etc/tempo.yml"
    volumes:
      - ./tempo.yml:/etc/tempo.yml
    ports:
      - "3200:3200/tcp"
      - "4317:4317/tcp"

# tempo.yml
compactor: null
server:
  http_listen_port: 3200
distributor:
    receivers:
      otlp:
        protocols:
          grpc:
ingester:
    trace_idle_period: 1s
    lifecycler:
        ring:
            replication_factor: 3
storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces
    wal:
      path: /var/tempo/wal

$ python manage.py runserver --noreload
Performing system checks...

System check identified no issues (0 silenced).
September 02, 2022 - 15:45:33
Django version 4.0.6, using settings 'incidents.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
[02/Sep/2022 15:45:47] "GET /health HTTP/1.1" 301 0
[02/Sep/2022 15:45:47] "GET /health/ HTTP/1.1" 200 2
Failed to export traces, error code: StatusCode.UNKNOWN

level=info ts=2022-09-02T13:49:20.242055169Z caller=main.go:200 msg="initialising OpenTracing tracer"      
level=warn ts=2022-09-02T13:49:20.242850817Z caller=main.go:98 --CONFIGURATIONWARNINGS--=(MISSING)
level=warn ts=2022-09-02T13:49:20.242880617Z caller=main.go:104 msg="compactor.compaction.compacted_block_timeout < storage.trace.blocklist_poll" explain="Queriers and Compactors may attempt to read a block that no longer exists"
level=warn ts=2022-09-02T13:49:20.242892887Z caller=main.go:104 msg="c.Compactor.Compactor.RetentionConcurrency must be greater than zero. Using default." explain="default=10"
level=warn ts=2022-09-02T13:49:20.242900037Z caller=main.go:104 msg="Local backend will not correctly retrieve traces with a distributed deployment unless all components have access to the same disk. You should probably be using object storage as a backend."
level=info ts=2022-09-02T13:49:20.242944507Z caller=main.go:115 msg="Starting Tempo" version="(version=, branch=main, revision=525b0265)"
level=info ts=2022-09-02T13:49:20.243261566Z caller=server.go:306 http=[::]:3200 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2022-09-02T13:49:20.244034503Z caller=memberlist_client.go:436 msg="Using memberlist cluster label and node name" cluster_label= node=2c1ae38e1f4f-ae516793
ts=2022-09-02T13:49:20Z level=info msg="OTel Shim Logger Initialized" component=tempo
level=info ts=2022-09-02T13:49:20.24541415Z caller=module_service.go:82 msg=initialising module=server     
level=info ts=2022-09-02T13:49:20.24544886Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=info ts=2022-09-02T13:49:20.24545973Z caller=module_service.go:82 msg=initialising module=overrides  
level=info ts=2022-09-02T13:49:20.24547574Z caller=module_service.go:82 msg=initialising module=ring       
level=info ts=2022-09-02T13:49:20.24548481Z caller=module_service.go:82 msg=initialising module=usage-report
level=info ts=2022-09-02T13:49:20.245730601Z caller=ring.go:263 msg="ring doesn't exist in KV store yet"   
level=info ts=2022-09-02T13:49:20.24579096Z caller=module_service.go:82 msg=initialising module=distributor
ts=2022-09-02T13:49:20Z level=info msg="Starting GRPC server on endpoint 0.0.0.0:4317" component=tempo     
level=info ts=2022-09-02T13:49:20.245946935Z caller=app.go:195 msg="Tempo started"
level=error ts=2022-09-02T13:50:26.634686845Z caller=rate_limited_logger.go:27 msg="pusher failed to consume trace data" err="DoBatch: InstancesCount <= 0"

For reference, I've tried swapping Tempo with OpenTelemetry collector and got the exact same error, but I still fail to find what is missing to my code to finally get traces ingested.
Thank you for your help

[joe-elliott]

It looks like you're trying to run Tempo in "single binary" mode, but also setting RF=3 which won't work.

Try reducing replication_factor to 1.

[chazapp]

I've set the replication_factor to 1 and restarted the container but still get the same error.

[joe-elliott]

ok, I see. you are starting up just a distributor. this is just one of a few independently scaleable components that make up Tempo.

you likely want to run tempo in a "single binary" or "monolith" configuration which will start all of the components in one process. to start Tempo in this manner remove the -target=distributor parameter.

If you're still having issues here is a local config file we use in our examples:
https://github.com/grafana/tempo/blob/main/example/docker-compose/local/tempo-local.yaml

It's a bit overkill but should work.

[chazapp]

This was it ! I had copied the docker-compose definition from the microservices examples when I'm clearly looking for the monolithic version (for now !).
Thank you very much !

[fnordian]

https://github.com/grafana/tempo/blob/main/example/docker-compose/local/tempo-local.yaml has disappeared

[mapno]

It's here now https://github.com/grafana/tempo/blob/main/example/docker-compose/shared/tempo.yaml