Invalid second event generated after span ended

[SgtSchultz02]

Describe the bug
We are manually instrumenting using the Java Api. There are two spans created for a trace. When the trace is created we are adding a single reference to another span that is ended. The reference shows up within the first span, as expected. Included with this reference is another reference with the current trace's trace id with a Span Id of all zeros. This is visible in Grafana.

To Reproduce
Steps to reproduce the behavior:

1. Start Application, Tempo, Collector, Grafana
2. Create Trace1 then Span1 and end it
3. Create Trace2 with a reference (Ref1) to Span1
4. Create Span2 - contains the reference
5. Create Span3
6. View Trace2 within Grafana. The spans references (Ref0 - invalid, Ref1 - valid) show up as described in the Description above

Expected behavior
We expect the single reference to Ref1

Environment:

• Infrastructure: linux laptop/docker. Also shows up in a linux test environment.
• Deployment tool: docker

Additional Context
Investigated by enabling logging in the below

• Java API - looks OK here with only one event
• Collector - looks OK here with only one event (logging at receiver)
• Will add logging at exporter and post update here

[joe-elliott]

The exact details of the issue are not quite clear to me. Could provide a Grafana screenshot with maybe a simple code sample that produces the issue?

Tempo does have the ability to log debug details about spans to help with situations like this:

https://github.com/grafana/tempo/blob/main/modules/distributor/distributor.go#L648

Here is the config:

https://github.com/grafana/tempo/blob/main/modules/distributor/config.go#L37

I'm not sure if it surface all the details we need but it might give clues.