New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Contexts#wrap(ExecutorService) doesn't propagate OTEL span #8722
Comments
@boris-senapt Thanks for the report. Can you provide a simple reproducer project that shows this? If not I'll start creating one but we'd prefer one from you so we are sure we are looking at the specific problem you are. |
Sure thing - should be able to get you a reproducer tomorrow. |
@tjquinno I have pushed a recreate up, apologies for the delay https://github.com/boris-senapt/helidon-trace-propagation-recreate |
Thanks very much for the reproducer. I am seeing what you are in the logging output but it looks to me as if the span context is being propagated properly. I made the following small change to the
and here is the output I see:
Although the log message prefix from the async code does not contain the span and trace IDs the message built using the retrieved span context does show those IDs and they match the IDs logged by the code that receives the request. Maybe the logging/MDC is not working quite correctly with the span context in this scenario. Based on what I've seen I'm going to recategorize this as a logging issue rather than a tracing one. Someone else might carry on from here as that's not an area I'm as familiar with. |
Interesting - so it might be that the Helidon Tracing context has kept the span in "scope" but the underlying OTEL framework hasn't; I suppose some battle of thread locals is happening. The logging uses This uses The implications are a bit deeper than just logging however - the reason I came across the issue is that we use OKhttp to make calls to other services, and those calls were also losing their traces. i.e. whilst Helidon seems to keep its |
Your note sent me back to the revised test project and, while More digging to do on my end. |
Definitely a bug in tracing. PR forthcoming. |
In using Helidon SE without the Java agent I tried to propagate context information to an async task using
io.helidon.common.context.Contexts#wrap(java.util.concurrent.ExecutorService)
- but the OTEL span information was dropped.This resulted in the async task traces being uncorrelated with the original request. Double wrapping resolves the immediate issue, but is counter-intuitive and breaks the abstraction Helidon Tracing provides
Environment Details
Problem Description
I have a request that spawns a long-running async task on an
ExecutorService
. When the request comes in, I callexecutor.submit(Runnable)
.I expect this async task to have the same
traceId
as the original request, so I wrap theExecutorService
inContexts.wrap
. This claims to propagate context across threads - which I take to mean Tracing context as well as other contexts.However, in the traces and in the logs the trace id is not the same - and a new trace is generated. On investigation there is only one implementation of
io.helidon.common.context.spi.DataPropagationProvider
and that propagates the Log4j2 MDC.Adding a second wrapping with the OTEL API -
io.opentelemetry.context.Context#taskWrapping(java.util.concurrent.ExecutorService)
resolves the issue, but requires direct usage of the OTEL API, breaking the Helidon Tracing abstraction layer.The text was updated successfully, but these errors were encountered: