Using "active span concept" in asynchronous code can make faulty span connections if not careful

[nic-godz]

I have been using opentelemetry-cpp for tracing & logging for a while now. The metrics part is still to be used(have been testing it)

All C++ applications, using otel-cpp, is asynchronous and based on Boost Asio. My experience is that setting spans to active when you have a lot of asynchronous operations going can can really mess up parent-child span connections.

For instance if an outgoing request is divided into several asynchronous operations and a span is created and set active until reply is received. The scope object is kept as well for the lifetime of the request. Some nested spans are created for DNS resolve, connection, write, ...

But when parallel requests are created the same way, things start to go bad as suddenly other requests starts getting active and child spans will get the wrong connections to parents. Same goes for Logs connected to a specific span context.

My experience is to avoid setting spans to active and instead explicitly set parent span like:
options.parent = parent_span->GetContext();

And if WithActiveSpan(span) is used, the scope should never exist over asynchronous operations. But if you are using Boost Asio with stackful coroutines async code looks like sync code(thats the point) and you can easily forget that execution will leave in the middle of a function and continue later. So if a span scope is a stack variable it will survive over multiple asynchronous operation.

Whats your take on this? I have really no good ideas here. I guess it all boils down to the execution of ::RuntimeContext::GetCurrent() ? But there is really no magic you can use here to solve this. How is this handled in other languages?

[lalitb]

The use of the "active span concept" is not recommended for asynchronous code. This is because the active span relies on thread-local context for its management, but asynchronous code may operate across multiple threads internally. Your best bet would be to pass the parent span id across the async code (by capturing in lambda or other means) and explicitly set it while creating child span.

How is this handled in other languages?

I know languages like Rust, JS, Python, .Net support propagating context across the async/await calls or the futures. C++ doesn't have any such mechanism supported.

[nic-godz]

Thanks for quick response. Yes it sounds like my experience and even if the asynchronous code is not using more then one thread you will run into problem. I guess threaded applications(running synchronous in each thread) can utilize the thread local context in order to pick up the correct active span. But that is not the way to go for an event driven design on Linux.

I will for sure check out Rust and it's capabilities of propagating context in async/wait calls.
Thanks

[lalitb]

I will for sure check out Rust and it's capabilities of propagating context in async/wait calls.

If it helps, I have written the HTTP client/server example code for Otel Rust to demonstrate propagating context to async function - https://github.com/open-telemetry/opentelemetry-rust/tree/main/examples/tracing-http-propagator/src. Basically, the HTTP server main task/thread spawns an async task for every HTTP request received, and it passes the parent span context to that async task.

[nic-godz]

I'm not fluent in Rust but I must say that ".with_context(cx).await" sure looks good and is of course something we miss in C++ & Boost Asio. Really nice exmple.