[Enhancement] Add Tracing support in the core of broker

[ajatprabha]

Is it possible to add tracing support in VerneMQ based on OpenTelemetry? I would also like to understand what performance implication it may bring.

Environment

• VerneMQ Version: NA
• OS: NA
• Erlang/OTP version (if building from source): OTP 23
• VerneMQ configuration (vernemq.conf) or the changes from the default
• Cluster size/standalone: NA

Expected behaviour

• Add support in VerneMQ to collect traces from various flows in code and push them to a tracing backend.

Actual behaviour

• Currently, there is no out-of-the-box support for tracing

PS: I'd be happy to contribute if this enhancement is taken into consideration.

[ioolkos]

@ajatprabha Thanks for your idea. I appreciate your remark on helping on this!

I have invested quite some work to explore this a couple months back, and laid out an architecture that can work with the Erlang SDK of OpenTelemetry. This allows OpenTelemetry tracing to be switched on and off. It still requires to add in-place wrappings around calls in the VerneMQ source code. In case tracing is switched off, they'd be no-ops.

There's a certain technical complexity in all this, and the Erlang implementation of OpenTelemetry is still somewhat moving or stabilising currently.

To be honest, though, the main decision here is wether this should be an open-source feature or not.

[ioolkos]

As of 2 days, OpenTelemetry Erlang is now 1.0.0 https://github.com/open-telemetry/opentelemetry-erlang
So potentially API-stable.

[ajatprabha]

As of 2 days, OpenTelemetry Erlang is now 1.0.0

That's awesome 🚀

I have invested quite some work to explore this a couple months back, and laid out an architecture that can work with the Erlang SDK of OpenTelemetry.

Great, how do we go about this? It'd be great if you can share some ideas.

To be honest, though, the main decision here is wether this should be an open-source feature or not.

Do you have a process in place which can be used to assess whether this should be open-sourced or not? I think since OpenTelemetry itself encourages standardisation of APIs/SDKs for building meaningful metrics/logs/traces and provide better interoperability with different observability options. Also, prometheus metrics are already open-sourced, can we look at supporting this as well?

[codeadict]

This would be great, i think is easier for MQTT 5 with a custom property to send a request ID

[zlb323]

Is this something that you are still considering adding at some point?

[galah92]

Looking for this as well as I'd like to use OpenTelemetry for observability.

[ioolkos]

Here's your starting point for anyone interested: https://opentelemetry.io/docs/instrumentation/erlang/
Will happily review any instrumentation and plugin implementation! (and if we can build up to a usable form, maintain as open source)

[codeadict]

I have also thought about this, will be simpler to do with MQTT5 using properties for the "request_id" but the protocol for 3.x MQTT is a bit limiting to implement this, the "trace id" will need to come thru the payload to not break the standard MQTT. If other systems are publishing to VerneMQ and we need to know a good way of knowing how the payload specifies that trace id

…

On Wed, Oct 19, 2022 at 10:13 AM ioolkos ***@***.***> wrote: Here's your starting point for anyone interested: https://opentelemetry.io/docs/instrumentation/erlang/ Will happily review any instrumentation and plugin implementation! (and if we can build up to a usable form, maintain as open source) — Reply to this email directly, view it on GitHub <#1919 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABJD6C35GKPZTWCTILFQJ3WD76Z7ANCNFSM5J5PGMWA> . You are receiving this because you commented.Message ID: ***@***.***>

-- [image: created with MySignature.io] Dairon Medina Caro Freelance Software Developer M: +1 (813) 325-6978 <+1+(813)+325-6978> E: ***@***.*** dairon.org [image: created with MySignature.io] <https://github.com/codeadict> [image: created with MySignature.io] <http://codeadict> <https://mysignature.io?utm_source=promotion&utm_medium=signature&utm_campaign=create_own_signature>

[ioolkos]

Semantic Conventions for messaging systems:
https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/messaging/

Suggestions on the v3/v5 format:

Trace Context: MQTT protocol
https://w3c.github.io/trace-context-mqtt/

[codeadict]

Ohh that is gold @ioolkos!

…

On Fri, Oct 21, 2022, 4:35 AM ioolkos ***@***.***> wrote: *Semantic Conventions for messaging systems:* https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/messaging/ Suggestions on the v3/v5 format: *Trace Context: MQTT protocol* https://w3c.github.io/trace-context-mqtt/ — Reply to this email directly, view it on GitHub <#1919 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABJD6CXD2QKUIS5ASMRNV3WEJIUHANCNFSM5J5PGMWA> . You are receiving this because you commented.Message ID: ***@***.***>

[ioolkos]

I'll go through the material linked above, time permitting in the next couple days.
I think we can add the opentelemetry_api lib to a VerneMQ release, and then add an implementation via a plugin (vmq_otel). We can then freely identify code points where we want to generate spans, and add them over time to the Verne code base. We'll have to see what's possibly for real distributed tracing, though.

[ioolkos]

Started something, if anyone wants to chime in #2101

[codeadict]

Not looking a the PR yet but ideas of what could/should be traced:

• Tracing of MQTT PUBLISH in and out
• Tracing of vmq_webhooks making external requests.
• Tracing of publish/subscribe Authorizations.
• Tracing of Hooks, ideally to measure plugin overhead.
• Tracing of vmq_diversity calls to external DBs.
• Tracing of bridge messages going out.

[ioolkos]

Great list of entry points.

[ioolkos]

I'm currently refining the points. One thing I'm tempted to do is to limit Distributed Tracing to MQTT v5. The reason for this is in the Trace Context specs linked above. For v5, we can have a traceparent and a tracestate in the user properties of MQTT frames. In v3 we'd have to use the binary approach and parse the tracing information from an MQTT payload. I'm suspecting nobody is going to want to do that.