MQTTnet and OpenTelemetry Roadmap

[bunkrur]

Hello! Posting to ask some questions from a consumer perspective!

As an engineer, I desire to have an end-to-end view of edge, broker, core services. By implementing OpenTelemetry in each layer of my solutions, I can more easily achieve deeper observability and profile my solutions.

As a MQTTnet user, I desire to know the roadmap for the observability plane within MQTTnet to better understand trace, metric. and logging capacities within MQTTnet and how I can create a more unified approach to observability driven design within my solutions.

• Does MQTTnet have existing backlog items for OpenTelemetry?
• Is MQTTnet intent on using dotnet's more modern Metrics API? (System.Diagnostics.Metrics)
• Is MQTTnet intent on using dotnet's more modern Logging API? (Systems.Diagnostics.Extensions.Logging
• Is MQTTnet intent on using a shim for tracing? Tracing

[chkr1011]

Hi, see my answers below...

• Does MQTTnet have existing backlog items for OpenTelemetry?

No

• Is MQTTnet intent on using dotnet's more modern Metrics API? ([System.Diagnostics.Metrics](https://learn.microsoft.com/en-
  us/dotnet/core/diagnostics/compare-metric-apis#systemdiagnosticsmetrics))

No

• Is MQTTnet intent on using dotnet's more modern Logging API? (Systems.Diagnostics.Extensions.Logging

There will be some extension libraries in the future which will provide adapters between MQTTnet logging and others like Serilog or the logging components from Microsoft

• Is MQTTnet intent on using a shim for tracing? Tracing

No

[bunkrur]

"There will be some extension libraries in the future which will provide adapters between MQTTnet logging and others like Serilog or the logging components from Microsoft

Is there something I can reference for these extension libraries?

[kallayj]

@chkr1011 following up on this.
Support for MQTT-specific OpenTelemetry metrics (session counters, etc.) can be added indirectly, without new dependencies, by using and enhancing the existing event hook mechanism for push-based counters and public methods on MqttServer for pull-based gauges, respectively. See #1866 for an example of new events I've proposed to support specific metrics.

However, I believe more modern tracing and structured logging are blocked specifically by the goal of maintaining compatibility with .NET Standard, as the System.Diagnostics.Activity API that implements Otel-compatible tracing and structured logging is not part of .NET Standard.

I think the compatibility between the IMqttNetLogger interface and Otel is a less significant issue.

Looking for guidance on how we can relax backward-compatibility with .NET Standard in the MQTTnet project to enable use of System.Diagnostics.Activity, while being good open-source contributors. Some possible options:

1. Contribute tracing code with compiler directives targeting non-Standard .NET
2. Maintaining a friendly fork without the .NET Standard targets.

Thoughts?

[CZEMacLeod]

@kallayj I believe System.Diagnostics.Activity et al are included in the nuget package System.Diagnostics.DiagnosticSource which targets net462, netstandard20 and net6+.
If you look at the OpenTelemetry.Api you will see this dependancy and indeed the same set of target frameworks.
I think the only issue would be keeping the netstandard13 target, but it could probably be done fairly easily via compiler directives. I would suggest that most of this would be better in an external library using hooks, so as not to bloat the main package if you don't need the additional facilities.

[kallayj]

Thanks @CZEMacLeod for the tip about the System.Diagnostics.DiagnosticSource package. I believe this project also has a goal of having zero external dependencies (which led to the internal IMqttNetLogger interface replacing ILogger), so guidance is still needed from this repo's maintainer.

Tracing appears to me to be, like logging and exception handling, too close to the code to do from an external library. As a point of comparison, the calls to IMqttNetLogger still need to be made in MqttNet code. So I need some help seeing how you think this could be done.

[thompson-tomo]

I too am interested in exploring progress on the OTEL front. If you would like to maintain the zero dependency path then we would need to restrict metrics to net 8.

I do think we should move to the standardised logging abstraction as it would open up alot of flexibility.