How to properly parse OTEL-format logs collected by Datadog Agent?

[julealgon]

I'm using the Datadog Agent as an intermediate to push logs written in OTEL Protocol format to Datadog. This is a workaround to the problem described here:

• [exporter/datadog] Log exporter support open-telemetry/opentelemetry-collector-contrib#2651

This is the configuration I'm using in the DD Agent right now:

logs:
  - type: file
    path: "C:\\otelcol-contrib_0.49.0_windows_amd64\\data.json"
    service: "MyService"
    source: "MySource"

And this is my OTEL collector setup:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
    timeout: 10s

exporters:
  file:
    path: ./data.json
    
  datadog/api:
    host_metadata:
      tags:
        - env:dev
    api:
      key: {my_key_here}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog/api]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog/api]
    logs:
      receivers: [otlp]
      processors: [logstransform]
      exporters: [file]

After configuring my OTEL Collector to output logs to a local file, then installing DD agent in the same server and pulling logs from that file, I was able to see log entries flowing through to Datadog, however they are "posted as-is": meaning all I see are raw JSON batches of OTEL-formatted json logs, and not the actual log entries themselves:

Each "entry" above is actually a 5 second OTEL batch of logs each containing several log entries inside.

I was expecting Datadog to be "smart enough" to recognize that the data being pushed is OTEL Protocol logs data and properly parse the json and create all the necessary individual log events with their specific services, messages, etc.

Am I missing something? Both in the original issue above, as well as in this other question:

• Confusion regarding DataDog Agent vs Vector vectordotdev/vector#13388

I was led to believe that it was possible to use the DD agent to overcome the current lack of a proper OTEL log exporter in the OpenTelemetry Collector, but if all it can do right now is post the raw batch json objects, then that's a fairly poor integration in my opinion and will make finding log entries later a nightmare.

Is there a way to set this up so that actual log entries are posted instead of the raw batch Json? Maybe something I can do either in the DD agent setup, or a Datadog ingestion pipeline to properly "parse" the OTEL Json object an split it into the actual log events, or even a configuration I could do in the collector itself that would allow for this?

@binarylogic and @jszwedko if you can help with this it would be much appreciated. Let me know if switching to Vector would be the sensible option here as well (assuming Vector can resolve this issue)

[jszwedko]

Hi @julealgon !

Gotcha, yeah I see that file exporter for the OpenTelemetry collector batches up the logs when it writes them.

Someone in the linked issue seems to have taken the tact of, rather than running the OpenTelemetry collector at all, just having the applications write OpenTelemetry formatted logs to stdout/stderr and have the Datadog Agent collect those.

Vector could help here. It is capable of splitting up the logs into multiple via the remap transform which could then be sent directly to Datadog. If you could grab one of those example JSON payloads, I could help write a configuration to convert it into multiple log events.

We also expect to release OpenTelemetry log support for Vector this quarter so another option would be to wait for that, depending on the urgency.

Hope this helps!

[julealgon]

Gotcha, yeah I see that file exporter for the OpenTelemetry collector batches up the logs when it writes them.

To be fair I'm not sure if its the file exporter that is batching the results, or if its the senders themselves that do some batching by default. I actually thought it was the latter but I guess it doesn't make a difference at the end of the day: I'm not willing to change default batching configurations on the log exporter on every application anyways, that would defeat some of the purpose of the OTel standardization and become another maintenance pain (and also probably become very inefficient in the process).

Someone in the linked issue seems to have taken the tact of, rather than running the OpenTelemetry collector at all, just having the applications write OpenTelemetry formatted logs to stdout/stderr and have the Datadog Agent collect those.

I'm probably missing something very obvious, but I have no idea how one even does that. In our setup in particular, we leverage a lot of Azure serverless stuff like AppService APIs, Azure Functions, etc. I don't think you can even tap into the console output for those things at all so I can't see this approach working ever.

Vector could help here. It is capable of splitting up the logs into multiple via the remap transform which could then be sent directly to Datadog. If you could grab one of those example JSON payloads, I could help write a configuration to convert it into multiple log events.

That's interesting. I'm currently considering another approach which would leverage Vector's receiver capabilities to pipe log events from the OTel collector into Vector using either the Splunk HEC format, or the Logstash (ElasticSearch) log format. My understanding is that this type of communication should work, and then Vector should be able to export these logs to DD using standard DD format.

That feels less "hacky" than attempting to manually parse and amend the OTel Json IMHO.

Please let me know if this sounds promising or if you immediately see any caveats.

We also expect to release OpenTelemetry log support for Vector this quarter so another option would be to wait for that, depending on the urgency.

That's good to know, though I'd rather have the much more general purpose OTel logs exporter prioritized.

[jszwedko]

Vector could help here. It is capable of splitting up the logs into multiple via the remap transform which could then be sent directly to Datadog. If you could grab one of those example JSON payloads, I could help write a configuration to convert it into multiple log events.

That's interesting. I'm currently considering another approach which would leverage Vector's receiver capabilities to pipe log events from the OTel collector into Vector using either the Splunk HEC format, or the Logstash (ElasticSearch) log format. My understanding is that this type of communication should work, and then Vector should be able to export these logs to DD using standard DD format.

That feels less "hacky" than attempting to manually parse and amend the OTel Json IMHO.

Please let me know if this sounds promising or if you immediately see any caveats.

👍 that does sound promising. I'd suggest giving that a shot with the splunk_hec Vector source and the Splunk HEC OpenTelemetry exporter.

We also expect to release OpenTelemetry log support for Vector this quarter so another option would be to wait for that, depending on the urgency.

That's good to know, though I'd rather have the much more general purpose OTel logs exporter prioritized.

Yeah, agreed. Just noting that we did actually merge in the initial implementation for the opentelemetry source recently so you could try the latest nightly build of Vector for ingesting logs if you want (docs here: https://master.vector.dev/docs/reference/configuration/sources/opentelemetry/).

[julealgon]

I can attest that using Splunk HEC as the intermediate protocol to send logs from otel collector to vector works fairly well indeed. I did a few tests over the past few days and I can properly integrate with Datadog.

The only issue that I wasn't expecting is that the vector exporter doesn't do any transformation on the payload before pushing the log records into Datadog using the Datadog Log exporter, so I had to actually mess around with the format using vector's replacement transformations. After doing that however I think I got logs showing up correctly using DD fields.

This should serve as my next "backup solution", before vector releases the OTel log ingestion, and before DD releases the otel-collector logs exporter.

[jszwedko]

Thanks for reporting back! Hopefully this helps others as well.