Distributed tracing correlation issue.

[z9fr]

Hey I do have quite similar issue to #266. My other services are sending the traceparent header and I'm using W3CTraceContextPropagator But it does not seems like the spans are correlated to the traceId from traceparent header. Instead the initial trace and the spans are created as new traces

I have attached the tracer.ts file

import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { NodeSDK } from '@opentelemetry/sdk-node';
import { AsyncLocalStorageContextManager } from '@opentelemetry/context-async-hooks';
import * as process from 'process';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import {
  CompositePropagator,
  W3CTraceContextPropagator,
  W3CBaggagePropagator,
} from '@opentelemetry/core';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import {
  SemanticResourceAttributes,
  TelemetrySdkLanguageValues,
} from '@opentelemetry/semantic-conventions';
import { Resource } from '@opentelemetry/resources';

// import { PrometheusExporter } from '@opentelemetry/exporter-prometheus';

//const metricReader = new PrometheusExporter({
//  port: 8081,
//});

const traceExporter = new OTLPTraceExporter({
  url: `http://otel-collector.tracing.svc:4318/v1/traces`,
});

const serviceName = 'api';

const spanProcessor = new BatchSpanProcessor(traceExporter);

const otelSDK = new NodeSDK({
  //  metricReader,
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: serviceName,
    [SemanticResourceAttributes.TELEMETRY_SDK_LANGUAGE]: TelemetrySdkLanguageValues.NODEJS,
  }),
  spanProcessor: spanProcessor,
  contextManager: new AsyncLocalStorageContextManager(),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': { enabled: false },
      '@opentelemetry/instrumentation-dns': { enabled: false },
      '@opentelemetry/instrumentation-net': { enabled: false },
      '@opentelemetry/instrumentation-winston': {
        enabled: true,
        logHook: (span, record) => {
          record['resource.service.name'] = serviceName;
        },
      },
      '@opentelemetry/instrumentation-http': { enabled: true },
      '@opentelemetry/instrumentation-express': { enabled: true },
      '@opentelemetry/instrumentation-aws-sdk': {
        enabled: true,
      },
      '@opentelemetry/instrumentation-redis-4': { enabled: true },
      '@opentelemetry/instrumentation-mongoose': { enabled: true },
    }),
  ],
  textMapPropagator: new CompositePropagator({
    propagators: [new W3CTraceContextPropagator(), new W3CBaggagePropagator()],
  }),
});

export default otelSDK;
// You can also use the shutdown method to gracefully shut down the SDK before process shutdown
// or on some operating system signal.
process.on('SIGTERM', () => {
  otelSDK
    .shutdown()
    .then(
      () => console.log('SDK shut down successfully'),
      (err) => console.log('Error shutting down SDK', err)
    )
    .finally(() => process.exit(0));
});

This is how the traces looks like atm.

[pragmaticivan]

Hi there, could you ensure both services are propagating the same way? I have a POC here https://github.com/pragmaticivan/nestjs-otel-prom-grafana-tempo/blob/main/services/actor/src/instrumentation.ts#L32-L41 which has 2 services and they indeed propagate properly.

[pragmaticivan]

I see from the screenshot you are using Istio.

It for that, you also need B3 propagation, check the link above.

Also check this blog post: https://www.aspecto.io/blog/opentelemetry-and-istio-everything-you-need-to-know/

(Note that OpenTelemetry uses, by default, the W3C context propagation specification, while Istio uses the B3 context propagation specification – this can be modified).