Kiali does not show jaeger traces

[sterziev88]

Describe the bug

I have installed Kiali operator via helm chart. I am able to see workload, services, applications and graphs for them but I am not able to see jaeger traces.

That is my configuration:

    server:

      observability:
        tracing:
          collector_type: otel
          collector_url: http://jaeger-operator-jaeger-collector.jaeger-operator.svc.cluster.local:4317
          enabled: false
          otel:
            ca_name: ''
            protocol: grpc
            skip_verify: true
            tls_enabled: false

      tracing:

        enabled: true
        use_grpc: true
        provider: "jaeger"
        in_cluster_url: "http://jaeger-operator-jaeger-query..jaeger-operator.svc.cluster.local:16685/jaeger"
        url: "http://jaeger.xxxxxxx"

Expected Behavior

To be able to see jaeger traces

Environment

Learn about how to determine versions here.

• **Kiali version:1.81.0
• **Istio version:1.21
• **Kubernetes version:1.29

[jmazzitelli]

(converted to Q&A Discussion as this is a question on configuration, not a bug. Also, this is a question about configuring the server, not the operator)

[jmazzitelli]

Maybe the docs can help:

• https://kiali.io/docs/configuration/p8s-jaeger-grafana/tracing/jaeger/
• the Kiali CR Jaeger config reference docs: https://kiali.io/docs/configuration/kialis.kiali.io/#.spec.external_services.tracing
• the tracing FAQ: https://kiali.io/docs/faq/distributed-tracing/

[sterziev88]

Hi, I have rad the documents but everything is set as described in them and still I don't see any traces.

Regarding: https://kiali.io/docs/configuration/p8s-jaeger-grafana/tracing/jaeger/
I have this:

    external_services:
      tracing:
        enabled: true
        use_grpc: true
        provider: "jaeger"
        in_cluster_url: "http://jaeger-operator-jaeger-query..jaeger-operator.svc.cluster.local:16685/jaeger"
        url: "http://jaeger.xxxxxxx"

and I am able to see the link to my jaeger UI but nothing related to traces on kiali UI.

Regarding: https://kiali.io/docs/configuration/kialis.kiali.io/#.spec.external_services.tracing
Everything which is need is set properly according provided documentation.

Regarding: https://kiali.io/docs/faq/distributed-tracing/
I don't receive any errors or logs with warnings/error.

Could someone help with this?

[jshaughn]

I assume you are seeing traces via the Jaeger UI, so the traces exist? Anything in the Kiali logs? Any indication in the Kiali masthead that Jaeger is unreachable? Are you sure the in_cluster_url is correct?

[sterziev88]

Hi @jshaughn Yes, I am able to see the traces via Jaeger UI. In the Kiali logs I see only this regarding jaeger -> 2024-05-08T10:05:22Z INF Create jaeger GRPC client jaeger-operator-jaeger-query.jaeger-operator.svc.cluster.local:16685.
I am using an opentelemetry collector in order to collect traces from the applications ( I am not sure if that can be related ) but after that I send them to jaeger to that endpoint jaeger-operator-jaeger-collector.jaeger-operator.svc.cluster.local and then I use query service of jeager so Yes, that should be the correct in_cluster_url -> jaeger-operator-jaeger-query.jaeger-operator.svc.cluster.local

[jshaughn]

Kiali is only consuming the traces, so using OTel as a collector is fine and should not be relevant.

Do you have a very low number of traces? Perhaps increase the sampling by adjusting the Limit per query setting in the Display dropdown?

Is it only the Traces tab on the workload detail page? If you select a node in the traffic graph, do you see anything under the Traces tab in the side panel?

@josunect , do you have any ideas here?

[josunect]

I wonder if the issue is related to some information missing in the trace.
The setting tracing.namespace_selector is true but default, could you try setting that to false and see if it can find any traces?

[sterziev88]

Hi @jshaughn and @josunect I set tracing.namespace_selector to false and now I am able to see traces.

I am wondering why this change fix the things.
Thank you very much both of you!

[josunect]

Thanks for let us know!
This is because, Kiali queries the trace service by the service name (Ex productpage-bookinfo), and it needs to match exactly. This is usually the service name and the namespace, but sometimes it is just stored without it (Ex productpage).

[gkotsev]

Hi, I have similar Kiali issue, but the problem for me is with the Inbound / Outbound Metrics. I'm able to see metrics in Grafana, but nothing in Kiali:

Grafana is on a remote cluster and this is the configuration I have:

grafana:
  auth:
    type: "basic"
    username: "?????"
    password: "?????"
  dashboards:
  - name: "Istio Service Dashboard"
    variables:
      namespace: "var-namespace"
      service: "var-service"
  - name: "Istio Workload Dashboard"
    variables:
      namespace: "var-namespace"
      workload: "var-workload"
  - name: "Istio Mesh Dashboard"
  - name: "Istio Control Plane Dashboard"
  - name: "Istio Performance Dashboard"
  - name: "Istio Wasm Extension Dashboard"
  enabled: true
  in_cluster_url: "http://grafana.???.com"
  url: "http://grafana.???.com

[josunect]

Thanks, @gkotsev

Is this from the same cluster?
If you follow the links:
Traffic > View metrics

Is there any data?
If you follow the Grafana link (In the right corner):

Do you see any data in the dashboard? Is the information correct? (The workload, and namespace)

[gkotsev]

Hi @josunect.
Yes, this is the same cluster.
It looks like I don't have the action link "View metrics":

But when I follow the Grafana link, I can see some graphs:

[josunect]

Thanks, @gkotsev
When you follow the Grafana link from Kiali, is this information correct? (Workload, namespace)

[gkotsev]

Yes, it's the same Workload, Namespace and Reporter.

[josunect]

Kiali uses the following metrics to calculate these graphs.
Setting the Kiali log level to "trace", will print the exact queries that Kiali is doing to Prometheus, this is an example:

TRC [Prom] fetchRange: sum(rate(istio_tcp_connections_opened_total{reporter="source",source_workload_namespace="bookinfo",source_workload="productpage-v1",source_cluster="Kubernetes"}[36s])) by (destination_canonical_service,destination_workload_namespace)

This can be very useful (You can use the exactly metric in Prometheus) to troubleshoot, and if some metrics are missing.

This is a very useful message at the end:
TRC [Prom] GetExistingMetricNames: exec time=[12.451443ms], results count=[1160], looking for count=[10], found count=[10]