-
|
I have setup Istio tracing using opentelemetry collector but the traces dont appear to be correlated as expected below are my configuration Below is the istiod release resource "helm_release" "istiod" {
name = "istiod"
repository = "https://istio-release.storage.googleapis.com/charts"
chart = "istiod"
version = "1.19.3"
namespace = var.istio_system_namespace
create_namespace = true
values = [templatefile("${path.module}/resources/istiod-values.yaml", {
istio_namespace = var.istio_system_namespace
sampling = "100"
otel_collector = "otel-collector.${var.tracing_namespace}.svc.cluster.local"
})]
depends_on = [
helm_release.istio_base,
helm_release.oauth2_proxy
]
}
below is my global:
istioNamespace: ${istio_namespace}
meshConfig:
ingressService: istio-ingressgateway
ingressSelector: istio-ingressgateway
accessLogFile: /dev/stdout
accessLogFormat: |
[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% "%UPSTREAM_TRANSPORT_FAILURE_REASON%" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME% traceID=%REQ(x-b3-traceid)% parentspanId=%REQ(x-b3-parentspanid)%
enableAutoMtls: true
enableTracing: true
extensionProviders:
- opentelemetry:
port: 4317
service: ${otel_collector}
name: otelresource "kubernetes_manifest" "telemetry_istio_system_otel_tracing" {
manifest = {
apiVersion = "telemetry.istio.io/v1alpha1"
kind = "Telemetry"
metadata = {
name = "otel"
namespace = var.istio_system_namespace
}
spec = {
tracing = [
{
providers = [
{
name = "otel"
},
]
randomSamplingPercentage = 100
},
]
}
}
depends_on = [helm_release.istiod]
}This is my istio configuration. resource "kubectl_manifest" "open_telemetry_collector" {
force_new = true
yaml_body = <<YAML
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel
namespace: ${var.tracing_namespace}
spec:
mode: deployment
config: |
receivers:
zipkin:
otlp:
protocols:
grpc:
http:
exporters:
otlp/tempo:
endpoint: tempo.${var.tracing_namespace}.svc.cluster.local:4317
tls:
insecure: true
logging/debug:
verbosity: detailed
sampling_initial: 5
sampling_thereafter: 200
loki:
endpoint: "http://loki.${var.observability_namespace}.svc.cluster.local:3100/loki/api/v1/push"
processors:
batch:
send_batch_size: 50
timeout: 5s
memory_limiter:
check_interval: 2s
limit_mib: 400
spike_limit_mib: 200
service:
pipelines:
traces:
receivers:
- zipkin
- otlp
processors:
- batch
- memory_limiter
exporters: [otlp/tempo, logging/debug]
logs:
receivers:
- otlp
processors:
- batch
- memory_limiter
exporters: [loki]
YAML
depends_on = [
helm_release.oetl,
]
}Above is my otel configuration. resource "helm_release" "tempo" {
name = "tempo"
chart = "tempo"
repository = "https://grafana.github.io/helm-charts"
namespace = var.tracing_namespace
create_namespace = true
values = [
<<YAML
distributor:
receivers:
otlp:
protocols:
grpc:
http:
ingester:
trace_idle_period: 35s
overrides:
defaults:
metrics_generator:
processors: [service-graphs, span-metrics] # enables metrics generator and metrics summary
storage:
trace:
backend: local
pool:
queue_depth: 2000
wal:
path: /tmp/tempo/wal
local:
path: /tmp/tempo/blocks
metrics_generator:
storage:
path: /tmp/tempo/generator/wal
remote_write:
- url: http://prometheus-kube-prometheus-prometheus.observability.svc.cluster.local:9090/api/v1/write"
send_exemplars: true
registry:
external_labels:
source: tempo
processor:
service_graphs:
span_metrics:
enable_target_info: true
YAML
]
}and this is my tempo config. for testing I have deployed bookinfo example
As you can see my traces are not correlated. also something odd I noticed is in the access logs I am logging the
The traceId here does not look right ? (by using I would really appreciate any help and any information about how to solve this issue |
Beta Was this translation helpful? Give feedback.
Answered by
z9fr
Apr 6, 2024
Replies: 2 comments 5 replies
-
|
|
Beta Was this translation helpful? Give feedback.
4 replies
-
|
I tried with --- a/kubernetes/resources/istiod-values.yaml
+++ b/kubernetes/resources/istiod-values.yaml
@@ -6,7 +6,7 @@ meshConfig:
ingressSelector: istio-ingressgateway
accessLogFile: /dev/stdout
accessLogFormat: |
- [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% "%UPSTREAM_TRANSPORT_FAILURE_REASON%" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME% traceID=%REQ(x-b3-traceid)% parentspanId=%REQ(x-b3-parentspanid)%
+ [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% "%UPSTREAM_TRANSPORT_FAILURE_REASON%" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME% traceID=%REQ(traceparent)%
enableAutoMtls: true
enableTracing: true
extensionProviders:
--
2.42.0
But still the correlation was not working as expected.
|
Beta Was this translation helpful? Give feedback.
-
|
Hi @zirain I just updated my istio version to As you can see in the first two requests the |
Beta Was this translation helpful? Give feedback.
-
|
as you can see in the logs the requestId is the same for all 3 requests. Also you might notice the productinfo service does not seems to be forwarding the |
Beta Was this translation helpful? Give feedback.
-
|
I think this was the issue. it seems like this was an issue with the bookinfo version I installed. I tried to use the https://raw.githubusercontent.com/istio/istio/release-1.21/samples/bookinfo/platform/kube/bookinfo.yaml This specific version and now seems to be working
|
Beta Was this translation helpful? Give feedback.
Answer selected by
zirain
-
|
I did more looking in to the issue and below is a example log of the otel container ( below is the log of curl request to /productpage in the bookinfo example) as you can see the request-id is same but traces are not correlating as expected. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
I cannot reproduce the issue with istioctl 1.21.0. 2024-04-06T11:26:25.743Z info ResourceSpans #0
Resource SchemaURL:
Resource attributes:
-> service.name: Str(reviews.default)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope
Span #0
Trace ID : aae9c5f1956b010a2eb81f06e9b9b02c
Parent ID : c6b56b0ce6a4682a
ID : 2b1fea894c079d5f
Name : egress ratings:9080
Kind : Client
Start time : 2024-04-06 11:26:24.538278 +0000 UTC
End time : 2024-04-06 11:26:24.547496 +0000 UTC
Status code : Unset
Status message :
Attributes:
-> node_id: Str(sidecar~10.244.0.83~reviews-v2-6f9b55c5db-j272b.default~default.svc.cluster.local)
-> zone: Str()
-> guid:x-request-id: Str(0d833790-6a2b-9d3b-84d7-b3a3d5bdf06d)
-> http.url: Str(http://ratings:9080/ratings/0)
-> http.method: Str(GET)
-> downstream_cluster: Str(-)
-> user_agent: Str(curl/8.4.0)
-> http.protocol: Str(HTTP/1.1)
-> peer.address: Str(10.244.0.83)
-> request_size: Str(0)
-> response_size: Str(48)
-> component: Str(proxy)
-> upstream_cluster: Str(outbound|9080||ratings.default.svc.cluster.local)
-> upstream_cluster.name: Str(outbound|9080||ratings.default.svc.cluster.local)
-> http.status_code: Str(200)
-> response_flags: Str(-)
-> istio.cluster_id: Str(Kubernetes)
-> istio.canonical_revision: Str(v2)
-> istio.namespace: Str(default)
-> istio.mesh_id: Str(cluster.local)
-> istio.canonical_service: Str(reviews)
Span #1
Trace ID : aae9c5f1956b010a2eb81f06e9b9b02c
Parent ID : 22ceceab09faf27d
ID : c6b56b0ce6a4682a
Name : ingress
Kind : Server
Start time : 2024-04-06 11:26:24.516169 +0000 UTC
End time : 2024-04-06 11:26:24.553054 +0000 UTC
Status code : Unset
Status message :
Attributes:
-> node_id: Str(sidecar~10.244.0.83~reviews-v2-6f9b55c5db-j272b.default~default.svc.cluster.local)
-> zone: Str()
-> guid:x-request-id: Str(0d833790-6a2b-9d3b-84d7-b3a3d5bdf06d)
-> http.url: Str(http://reviews:9080/reviews/0)
-> http.method: Str(GET)
-> downstream_cluster: Str(-)
-> user_agent: Str(curl/8.4.0)
-> http.protocol: Str(HTTP/1.1)
-> peer.address: Str(10.244.0.86)
-> request_size: Str(0)
-> response_size: Str(442)
-> component: Str(proxy)
-> upstream_cluster: Str(inbound|9080||)
-> upstream_cluster.name: Str(inbound|9080||)
-> http.status_code: Str(200)
-> response_flags: Str(-)
-> istio.canonical_service: Str(reviews)
-> istio.namespace: Str(default)
-> istio.canonical_revision: Str(v2)
-> istio.cluster_id: Str(Kubernetes)
-> istio.mesh_id: Str(cluster.local)
ResourceSpans #1
Resource SchemaURL:
Resource attributes:
-> service.name: Str(istio-ingressgateway.istio-system)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope
Span #0
Trace ID : aae9c5f1956b010a2eb81f06e9b9b02c
Parent ID :
ID : 453d3a94b60752c1
Name : egress 172.18.255.200
Kind : Client
Start time : 2024-04-06 11:26:24.491089 +0000 UTC
End time : 2024-04-06 11:26:24.557504 +0000 UTC
Status code : Unset
Status message :
Attributes:
-> node_id: Str(router~10.244.0.80~istio-ingressgateway-7f648ccf55-ht2pt.istio-system~istio-system.svc.cluster.local)
-> zone: Str()
-> guid:x-request-id: Str(0d833790-6a2b-9d3b-84d7-b3a3d5bdf06d)
-> http.url: Str(http://172.18.255.200/productpage)
-> http.method: Str(GET)
-> downstream_cluster: Str(-)
-> user_agent: Str(curl/8.4.0)
-> http.protocol: Str(HTTP/1.1)
-> peer.address: Str(10.244.0.1)
-> request_size: Str(0)
-> response_size: Str(5294)
-> component: Str(proxy)
-> upstream_cluster: Str(outbound|9080||productpage.default.svc.cluster.local)
-> upstream_cluster.name: Str(outbound|9080||productpage.default.svc.cluster.local)
-> http.status_code: Str(200)
-> response_flags: Str(-)
-> istio.canonical_service: Str(istio-ingressgateway)
-> istio.namespace: Str(istio-system)
-> istio.mesh_id: Str(cluster.local)
-> istio.cluster_id: Str(Kubernetes)
-> istio.canonical_revision: Str(latest)
ResourceSpans #2
Resource SchemaURL:
Resource attributes:
-> service.name: Str(ratings.default)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope
Span #0
Trace ID : aae9c5f1956b010a2eb81f06e9b9b02c
Parent ID : 2b1fea894c079d5f
ID : d106f6b45f74ab84
Name : ingress
Kind : Server
Start time : 2024-04-06 11:26:24.540067 +0000 UTC
End time : 2024-04-06 11:26:24.545431 +0000 UTC
Status code : Unset
Status message :
Attributes:
-> node_id: Str(sidecar~10.244.0.81~ratings-v1-7c4bbf97db-8pcxp.default~default.svc.cluster.local)
-> zone: Str()
-> guid:x-request-id: Str(0d833790-6a2b-9d3b-84d7-b3a3d5bdf06d)
-> http.url: Str(http://ratings:9080/ratings/0)
-> http.method: Str(GET)
-> downstream_cluster: Str(-)
-> user_agent: Str(curl/8.4.0)
-> http.protocol: Str(HTTP/1.1)
-> peer.address: Str(10.244.0.83)
-> request_size: Str(0)
-> response_size: Str(48)
-> component: Str(proxy)
-> upstream_cluster: Str(inbound|9080||)
-> upstream_cluster.name: Str(inbound|9080||)
-> http.status_code: Str(200)
-> response_flags: Str(-)
-> istio.namespace: Str(default)
-> istio.mesh_id: Str(cluster.local)
-> istio.canonical_service: Str(ratings)
-> istio.cluster_id: Str(Kubernetes)
-> istio.canonical_revision: Str(v1)
ResourceSpans #3
Resource SchemaURL:
Resource attributes:
-> service.name: Str(details.default)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope
Span #0
Trace ID : aae9c5f1956b010a2eb81f06e9b9b02c
Parent ID : eff63945e745225b
ID : 9195eeefb54c76c7
Name : ingress
Kind : Server
Start time : 2024-04-06 11:26:24.504555 +0000 UTC
End time : 2024-04-06 11:26:24.510257 +0000 UTC
Status code : Unset
Status message :
Attributes:
-> node_id: Str(sidecar~10.244.0.82~details-v1-cf74bb974-k4ds8.default~default.svc.cluster.local)
-> zone: Str()
-> guid:x-request-id: Str(0d833790-6a2b-9d3b-84d7-b3a3d5bdf06d)
-> http.url: Str(http://details:9080/details/0)
-> http.method: Str(GET)
-> downstream_cluster: Str(-)
-> user_agent: Str(curl/8.4.0)
-> http.protocol: Str(HTTP/1.1)
-> peer.address: Str(10.244.0.86)
-> request_size: Str(0)
-> response_size: Str(178)
-> component: Str(proxy)
-> upstream_cluster: Str(inbound|9080||)
-> upstream_cluster.name: Str(inbound|9080||)
-> http.status_code: Str(200)
-> response_flags: Str(-)
-> istio.canonical_service: Str(details)
-> istio.mesh_id: Str(cluster.local)
-> istio.namespace: Str(default)
-> istio.canonical_revision: Str(v1)
-> istio.cluster_id: Str(Kubernetes)
ResourceSpans #4
Resource SchemaURL:
Resource attributes:
-> service.name: Str(productpage.default)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope
Span #0
Trace ID : aae9c5f1956b010a2eb81f06e9b9b02c
Parent ID : 2d7f990a78e29551
ID : 22ceceab09faf27d
Name : egress reviews:9080
Kind : Client
Start time : 2024-04-06 11:26:24.515831 +0000 UTC
End time : 2024-04-06 11:26:24.553382 +0000 UTC
Status code : Unset
Status message :
Attributes:
-> node_id: Str(sidecar~10.244.0.86~productpage-v1-87d54dd59-vsjfb.default~default.svc.cluster.local)
-> zone: Str()
-> guid:x-request-id: Str(0d833790-6a2b-9d3b-84d7-b3a3d5bdf06d)
-> http.url: Str(http://reviews:9080/reviews/0)
-> http.method: Str(GET)
-> downstream_cluster: Str(-)
-> user_agent: Str(curl/8.4.0)
-> http.protocol: Str(HTTP/1.1)
-> peer.address: Str(10.244.0.86)
-> request_size: Str(0)
-> response_size: Str(442)
-> component: Str(proxy)
-> upstream_cluster: Str(outbound|9080||reviews.default.svc.cluster.local)
-> upstream_cluster.name: Str(outbound|9080||reviews.default.svc.cluster.local)
-> http.status_code: Str(200)
-> response_flags: Str(-)
-> istio.cluster_id: Str(Kubernetes)
-> istio.canonical_revision: Str(v1)
-> istio.canonical_service: Str(productpage)
-> istio.mesh_id: Str(cluster.local)
-> istio.namespace: Str(default)
Span #1
Trace ID : aae9c5f1956b010a2eb81f06e9b9b02c
Parent ID : 453d3a94b60752c1
ID : 2d7f990a78e29551
Name : ingress
Kind : Server
Start time : 2024-04-06 11:26:24.493642 +0000 UTC
End time : 2024-04-06 11:26:24.555437 +0000 UTC
Status code : Unset
Status message :
Attributes:
-> node_id: Str(sidecar~10.244.0.86~productpage-v1-87d54dd59-vsjfb.default~default.svc.cluster.local)
-> zone: Str()
-> guid:x-request-id: Str(0d833790-6a2b-9d3b-84d7-b3a3d5bdf06d)
-> http.url: Str(http://172.18.255.200/productpage)
-> http.method: Str(GET)
-> downstream_cluster: Str(-)
-> user_agent: Str(curl/8.4.0)
-> http.protocol: Str(HTTP/1.1)
-> peer.address: Str(10.244.0.80)
-> request_size: Str(0)
-> response_size: Str(5294)
-> component: Str(proxy)
-> upstream_cluster: Str(inbound|9080||)
-> upstream_cluster.name: Str(inbound|9080||)
-> http.status_code: Str(200)
-> response_flags: Str(-)
-> istio.mesh_id: Str(cluster.local)
-> istio.canonical_service: Str(productpage)
-> istio.cluster_id: Str(Kubernetes)
-> istio.canonical_revision: Str(v1)
-> istio.namespace: Str(default)
ResourceSpans #5
Resource SchemaURL:
Resource attributes:
-> service.name: Str(productpage.default)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope
Span #0
Trace ID : aae9c5f1956b010a2eb81f06e9b9b02c
Parent ID : 2d7f990a78e29551
ID : eff63945e745225b
Name : egress details:9080
Kind : Client
Start time : 2024-04-06 11:26:24.502892 +0000 UTC
End time : 2024-04-06 11:26:24.512444 +0000 UTC
Status code : Unset
Status message :
Attributes:
-> node_id: Str(sidecar~10.244.0.86~productpage-v1-87d54dd59-vsjfb.default~default.svc.cluster.local)
-> zone: Str()
-> guid:x-request-id: Str(0d833790-6a2b-9d3b-84d7-b3a3d5bdf06d)
-> http.url: Str(http://details:9080/details/0)
-> http.method: Str(GET)
-> downstream_cluster: Str(-)
-> user_agent: Str(curl/8.4.0)
-> http.protocol: Str(HTTP/1.1)
-> peer.address: Str(10.244.0.86)
-> request_size: Str(0)
-> response_size: Str(178)
-> component: Str(proxy)
-> upstream_cluster: Str(outbound|9080||details.default.svc.cluster.local)
-> upstream_cluster.name: Str(outbound|9080||details.default.svc.cluster.local)
-> http.status_code: Str(200)
-> response_flags: Str(-)
-> istio.mesh_id: Str(cluster.local)
-> istio.canonical_service: Str(productpage)
-> istio.cluster_id: Str(Kubernetes)
-> istio.canonical_revision: Str(v1)
-> istio.namespace: Str(default)
{"kind": "exporter", "data_type": "traces", "name": "logging"} |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I think this was the issue. it seems like this was an issue with the bookinfo version I installed. I tried to use the https://raw.githubusercontent.com/istio/istio/release-1.21/samples/bookinfo/platform/kube/bookinfo.yaml
This specific version and now seems to be working