[Bug]: Operator showing error with injection sidecar

[minh-oolio]

What happened?

hello team,
I implemented sidecar injected on deployment as this document https://github.com/jaegertracing/jaeger-operator/blob/main/examples/business-application-injected-sidecar.yaml
and found error message from jaeger operator

1.6672863326932034e+09	INFO	annotation present on deployment	{"namespace": "applicationA", "deployment": "accounts"}
1.6672863326933312e+09	INFO	no suitable Jaeger instances found to inject a sidecar	{"namespace": "applicationA"}

Steps to reproduce

1. deployed jaeger and jaeger operator by helm chart with latest version (namespace jaeger)
2. implement sidecar inject on deployment of application (other namespace applicationA )
3. we are using traefik as service mesh https://traefik.io/blog/application-request-tracing-with-traefik-and-jaeger-on-kubernetes/
4. check log operator and found error message no suitable Jaeger instances found to inject a sidecar

here is services of jaeger was deployed

Expected behavior

Able to tracing appliacation api.
something like this picture

Relevant log output

Operator pod logs

1.6672916883622577e+09	INFO	no suitable Jaeger instances found to inject a sidecar	{"namespace": "applicationA"}

1.6672916883625038e+09	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-v1-deployment", "code": 200, "reason": "no suitable Jaeger instances found to inject a sidecar", "UID": "294a91d5-fe16-417e-a19c-c3ac5664599e", "allowed": true}

Screenshot

No response

Additional context

No response

Jaeger backend version

v1.37.0

SDK

No response

Pipeline

No response

Stogage backend

No response

Operating system

No response

Deployment model

kubernetes (EKS)

Deployment configs

No response

[frzifus]

Seems your deployment needs an injection. But can not find a Jaeger instance in the same namespace. In that case your sidecar doesnt know where it should send the collected data.

Have a look here:

• Inject sidecars from different namespaces #60

[minh-oolio]

hi @frzifus thanks for your response
Is there any solution for my case?
I deployed jaeger and jaeger operator in jaeger namespace and tracing application on other namespace.
thank you very much

[frzifus]

Hi @minh-oolio, from my understanding you need to define WATCH_NAMESPACES in your operator.yaml file. To make sure the right jaeger instance is picked, change your annotation value to your instance name.

kind: Deployment
metadata:
  name: myapp
  annotations:
    "sidecar.jaegertracing.io/inject": "<INSTANCE_NAME>"

On the operator side, your jaeger instance should then be found here.

[minh-oolio]

hi @frzifus thanks a lot for your suggestion
it seem it able to inject sidecar to pod
but it still trace on the console
i am concern, do i need to change jaeger-operator-jaeger to jaeger-operator-jaeger.jaeger.svc
please advise on it

k get jaegers
NAME                     STATUS    VERSION   STRATEGY   STORAGE   AGE
jaeger-operator-jaeger   Running   1.38.0    allinone   memory    3h39m

with this annotaion

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gateway
  annotations:
    sidecar.jaegertracing.io/inject: jaeger-operator-jaeger

and here is a log of jageragent injected to gateway pod

31
2022/11/08 00:46:53 maxprocs: Leaving GOMAXPROCS=2: CPU quota undefined
30
{"level":"info","ts":1667868413.1804988,"caller":"flags/service.go:119","msg":"Mounting metrics handler on admin server","route":"/metrics"}
29
{"level":"info","ts":1667868413.180543,"caller":"flags/service.go:125","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
28
{"level":"info","ts":1667868413.1809232,"caller":"flags/admin.go:129","msg":"Mounting health check on admin server","route":"/"}
27
{"level":"info","ts":1667868413.1809716,"caller":"flags/admin.go:143","msg":"Starting admin HTTP server","http-addr":":14271"}
26
{"level":"info","ts":1667868413.1809838,"caller":"flags/admin.go:121","msg":"Admin server started","http.host-port":"[::]:14271","health-status":"unavailable"}
25
{"level":"info","ts":1667868413.1815352,"caller":"grpc/builder.go:73","msg":"Agent requested insecure grpc connection to collector(s)"}
24
{"level":"info","ts":1667868413.1815639,"caller":"channelz/funcs.go:340","msg":"[core][Channel #1] Channel created","system":"grpc","grpc_log":true}
23
{"level":"info","ts":1667868413.1815975,"caller":"channelz/funcs.go:340","msg":"[core][Channel #1] original dial target is: \"dns:///jaeger-operator-jaeger-collector-headless.jaeger.svc:14250\"","system":"grpc","grpc_log":true}
22
{"level":"info","ts":1667868413.181634,"caller":"channelz/funcs.go:340","msg":"[core][Channel #1] parsed dial target is: {Scheme:dns Authority: Endpoint:jaeger-operator-jaeger-collector-headless.jaeger.svc:14250 URL:{Scheme:dns Opaque: User: Host: Path:/jaeger-operator-jaeger-collector-headless.jaeger.svc:14250 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}","system":"grpc","grpc_log":true}
21
{"level":"info","ts":1667868413.1816468,"caller":"channelz/funcs.go:340","msg":"[core][Channel #1] Channel authority set to \"jaeger-operator-jaeger-collector-headless.jaeger.svc:14250\"","system":"grpc","grpc_log":true}
20
{"level":"info","ts":1667868413.1829572,"caller":"grpc/builder.go:113","msg":"Checking connection to collector"}
19
{"level":"info","ts":1667868413.182981,"caller":"grpc/builder.go:124","msg":"Agent collector connection state change","dialTarget":"dns:///jaeger-operator-jaeger-collector-headless.jaeger.svc:14250","status":"IDLE"}
18
{"level":"info","ts":1667868413.183288,"caller":"./main.go:88","msg":"Starting agent"}
17
{"level":"info","ts":1667868413.1833262,"caller":"healthcheck/handler.go:129","msg":"Health Check state change","status":"ready"}
16
{"level":"info","ts":1667868413.1833835,"caller":"app/agent.go:69","msg":"Starting jaeger-agent HTTP server","http-port":5778}
15
{"level":"info","ts":1667868413.1956995,"caller":"channelz/funcs.go:340","msg":"[core][Channel #1] Resolver state updated: {\n  \"Addresses\": [\n    {\n      \"Addr\": \"192.168.112.106:14250\",\n      \"ServerName\": \"\",\n      \"Attributes\": null,\n      \"BalancerAttributes\": null,\n      \"Type\": 0,\n      \"Metadata\": null\n    }\n  ],\n  \"ServiceConfig\": null,\n  \"Attributes\": null\n} (resolver returned new addresses)","system":"grpc","grpc_log":true}
14
{"level":"info","ts":1667868413.1957407,"caller":"channelz/funcs.go:340","msg":"[core][Channel #1] Channel switches to new LB policy \"round_robin\"","system":"grpc","grpc_log":true}
13
{"level":"info","ts":1667868413.1957922,"caller":"grpclog/component.go:55","msg":"[balancer]base.baseBalancer: got new ClientConn state: {{[{\n  \"Addr\": \"192.168.112.106:14250\",\n  \"ServerName\": \"\",\n  \"Attributes\": null,\n  \"BalancerAttributes\": null,\n  \"Type\": 0,\n  \"Metadata\": null\n}] <nil> <nil>} <nil>}","system":"grpc","grpc_log":true}
12
{"level":"info","ts":1667868413.1958196,"caller":"channelz/funcs.go:340","msg":"[core][Channel #1 SubChannel #2] Subchannel created","system":"grpc","grpc_log":true}
11
{"level":"info","ts":1667868413.1958375,"caller":"grpclog/component.go:71","msg":"[roundrobin]roundrobinPicker: Build called with info: {map[]}","system":"grpc","grpc_log":true}
10
{"level":"info","ts":1667868413.1958477,"caller":"channelz/funcs.go:340","msg":"[core][Channel #1] Channel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
9
{"level":"info","ts":1667868413.1958697,"caller":"channelz/funcs.go:340","msg":"[core][Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
8
{"level":"info","ts":1667868413.1958842,"caller":"channelz/funcs.go:340","msg":"[core][Channel #1 SubChannel #2] Subchannel picks a new address \"192.168.112.106:14250\" to connect","system":"grpc","grpc_log":true}
7
{"level":"info","ts":1667868413.195938,"caller":"grpc/builder.go:124","msg":"Agent collector connection state change","dialTarget":"dns:///jaeger-operator-jaeger-collector-headless.jaeger.svc:14250","status":"CONNECTING"}
6
{"level":"info","ts":1667868413.1959598,"caller":"grpclog/component.go:71","msg":"[balancer]base.baseBalancer: handle SubConn state change: 0xc00036ab50, CONNECTING","system":"grpc","grpc_log":true}
5
{"level":"info","ts":1667868413.2003353,"caller":"channelz/funcs.go:340","msg":"[core][Channel #1 SubChannel #2] Subchannel Connectivity change to READY","system":"grpc","grpc_log":true}
4
{"level":"info","ts":1667868413.200361,"caller":"grpclog/component.go:71","msg":"[balancer]base.baseBalancer: handle SubConn state change: 0xc00036ab50, READY","system":"grpc","grpc_log":true}
3
{"level":"info","ts":1667868413.2003944,"caller":"grpclog/component.go:71","msg":"[roundrobin]roundrobinPicker: Build called with info: {map[0xc00036ab50:{{\n  \"Addr\": \"192.168.112.106:14250\",\n  \"ServerName\": \"\",\n  \"Attributes\": null,\n  \"BalancerAttributes\": null,\n  \"Type\": 0,\n  \"Metadata\": null\n}}]}","system":"grpc","grpc_log":true}
2
{"level":"info","ts":1667868413.2004066,"caller":"channelz/funcs.go:340","msg":"[core][Channel #1] Channel Connectivity change to READY","system":"grpc","grpc_log":true}
1
{"level":"info","ts":1667868413.2004158,"caller":"grpc/builder.go:124","msg":"Agent collector connection state change","dialTarget":"dns:///jaeger-operator-jaeger-collector-headless.jaeger.svc:14250","status":"READY"}