[Bug]: operator clashes with calico-system namespace

[ViliusS]

What happened?

I have a deployed namespaced jaeger-operator to standard Kubernetes 1.23 cluster. Cluster networking runs on Calico which was deployed using pretty standard Tigera operator.

Every few minutes Jaeger operator pod crashes with errors pointing to calico-system namespace. My WATCH_NAMESPACE variable is only set to watch one other namespace with my custom application. Nothing points to calico-system.

Full logs and
deployment YAMLs attached. The deployment scripts will deploy Jaeger operator itself into jaeger-operator namespace (I've made changes from observability), and it will also deploy needed role to the custom myapp namespace.

Steps to reproduce

1. Install Calico using Tigera operator on your Kubernetes cluster.
2. Install namespaced Jaeger operator.
3. Watch the logs.

Expected behavior

Jaeger operator should not crash or have anything request a Calico namespace.

Relevant log output

1.668170281347599e+09	INFO	verify deployment	{"namespace": "calico-system"}
W1111 12:38:01.359861       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.25.3/tools/cache/reflector.go:169: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:jaeger-operator:jaeger-operator" cannot list resource "namespaces" in API group "" at the cluster scope
E1111 12:38:01.359912       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.25.3/tools/cache/reflector.go:169: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:jaeger-operator:jaeger-operator" cannot list resource "namespaces" in API group "" at the cluster scope
1.668170291340387e+09	ERROR	failed to get the namespace for the deployment, skipping injection based on namespace annotation	{"namespace": "calico-system", "error": "Timeout: failed waiting for *v1.Namespace Informer to sync"}
github.com/jaegertracing/jaeger-operator/controllers/appsv1.(*deploymentInterceptor).Handle
	/workspace/controllers/appsv1/deployment_webhook.go:84
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/webhook/admission/webhook.go:169
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/webhook/admission/http.go:98
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/go/pkg/mod/github.com/prometheus/client_golang@v1.13.1/prometheus/promhttp/instrument_server.go:56
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/go/pkg/mod/github.com/prometheus/client_golang@v1.13.1/prometheus/promhttp/instrument_server.go:142
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/go/pkg/mod/github.com/prometheus/client_golang@v1.13.1/prometheus/promhttp/instrument_server.go:104
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/usr/local/go/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/usr/local/go/src/net/http/server.go:2916
net/http.(*conn).serve
	/usr/local/go/src/net/http/server.go:1966

Screenshot

No response

Additional context

Deploying Jaeger operator cluster-wide doesn't crash the pod, but I still see calico-system namespace in the logs:

1.6681757246274092e+09 INFO found the horizontalpodautoscalers api in autoscaling/v2
1.6681757246274357e+09 INFO detecting orphaned deployments.
1.6681757296270556e+09 INFO Determining whether we should enable the Elasticsearch Operator integration
1.6681757296270943e+09 INFO Determining whether we should enable the Kafka Operator integration
1.6681757296289122e+09 INFO found the cronjobs api in batch/v1
1.6681757296306121e+09 INFO found the horizontalpodautoscalers api in autoscaling/v2
1.668175729630688e+09 INFO detecting orphaned deployments.
1.66817573141142e+09 INFO verify deployment {"namespace": "calico-system"}
1.6681757315263824e+09 INFO verify deployment {"namespace": "calico-system"}
1.6681757346238828e+09 INFO Determining whether we should enable the Elasticsearch Operator integration
1.6681757346239202e+09 INFO Determining whether we should enable the Kafka Operator integration
1.6681757346257246e+09 INFO found the cronjobs api in batch/v1
1.6681757346268795e+09 INFO found the horizontalpodautoscalers api in autoscaling/v2
1.6681757346269002e+09 INFO detecting orphaned deployments.

Jaeger backend version

No response

SDK

No response

Pipeline

No response

Stogage backend

No response

Operating system

No response

Deployment model

No response

Deployment configs

No response

[iblancasa]

Taking a look to the logs and your CRs, I think something is not properly set. I cannot see this in your CRs:

                - name: WATCH_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.annotations['olm.targetNamespaces']

This is set in the ./bundle/manifests/jaeger-operator.clusterserviceversion.yaml file, that is the one used to deploy the operator. So, the operator is trying to reach all the namespaces in your cluster.

Also, it is not a crash. It shows why is not doing able to watch that namespace. And, after that, a timeout is raised.

[ViliusS]

There is:

        envFrom:
        - configMapRef:
            name: jaeger-config

which points to ConfigMap containing only WATCH_NAMESPACE: myappnamespace