Multi mesh setup results in an error when fetching workloads

[InverseIntegral]

I'm running a multi-mesh / multi-cluster setup with two instances of Kiali. Whenever I access Kiali on cluster A, I get the following error:

2023-10-23T16:32:21Z INF Kiali: Version: v1.75.0, Commit: 45d72534b0556c0425f52a1488811eb372c92e43, Go: 1.20.5
2023-10-23T16:32:21Z INF Using authentication strategy [anonymous]
2023-10-23T16:32:21Z WRN Kiali auth strategy is configured for anonymous access - users will not be authenticated.
2023-10-23T16:32:21Z INF Some validation errors will be ignored [KIA1301]. If these errors do occur, they will still be logged. If you think the validation errors you see are incorrect, please report them to the Kiali team if you have not done so already and provide the details of your scenario. This will keep Kiali validations strong for the whole community.
2023-10-23T16:32:21Z INF Initializing Kiali Cache
2023-10-23T16:32:21Z INF Adding a RegistryRefreshHandler
2023-10-23T16:32:21Z INF [Kiali Cache] Waiting for cluster-scoped cache to sync
2023-10-23T16:32:21Z INF [Kiali Cache] Started
2023-10-23T16:32:21Z INF [Kiali Cache] Kube cache is active for cluster: [A]
2023-10-23T16:32:21Z INF Server endpoint will start at [:20001/kiali]
2023-10-23T16:32:21Z INF Server endpoint will serve static content from [/opt/kiali/console]
2023-10-23T16:32:21Z INF Starting Metrics Server on [:9090]
2023-10-23T16:32:26Z INF [Prom Cache] Enabled
2023-10-23T16:32:26Z ERR grafana URL is not set in Kiali configuration
2023-10-23T16:32:26Z INF Create jaeger GRPC client tracing.istio-system:16685
2023-10-23T16:32:26Z ERR This controller DeploymentRevision is assigned in a Pod and it's not properly managed
2023-10-23T16:32:26Z ERR This controller DeploymentRevision is assigned in a Pod and it's not properly managed
2023-10-23T16:32:28Z ERR This controller DeploymentRevision is assigned in a Pod and it's not properly managed
2023-10-23T16:32:28Z ERR This controller DeploymentRevision is assigned in a Pod and it's not properly managed
2023-10-23T16:32:28Z ERR This controller DeploymentRevision is assigned in a Pod and it's not properly managed
2023-10-23T16:32:28Z ERR Error fetching Workloads per namespace <namespace>: Cluster [B] is not found or is not accessible for Kiali
2023-10-23T16:32:28Z ERR Cluster [B] is not found or is not accessible for Kiali: goroutine 806 [running]:
runtime/debug.Stack()
    /opt/hostedtoolcache/go/1.20.5/x64/src/runtime/debug/stack.go:24 +0x65
github.com/kiali/kiali/handlers.handlePanic({0x23967e0, 0xc00216a810})
    /home/runner/work/kiali/kiali/handlers/graph.go:86 +0x219
panic({0x1c78aa0, 0xc002014de0})
    /opt/hostedtoolcache/go/1.20.5/x64/src/runtime/panic.go:884 +0x213
github.com/kiali/kiali/graph.CheckError(...)
    /home/runner/work/kiali/kiali/graph/util.go:38
github.com/kiali/kiali/graph/telemetry/istio/appender.getWorkloadList({0xc000f1c780, 0x10}, {0xc000f1c7b0, 0xe}, 0xc00217fb90)
    /home/runner/work/kiali/kiali/graph/telemetry/istio/appender/appender.go:345 +0x40a
github.com/kiali/kiali/graph/telemetry/istio/appender.getAppWorkloads({0xc000f1c780, 0x10}, {0xc000f1c7b0, 0xe}, {0xc000529158, 0x16}, {0xc000f1c770, 0x6}, 0xc0020da8e8?)
    /home/runner/work/kiali/kiali/graph/telemetry/istio/appender/appender.go:373 +0x14c
github.com/kiali/kiali/graph/telemetry/istio/appender.(*MeshCheckAppender).applyMeshChecks(0xc0020daaf0, 0x33b39e0?, 0x7fe7166c7010?, 0xc001631050)
    /home/runner/work/kiali/kiali/graph/telemetry/istio/appender/mesh_check.go:73 +0x254
github.com/kiali/kiali/graph/telemetry/istio/appender.MeshCheckAppender.AppendGraph({0xc00217f8f0}, 0x9?, 0xc000d3e330?, 0x12?)
    /home/runner/work/kiali/kiali/graph/telemetry/istio/appender/mesh_check.go:34 +0x3e
github.com/kiali/kiali/graph/telemetry/istio.BuildNamespacesTrafficMap({_, _}, {0xc00217f8f0, {0x0, {0xc000a726c0, 0x6, 0x6}}, 0x0, 0x1, 0xc00217f830, ...}, ...)
    /home/runner/work/kiali/kiali/graph/telemetry/istio/istio.go:80 +0x6a3
github.com/kiali/kiali/graph/api.graphNamespacesIstio({_, _}, _, _, {{0x20932cd, 0x9}, {0x208d8a0, 0x5}, {{0xc0006946b9, 0x15}, ...}, ...})
    /home/runner/work/kiali/kiali/graph/api/api.go:52 +0x126
github.com/kiali/kiali/graph/api.GraphNamespaces({_, _}, _, {{0x20932cd, 0x9}, {0x208d8a0, 0x5}, {{0xc0006946b9, 0x15}, {0xdf8475800, ...}}, ...})
    /home/runner/work/kiali/kiali/graph/api/api.go:33 +0x2fb
github.com/kiali/kiali/handlers.GraphNamespaces({0x23967e0, 0xc00216a810}, 0xc001f73000)
    /home/runner/work/kiali/kiali/handlers/graph.go:51 +0x165
net/http.HandlerFunc.ServeHTTP(0x209b862?, {0x23967e0?, 0xc00216a810?}, 0x7fe716873b00?)
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2122 +0x2f
github.com/kiali/kiali/routing.metricHandler.func1({0x23a55a0?, 0xc0007267e0}, 0xc001f10a01?)
    /home/runner/work/kiali/kiali/routing/router.go:145 +0x13b
net/http.HandlerFunc.ServeHTTP(0x23a6a38?, {0x23a55a0?, 0xc0007267e0?}, 0xc001f10a80?)
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2122 +0x2f
github.com/kiali/kiali/handlers.AuthenticationHandler.Handle.func1({0x23a55a0, 0xc0007267e0}, 0xc001f72e00)
    /home/runner/work/kiali/kiali/handlers/authentication.go:79 +0x3fd
net/http.HandlerFunc.ServeHTTP(0x23a6a38?, {0x23a55a0?, 0xc0007267e0?}, 0x2379c18?)
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2122 +0x2f
github.com/kiali/kiali/server.plainHttpMiddleware.func1({0x23a55a0?, 0xc0007267e0?}, 0xc00217f650?)
    /home/runner/work/kiali/kiali/server/server.go:157 +0x65
net/http.HandlerFunc.ServeHTTP(0xc001f72d00?, {0x23a55a0?, 0xc0007267e0?}, 0x2?)
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2122 +0x2f
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00052c300, {0x23a55a0, 0xc0007267e0}, 0xc001f72c00)
    /home/runner/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1cf
github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1({0x23a5450, 0xc0010be0e0}, 0x0?)
    /home/runner/go/pkg/mod/github.com/!n!y!times/gziphandler@v1.1.1/gzip.go:336 +0x24e
net/http.HandlerFunc.ServeHTTP(0xc001394480?, {0x23a5450?, 0xc0010be0e0?}, 0x40d92a?)
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2122 +0x2f
net/http.(*ServeMux).ServeHTTP(0xc000694660?, {0x23a5450, 0xc0010be0e0}, 0xc001f72c00)
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2500 +0x149
net/http.serverHandler.ServeHTTP({0xc001dab8f0?}, {0x23a5450, 0xc0010be0e0}, 0xc001f72c00)
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2936 +0x316
net/http.(*conn).serve(0xc00088b3b0, {0x23a6a38, 0xc000a0d8c0})
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:1995 +0x612
created by net/http.(*Server).Serve
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:3089 +0x5ed

I install Kiali as described here by applying the addon resources. I've tried to disable the clustering feature since I'm currently not interested in having a single Kiali. Do you have an idea how I could resolve this issue? Thank you in advance.

[jshaughn]

Hmm, this is interesting. So you have multi-cluster traffic but don't want to unify metrics, just have kiali installed on A and see what it can see? It should be able to see traffic going to B, but not be able to get any additional information about those remote workloads. Maybe in recent changes to support MC we've broken something in this scenario... Will have to look more closely...

[InverseIntegral]

So you have multi-cluster traffic but don't want to unify metrics, just have kiali installed on A and see what it can see?

Correct, that's what I'm currently trying to get. But for some reason it tries to contact cluster B which is not reachable for most traffic. Moreover, cluster B is not just a different cluster but also a different mesh.

[nrfox]

I'm guessing you've got 2 separate kialis sharing one prometheus? Kiali should probably be more defensive about trying to fetch workloads based off the cluster field in metrics. Can you try configuring query_scope on each Kiali to limit metrics queries to each particular cluster?

Something like:

query_scope:
  cluster: "cluster-A"

or:

query_scope:
  destination_cluster: "cluster-A"
  source_cluster: "cluster-A"

[jshaughn]

@InverseIntegral That's a good point, if you are unifying metrics into a single Prometheus but are pointing multiple Kiali instances at that Prom instance, you want to limit the scope of each Kiali to what it's associated Kube cluster knows about. But, @nrfox, that second example is not exactly how query_scope is used. The first example is more correct; query_scope is not applied on the standard Istio metric attributes, like you show above in the second example. The idea is that you must ADD an attribute reflecting the reporting cluster, and then scope to that new attribute. So, if you have A and B, the scraping on A should add somthing like cluster: "A" and the scraping on B should add cluster: "B". And then you set things like in your first example: On the A Kiali:

query_scope:
  cluster: "A"

And on the B Kiali:

query_scope:
  cluster: "B"

[InverseIntegral]

@nrfox thank you for the prompt answer. I'm currently using two separate Kiali each with their own Prometheus. Changing the query_scope probably won't make a difference in this case but I still want to give it a shot.

@jshaughn I understood the basic idea. But I'm wondering about this:

The idea is that you must ADD an attribute reflecting the reporting cluster, and then scope to that new attribute

How would I add the cluster attribute on the reporting cluster? Or is the cluster attribute something that is set by default?

[jshaughn]

I'm currently using two separate Kiali each with their own Prometheus. Changing the query_scope probably won't make a difference

You are right, it won't make a difference. It's only needed when you have unified metrics, it's a way of proding some level of tenancy to the metrics.

How would I add the cluster attribute on the reporting cluster? Or is the cluster attribute something that is set by default?

This would probably make a good blog, I don't know the details off the top of my head. Basically, you want to update the Prom scrape config to apply a label combination for everything that gets scraped. So, when metrics are ingested they get the configured, static attribute-value pair applied to every time-series. It increases storage slightly but does not affect cardinality. And then you can query on that setting to discriminate.

[jshaughn]

Correct, that's what I'm currently trying to get. But for some reason it tries to contact cluster B which is not reachable for most traffic. Moreover, cluster B is not just a different cluster but also a different mesh.

I think the fact that you have separate Prometheus means that we have a regression or bug. We will look into this. I'm going to convert this discussion into a Bug issue.

[minuyim]

Hello. A similar issue was found in DeadNodeAppender in the same environment.(dead_node.go:106)
The difference is that our deployment is deployed without service and uses service entry.

kiali version is v1.78.0 and istio version is v1.20.0.

[jshaughn]

@minuyim I see that the original fix associated with this discussion did not touch DeadNodeAppender. Not sure if it's exactly the same problem, but maybe. Can you please open an issue with any details you can provide, thanks.

[minuyim]

#7179