How rename or recreate metrics based on existing metrics?

[ningyougang]

I am using istio v1.17.2

I want to show below istio healthcheck metrics on grafana. e.g.

envoy_cluster_test_nyg_health_check_failure{cluster_name="outbound|80||test"} 164
envoy_cluster_test_nyg_health_check_network_failure{cluster_name="outbound|80||test"} 0
envoy_cluster_test_nyg_health_check_passive_failure{cluster_name="outbound|80||test"} 0

(I got the metrics by curl http://localhost:15020/stats/prometheus)

But above metrics includes dynamic variable(e.g. $namespace, test-nyg is the namespace)
Then, i want to rename or recreate the metric as below (e.f remove test_nyg)

envoy_cluster_health_check_failure                      
envoy_cluster_health_check_network_failure
envoy_cluster_health_check_passive_failure

In old isito version: 1.3, there has kind: handler, kind: instance, kind: rule to recreate custom metric based on existing metric: https://istio.io/v1.3/docs/reference/config/policy-and-telemetry/mixer-overview/

But unfortunately, in current 1.17.2, above three object does not exist on 1.17.2: https://github.com/istio/istio/blob/1.17.2/manifests/charts/base/crds/crd-all.gen.yaml

Has any idea of how to implement it in istio v1.17.2?

Btw, i did not find the source code of how to generate the metrics in istio source codes or istio-proxy(envoy) source codes. someone can point me where can i found the source code?

[zirain]

mixer is removed for a long time.
you can try https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/metrics/v3/stats.proto#config-metrics-v3-tagspecifier with a custom bootstrap yaml

[ningyougang]

@zirain thanks for your info.

It seems that your document is add new tag on existing metric, but i want recreate a new metric based on existing metric
original metric:

envoy_cluster_test_nyg_health_check_failure{cluster_name="outbound|80||test"} 164

my want new metric:

envoy_cluster_health_check_failure{cluster_name="outbound|80||test"} 164

removed _test_nyg

[zirain]

you can try it, I recall it's what you need.

[ningyougang]

@zirain

It seems the document https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/metrics/v3/stats.proto#config-metrics-v3-tagspecifier is just add tag on existing metric rather than add new metric, can you share me the detail demo or detail configuration.

Thanks for your help.

[ningyougang]

@zirain

I solved by below workaround

• metric:

  sum(increase({name=~"envoy_cluster_.*_health_check_failure", istio="istio-ingressgateway-${service}"}[1m])) by (cluster, cluster_name)

• Legend

  {{cluster}}-{{cluster_name}}

• panel title

  healthcheck failure - ${service}

I will close this discussion, anyway, thanks for your help.