How to use grafana cloud and its prometheus with istio and kiali? #7121
Replies: 1 comment
-
Kiali does not have access (in fact, cannot even DNS resolve) the host
That looks like Kiali is reaching Prometheus but can't obtain the config - its getting a 404. This sounds like an issue similar with using a Thanos proxy - in that case, you tell Kiali "thanos is enabled" and you also directly tell Kiali what the prometheus scrape interval and retention period is in the Kiali configuration. See https://kiali.io/docs/configuration/kialis.kiali.io/#.spec.external_services.prometheus.thanos_proxy |
Beta Was this translation helpful? Give feedback.
-
Hello Team,
I've added all steps which I took to install and configure istio, can you please correct me if I'm missing anything:
Step 1. I enabled port 15017 and 15012 ingress for istio in my eks cluster security group.
Step2. I'm installing istio-base, istiod and istio-ingress using following terraform code:
`resource "kubectl_manifest" "istio_namespace" {
yaml_body = <<YAML
apiVersion: v1
kind: Namespace
metadata:
labels:
name: istio-system
name: istio-system
YAML
}
resource "helm_release" "istio-base" {
repository = "https://istio-release.storage.googleapis.com/charts"
chart = "base"
name = "istio-base"
namespace = "istio-system"
version = "1.20.2"
timeout = 120
cleanup_on_fail = true
force_update = false
depends_on = [kubectl_manifest.istio_namespace]
}
resource "helm_release" "istiod" {
repository = "https://istio-release.storage.googleapis.com/charts"
chart = "istiod"
name = "istiod"
namespace = "istio-system"
version = "1.20.2"
timeout = 120
cleanup_on_fail = true
force_update = false
set {
name = "meshConfig.accessLogFile"
value = "/dev/stdout"
}
depends_on = [helm_release.istio-base]
}
resource "kubectl_manifest" "istio_ingress_namespace" {
yaml_body = <<YAML
apiVersion: v1
kind: Namespace
metadata:
labels:
istio-injection: enabled
name: istio-ingress
YAML
}
resource "helm_release" "istio-ingress" {
repository = "https://istio-release.storage.googleapis.com/charts"
chart = "gateway"
name = "istio-ingress"
namespace = "istio-ingress"
version = "1.20.2"
timeout = 500
cleanup_on_fail = true
force_update = false
depends_on = [helm_release.istiod, kubectl_manifest.istio_ingress_namespace]
}`
Step 3. I added following scrape_configs to push metrics to prometheus endpoint provided by grafana cloud:
configs: - name: integrations remote_write: - url: var.prom_endpoint basic_auth: username: var.username password: var.cloud_token scrape_configs: - job_name: 'pilot' kubernetes_sd_configs: - role: endpoints namespaces: names: - istio-system relabel_configs: - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: istio-pilot;http-monitoring - job_name: 'istiod' kubernetes_sd_configs: - role: endpoints namespaces: names: - istio-system relabel_configs: - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: istiod;http-monitoring - job_name: 'envoy-stats' metrics_path: /stats/prometheus kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_container_port_name] action: keep regex: '.*-envoy-prom' - job_name: 'istio-prometheus' honor_labels: true metrics_path: '/federate' kubernetes_sd_configs: - role: pod namespaces: names: ['istio-system'] metric_relabel_configs: - source_labels: [__name__] regex: 'workload:(.*)' target_label: __name__ action: replace params: 'match[]': - '{__name__=~"workload:(.*)"}' - '{__name__=~"pilot(.*)"}' - job_name: kubernetes-pods metric_relabel_configs: - action: drop source_labels: [__name__] regex: istio_agent_.*|istiod_.*|istio_build|citadel_.*|galley_.*|pilot_[^p].*|envoy_cluster_[^u].*|envoy_cluster_update.*|envoy_listener_[^dh].*|envoy_server_[^mu].*|envoy_wasm_.* - action: labeldrop regex: chart|destination_app|destination_version|heritage|.*operator.*|istio.*|release|security_istio_io_.*|service_istio_io_.*|sidecar_istio_io_inject|source_app|source_version
Step 4. Next I installed kiali using following step:
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/kiali.yaml
Step 5. I updated kiali configmap to use prometheus endpoint provided by grafana cloud to read the metrics:
external_services: prometheus: url: https://prometheus-xxx-xx-xxxxx-xx-xxxx-x.grafana.net/api/prom auth: type: basic username: xxxxxxx password: xxxxxxxx
After I access kiali I still see following errors:
Kiali logs:
2024-02-15T06:48:20Z INF Initializing Kiali Cache 2024-02-15T06:48:20Z INF Adding a RegistryRefreshHandler 2024-02-15T06:48:20Z INF [Kiali Cache] Waiting for cluster-scoped cache to sync 2024-02-15T06:48:21Z INF [Kiali Cache] Started 2024-02-15T06:48:21Z INF [Kiali Cache] Kube cache is active for cluster: [Kubernetes] 2024-02-15T06:48:21Z INF Server endpoint will start at [:20001/kiali] 2024-02-15T06:48:21Z INF Server endpoint will serve static content from [/opt/kiali/console] 2024-02-15T06:48:21Z INF Starting Metrics Server on [:9090] 2024-02-15T06:49:00Z INF [Prom Cache] Enabled 2024-02-15T06:49:00Z ERR Failed to fetch Prometheus configuration: client_error: client error: 404 2024-02-15T06:49:00Z ERR Failed to fetch Prometheus flags: client_error: client error: 404 2024-02-15T06:49:00Z ERR grafana URL is not set in Kiali configuration 2024-02-15T06:49:00Z INF Create jaeger GRPC client tracing.istio-system:16685 2024-02-15T06:49:00Z ERR Error fetching availability of the tracing service: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tracing.istio-system on 172.20.0.10:53: no such host" 2024-02-15T06:49:01Z ERR Error fetching availability of the tracing service: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tracing.istio-system on 172.20.0.10:53: no such host" 2024-02-15T06:49:01Z ERR Error while fetching app health: client_error: client error: 401 2024-02-15T06:49:01Z ERR Error while fetching app health: client_error: client error: 401 2024-02-15T06:49:27Z ERR Error fetching availability of the tracing service: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tracing.istio-system on 172.20.0.10:53: no such host" 2024-02-15T06:50:27Z ERR Error fetching availability of the tracing service: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tracing.istio-system on 172.20.0.10:53: no such host" 2024-02-15T06:51:24Z ERR : client_error: client error: 401 2024-02-15T06:51:24Z ERR client_error: client error: 401 2024-02-15T06:51:26Z ERR Error fetching availability of the tracing service: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tracing.istio-system on 172.20.0.10:53: no such host" 2024-02-15T06:51:26Z ERR Error while fetching app health: client_error: client error: 401 2024-02-15T06:51:26Z ERR Error while fetching app health: client_error: client error: 401 2024-02-15T06:52:26Z ERR Error fetching availability of the tracing service: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tracing.istio-system on 172.20.0.10:53: no such host" 2024-02-15T06:52:26Z ERR Error while fetching app health: client_error: client error: 401 2024-02-15T06:52:26Z ERR Error while fetching app health: client_error: client error: 401 2024-02-15T06:52:27Z ERR Error fetching availability of the tracing service: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tracing.istio-system on 172.20.0.10:53: no such host" 2024-02-15T06:53:26Z ERR Error fetching availability of the tracing service: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tracing.istio-system on 172.20.0.10:53: no such host" 2024-02-15T06:53:26Z ERR Error while fetching app health: client_error: client error: 401 2024-02-15T06:53:26Z ERR Error while fetching app health: client_error: client error: 401 2024-02-15T06:53:26Z ERR Error fetching availability of the tracing service: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tracing.istio-system on 172.20.0.10:53: no such host" 2024-02-15T06:54:26Z ERR Error fetching availability of the tracing service: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tracing.istio-system on 172.20.0.10:53: no such host" 2024-02-15T06:54:26Z ERR Error while fetching app health: client_error: client error: 401 2024-02-15T06:54:26Z ERR Error while fetching app health: client_error: client error: 401 2024-02-15T06:54:27Z ERR Error fetching availability of the tracing service: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tracing.istio-system on 172.20.0.10:53: no such host" 2024-02-15T06:55:26Z ERR Error fetching availability of the tracing service: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tracing.istio-system on 172.20.0.10:53: no such host" 2024-02-15T06:55:26Z ERR Error while fetching app health: client_error: client error: 401 2024-02-15T06:55:26Z ERR Error while fetching app health: client_error: client error: 401 2024-02-15T06:55:27Z ERR Error fetching availability of the tracing service: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tracing.istio-system on 172.20.0.10:53: no such host"
Beta Was this translation helpful? Give feedback.
All reactions