docs/prometheus/

[giscus[bot]]

docs/prometheus/

How to deploy kuberentes cluster monitoring solution based on Prometheus. Installation based on Prometheus Operator using kube-prometheus-stack project.

https://picluster.ricsanfre.com/docs/prometheus/

[i5Js]

I have had to uncheck the client authentication in the Client configuration, otherwise I have had an error after login: code_to_token_error invalid client credentials

[i5Js]

FIXED! Finally! I found it checking the env variables inside the grafana pod:

kube-prometheus-stack-grafana-8f748478-5z7jr:/usr/share/grafana$ env | grep GF_
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=<secret_uncoded>
GF_PATHS_HOME=/usr/share/grafana
GF_PATHS_PROVISIONING=/etc/grafana/provisioning
GF_SECURITY_ADMIN_PASSWORD=prom-operator
GF_SECURITY_ADMIN_USER=admin
GF_PATHS_DATA=/var/lib/grafana/
GF_PATHS_LOGS=/var/log/grafana
GF_PATHS_PLUGINS=/var/lib/grafana/plugins
GF_PATHS_CONFIG=/etc/grafana/grafana.ini

Then I have realized there was one miss: GF_AUTH_GENERIC_OAUTH_CLIENT_ID

And that explains why always had userId="null". So I modified the secret:

apiVersion: v1
data:
  GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: <clientsecret_base64>
  GF_AUTH_GENERIC_OAUTH_CLIENT_ID: <clientid_base64>
kind: Secret
metadata:
  creationTimestamp: null
  name: grafana-env-secret
  namespace: monitoring

And restarted the pod: kubectl -n monitoring rollout restart deployment kube-prometheus-stack-grafana

And finally, checked the envs:

kube-prometheus-stack-grafana-8f748478-5z7jr:/usr/share/grafana$ env | grep GF_
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=<secret_uncoded>
**GF_AUTH_GENERIC_OAUTH_CLIENT_ID**=grafana-oauth
GF_PATHS_HOME=/usr/share/grafana
GF_PATHS_PROVISIONING=/etc/grafana/provisioning
GF_SECURITY_ADMIN_PASSWORD=prom-operator
GF_SECURITY_ADMIN_USER=admin
GF_PATHS_DATA=/var/lib/grafana/
GF_PATHS_LOGS=/var/log/grafana
GF_PATHS_PLUGINS=/var/lib/grafana/plugins
GF_PATHS_CONFIG=/etc/grafana/grafana.ini

And boom! working:

logger=context userId=0 orgId=0 uname= t=2024-04-19T12:40:28.158783866Z level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=10.244.0.0 time_ms=0 duration=239.164µs size=345 referer= handler=/login/:name status_source=server
logger=oauth.generic_oauth t=2024-04-19T12:40:28.259028574Z level=debug msg="Getting user info"
logger=oauth.generic_oauth t=2024-04-19T12:40:28.259125406Z level=debug msg="Extracting user info from OAuth token"
logger=oauth.generic_oauth t=2024-04-19T12:40:28.259833854Z level=debug msg="Received id_token" raw_json="{\"exp\":1713530731,\"iat\":1713530431,\"auth_time\":1713529886,\"jti\":\"e74d1c01-a1ac-4fd8-ac49-0362be7c2021\",\"iss\":\"https://sso.home.domain.com/realms/home\",\"aud\":\"grafana-oauth\",\"sub\":\"d67a175d-d4ae-4d62-9070-09aa309b2ee0\",\"typ\":\"ID\",\"azp\":\"grafana-oauth\",\"session_state\":\"8dfd088e-7055-4c82-aa87-e1bc06868814\",\"at_hash\":\"mcxdogCSuT-NeDvnEoeExw\",\"acr\":\"0\",\"sid\":\"8dfd088e-7055-4c82-aa87-e1bc06868814\",\"email_verified\":true,\"roles\":[\"editor\",\"viewer\",\"offline_access\",\"admin\",\"default-roles-home.javisalas.com\",\"uma_authorization\"],\"name\":\"Grafana Admin\",\"preferred_username\":\"grafanadmin\",\"given_name\":\"Grafana\",\"family_name\":\"Admin\",\"email\":\"grafana@domain.com\"}" data="Name: Grafana Admin, Displayname: , Login: , Username: , Email: grafana@domain.com, Upn: , Attributes: map[]"

[ricsanfre]

Hi Javi,

In my Prometheus packaged application for ArgoCD, I am specifying both environment variables GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET and GF_AUTH_GENERIC_OAUTH_CLIENT_ID. That way I can configure not only the password but the client_id from a external secret defined in my Secret Manager. See External Secret definition https://github.com/ricsanfre/pi-cluster/blob/master/argocd/system/monitoring/templates/grafana-env-externalsecret.yaml

When I did manual installation tests specifying GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET as environment variable, adding GF_AUTH_GENERIC_OAUTH_CLIENT_ID was not a requirement,. client_id could be specified in the grafana.ini section within the values.yaml.

grafana:
  grafana.ini:
      # SSO configuration
      auth.generic_oauth:
        enabled: true
        name: Keycloak-OAuth
        allow_sign_up: true
        client_id: grafana

According to Grafana's documentation the "GF_" environment variables are only overriding specific configuration parameters you have in the grafana.ini file. So it should not be mandatory to add both environment variables if the client_id is part of the grafana.ini.

Is that not working? Is it needed to a both as Grafana's environment variables in the external secret?
If it is mandatory to add both variables, I will modify the documentation.

Thanks
Regards
Ricardo

[i5Js]

Hello Ricardo,

I made a new test: I have removed the client_id from the values files, and it's still working, so, it's not needed.

Happy to help.

Regards

Javier.

[ricsanfre]

Hi Javi,

What I wanted to test is the opposite. Keeping client_id in the values file and removing GF_AUTH_GENERIC_OAUTH_CLIENT_ID from the secret. If that was your initial set-up, and it was not working, I will modify the documentation.

Regards
Ricardo

[i5Js]

Hi Ricardo,

That was my initial setup, and it was not working. Perhaps, they have changed something and it's something different than client_id, but trust me, it was not working. Two weeks on this!! :)

Cheers

Javi

[i5Js]

The Nginx Grafana dashboard doesn't work "out of the box", we need to adjust nginx settings and promethus settings described here:

https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/monitoring.md

[ricsanfre]

Hi @i5Js,

I am not finding any issue with my nginx grafana dashboard.
Prometheus ServiceMonitor resource is created by default by nginx helm chart using this configuration:

controller:
  # Enabling Prometheus metrics and Service Monitoring
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true

My nginx dashboard works fine without any additional prometheus configuration applied to nginx

Regards
Ricardo

[benbourner]

The Ingress for the grafana dashboard rewrites the URL to remove /grafana, but grafana is configured to be served from /grafana, so the ingress doesn't work. Commenting out these lines in the grafana ingress solves it:

    # Rewrite target
    # nginx.ingress.kubernetes.io/use-regex: "true"
    # nginx.ingress.kubernetes.io/rewrite-target: /$1

Also no further need for the regex group match on the path:

    path: /grafana

[benbourner]

Also the metrics is being queried from Prometheus on the wrong path, /metrics instead of /grafana/metrics, resulting in a failed scraper but I have not yet figured out how to fix that 😊

[benbourner]

Solved, needs to be added so:

 # ServiceMonitor label and job relabel
  serviceMonitor:
    path: /grafana/metrics
    labels:
      release: kube-prometheus-stack
    relabelings:
 ...

[ricsanfre]

Hi @benbourner,

In my cluster, I have the configuration as it is stated in the guide and everything works fine. Grafana endpoint is reachable at https://monitoring.picluster.ricsanfre.com/grafana/ and grafana metrics also are available at /metrics endpoint.

Have you check whether you have this grafana configuration to make it can work behind a proxy.

grafana:
  # Configuring grafana to run behind a proxy under a subpath
  grafana.ini:
    server:
      domain: monitoring.picluster.ricsanfre.com
      root_url: "%(protocol)s://%(domain)s:%(http_port)s/grafana/"
      # Running Grafana behind proxy rewrite path
      # https://grafana.com/tutorials/run-grafana-behind-a-proxy/
      serve_from_sub_path: true

My working ingress resource is

# Ingress Grafana
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-grafana
  namespace: monitoring
  annotations:
    # Linkerd configuration. Configure Service as Upstream
    nginx.ingress.kubernetes.io/service-upstream: "true"
    # Rewrite target
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    # Enable cert-manager to create automatically the SSL certificate and store in Secret
    cert-manager.io/cluster-issuer: letsencrypt-issuer
    cert-manager.io/common-name: monitoring.picluster.ricsanfre.com

spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - monitoring.picluster.ricsanfre.com
      secretName: monitoring-tls
  rules:
    - host: monitoring.picluster.ricsanfre.com
      http:
        paths:
          - path: /grafana/(.*)
            pathType: Prefix
            backend:
              service:
                name: kube-prometheus-stack-grafana
                port:
                  number: 80

Please, let me know if you have followed all the steps in the guide and if you still have issues.

Regards
Ricardo