Service Discovery Demo for Prometheus on okd

Prerequisites

Default Minishift installation.

Setup

$ oc apply -f s2i-java11.yml
$ oc apply -f demo.yml
$ oc apply -f grafana
$ oc apply -f prometheus
$ oc start-build demo --from-dir=demo

Key Ideas

Create a service account (prometheus) and bind a dedicated role (e.g. podview) which is only allowed to list pods within the project. This service account must be used by the prometheus pod which is configured by the deployment config variable spec.template.spec.serviceAccount.

Pods with a Prometheus metric endpoint can be configured by utilizing Kubernetes annotations such that one Prometheus scrape job is sufficient to setup the pod as target (1, 2).

The annotations are used together with Prometheus relabel config mechanism:

relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]              # (1)
  regex: true
  action: keep
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]   # (2)
  regex: (.+)(?::\d+);(\d+)
  replacement: $1:$2
  target_label: __address__
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]                # (3)
  regex: ((?:/[^/ ]+)+)
  target_label: __metrics_path__
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_scheme] # (4)
  regex: https?
  target_label: __scheme__

1. keep only targets annotated w/ prometheus.io/scrape: "true"

2. if prometheus.io/port: "…​" annotation is present use the given port (otherwise use pods exposed port)

3. if prometheus.io/path: …​ is present use the given path (otherwise /prometheus)

4. if one of prometheus.io/scheme: http|https is present, use the given scheme (default is http)

A deployment configuration for a pod with a common Prometheus enabled Spring Boot application has these annotations:

template:
  metadata:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/path: /actuator/prometheus
      prometheus.io/port: "8080"

Links

• Sebastian Daschner Prometheus Kubernetes Discovery Blogpost

• Prometheus Kubernetes SD Config

• Prometheus Relabel Config

• Kubernetes Using RBAC Authorization