OpenShift Prometheus Pushgateway

Table of Contents

• Overview
• License
• Bill of Materials
• Setup Instructions
• Presenter Notes
  • Environment Setup
  • Clone the Git Repository
  • Create Objects
  • Start Docker Build
  • Update Prometheus ConfigMap
  • Test the Pushgateway
    • Validate the metric in Pushgateway
    • Validate the metric in Prometheus
• Resources
• Contributing

Overview

The following repositor shows how to setup a Prometheus Pushgateway in OpenShift. The source for the Prometheus Pushgateway can be found here.

License

The project utilizes code licensed under the terms of the Apache Software License and therefore is licensed under ASL v2 or later.

This program is free software: you can redistribute it and/or modify it under the terms of the Apache Software License version 2, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the Apache Software License for more details.

You should have received a copy of the Apache Software License along with this program. If not, see http://www.apache.org/licenses/LICENSE-2.0.html

Bill of Materials

Environment Specifications

This quickstart should be run on an installation of OpenShift Enterprise V3 with an existing Prometheus deployment.

Setup Instructions

The project contains a single template file (template/prometheus-pushgateway-template.yaml) that creates all the necessary objects.

Environment Setup

The Prometheus stack should already be configured in the openshift-metrics namespace.

Clone the Git Repository

cd ~/git
git clone https://github.com/themoosman/openshift-prometheus-pushgateway.git
cd openshift-prometheus-pushgateway

Create Objects

Process the temaplate to create the necessary objects, update the ROUTE_URL as necessary.

oc project openshift-metrics
oc process -f templates/prometheus-pushgateway-template.yaml -p ROUTE_URL=pushgateway.apps.ocp.example.com  | oc create -f -

Start Docker Build

Start the build.

oc start-build prometheus-pushgateway --follow

Update Prometheus ConfigMap

Currently this is manual, so PRs are greatly appreciated.

oc edit configmap prometheus

Add this yaml block as an additional scrape config.

- job_name: 'pushgateway'
  honor_labels: true

  kubernetes_sd_configs:
    - role: service

  relabel_configs:
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
      action: keep
      regex: "pushgateway"
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - source_labels: [__name__]
      action: replace
      regex: (.+):(?:\d+);(\d+)
      replacement: ${1}:${2}
      target_label: __address__

Recycle the Prometheus pod to make sure the ConfigMap is updated.

oc delete pod -l app=prometheus

Test the Pushgateway

Push a metric to the gateway.

echo "test_metric 1.00" | curl -k --data-binary @- https://pushgateway.apps.ocp.example.com/metrics/job/test_job

Validate the metric in Pushgateway

Visit the Pushgateway and search for metric.

Validate the metric in Prometheus

Visit Prometheus and query for the metric.

Resources

• Prometheus Pushgateway Source
• Prometheus Pushgateway
• Monitoring multiple federated clusters with Prometheus

Contributing

Anyone is encouraged to contribute to the project by forking and submitting a pull request. (If you are new to GitHub, you might start with a basic tutorial.) By contributing to this project, you grant a world-wide, royalty-free, perpetual, irrevocable, non-exclusive, transferable license to all users under the terms of the Apache Software License v2 or later.