In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. --Wikipedia
Kubernetes is a container management system with a growing wealth of features. Your application running in containers is also a complex system as it embraces the distributed architecture of loosely coupled, highly cohesive services. As these modular containers run, things may not always behave as smoothly as you hope. Embracing the notions of antifragility and designing a system to be resilient despite the realities of resource limitations, network failures, hardware failures and failed software logic. This demands a robust monitoring stack to open views into the behaviors and health of your applications running in a cluster.
Three important types of monitoring are metric data events, log streams and tracing. Over time there are services that gather, store and export the metrics and logs. There are Time Series Databases (TSDBs) that append each temporal piece of data. There are monitors that evaluate alerting rules against key changes in data. Lastly, there are dashboards and reports offering views into states based on the metrics and logs gathered during a time range. Ultimately, to provide viewports into the state and health of your cluster and its hosted applications.
For metrics and data events in Kubernetes it is common to use:
- Prometheus to gather runtime data events and fire alerts
- Alertmanager to gather and route alerts based on changing data
- Grafana to visually aggregate and decipher the Prometheus data.
For logging in Kubernetes it's common to use:
- ElasticSearch to store the collated stream data into a searchable time series database
- Fluentd to gather the various log streams across the containers
- Kibana to visually interact with frequent searches against the ElasticSearch data.
For tracing in Kubernetes it's possible to use:
- ZOP stack (ZipKin, OpenTracing, Prometheus)
- JOP stack (Jaeger, OpenTracing, Prometheus)
- Start a personal Kubernetes cluster
- Add a private Docker registry along with a dashboard
- Build and push 3 Spring Boot microservices to the registry
- Add a monitoring tool stack with Prometheus, Alertmanager and Grafana
- Start the 3 microservices on Kubernetes
- Monitor the microservices metrics through Prometheus and Grafana
- Clone this project from GitHub
- Install Minikube. Other Kubernetes solutions are available, however these steps below are specific to Minikube). Do not start Minikube, just install the command line tool and VirtualBox. Below the script will start Minikube with the correct sizing and configuration.
- Install [KubeCtl] (https://kubernetes.io/docs/tasks/tools/install-kubectl/)
- From project root run
./minikube-start.sh. This provisions a personal Kubernetes cluster with a few addons for you. - Verify
minikube statusandkubectl versionrun correctly
The typical tool used to interact with Kubernetes is KubeCtl. However, the Helm tool further simplifies deployments and makes the provisioning experience more consistent. The quick start doc provides the best instructions:
- Install the Kubernetes package manager Helm.
Helm is a command line tool that also installs a small complimentary service called Tiller onto your cluster. To setup Tiller for this demonstration run the shell script from within the helm directory:
cd helm
helm-init-rbac.sh
This enables Tiller and gives permissions to add the registry UI and monitoring tool stack in the next steps.
At this point serveral pods are starting and it takes a few minutes before the deployment is healthy. The startup progress can be observed in the kube-system namespace in the dashboard.
kubectl get pods -n kube-system
or
minikube dashboard
There is a Minikube addon called registry that was enabled in the minikube-start.sh script. It's a private Docker registry, but it lacks a user interface. There is a container that fronts this registry with a helpful dashboard. Start the Docker registry UI from the project helm directory by running:
cd helm
helm install charts/docker-registry-web --namespace kube-system --name ui
In a few minutes a service allots access to a browser based UI. Here you can verify the dashboard service is running and displays the registry contents. At this step the registry is empty but shortly you will deploy three microservices to this Docker registry and are listed with this command:
minikube service -n kube-system ui-docker-registry-web
The above command opens your default browser to this Docker registry UI page.
This browser URL brings you to the same page:
http://registry-ui.minikube.st81ess.com
This uses a public DNS server that maps minikube.st81ess.com to 192.168.99.100 (the likely, but not guaranteed, Minikube IP on VirtualBox) then with an ingress it maps registry-ui to the service.
There are three Spring Boot microservices that communicate with each other: Quotes, Authors and Biographies. The above minikube-start.sh script enables the Minikube addon called "registry". This is a private Docker registry running as a container in your cluster. A simple script builds and deploys these 3 microservices and pushes their Docker images to this registry. In the project directory <project>/microservices there is the source for the microservices and a script that builds and pushes the Docker images. Run:
./pushImages.sh
and it runs the Gradle task pushImages on each microservice project. If exceptions occur when attempting to push the image to the registry make sure you have run the env.sh script at the command line.
Explore the docker.gradle file for each service to see how the plugin builds, tags and pushes the Docker images. Once the images are pushed verify they are in the registry with the command:
curl -X GET $(minikube service -n kube-system registry --url)/v2/_catalog
This curl request returns: {"repositories":["authors","biographies","quotes"]}
Observe the 3 microservice Docker images in the registry UI:
minikube service -n kube-system ui-docker-registry-web
The GitHub project coreos/prometheus-operator provides a configured Grafana dashboards and Prometheus settings for many Kubernetes clusters. It also has documented installation procedures that provision your cluster, even a small one on Minikube is an effective monitoring solution.
Before continuing ensure the Helm Tiller is configured by running the script mentioned above helm\helm-init-rbac.sh.
There are two key charts combined to offer a helpful monitoring stack: prometheus-operator and kube-prometheus. To install these charts run this script:
cd ../configurations
./prometheus-stack.sh
You will see a list of exposed http endpoints for Prometheus, Alertmanager and Grafana. The containers become available after downloaded and initialized. Observe the resources created for this monitoring stack:
minikube service list
or
minikube dashboard
Now that microservice images are deployed to the private registry, the services can now start. Start the 3 microservice containers by changing to the helm directory and running this install:
cd helm
helm install charts/microservices --namespace quotes --name ms
Next, tell Prometheus there is a new target for metrics to scrape
cd configurations
kubectl create -n monitoring -f prometheus-targets.yaml
This project contains three microservices based on Java Spring Boot. They provide JSON based REST APIs and coordinate to return random famous quotes, biographies and authors from the respective services.
To view the 3 microservices exposed via their service's NodePorts:
minikube service list -n quotes
While services are instantly available, the pods may take longer to enter their ready states.
curl $(minikube service -n quotes ms-authors --url)/author/random
curl $(minikube service -n quotes ms-biographies --url)/bio/random
curl $(minikube service -n quotes ms-quotes --url)/quote/random
Invoking a get /quotes/full on the ms-quotes service returns a random author's small biography and a random quote from their notable curation.
curl $(minikube service -n quotes ms-quotes --url)/quote/full
This invokes the 3 microservices in a chained transaction. Each REST request produces a random quote from a random author.
Observe the metrics for each service at the relative REST location /metrics.
curl $(minikube service -n quotes ms-quotes --url)/metrics
As you are exercising these images the monitoring data is being collected by Prometheus. Access the three monitoring dashboards with these commands:
minikube service -n monitoring kube-prometheus-prometheus
minikube service -n monitoring kube-prometheus-alertmanager
minikube service -n monitoring kube-prometheus-grafana
Activity in the namespace 'quotes' is observed in these interfaces as the services are exercised. Pod replications can be increased and the services can be requested to generate metrics.
while true; do curl -w " ms: %{time_total}\n" $(minikube service -n quotes ms-quotes --url)/quote/full; done
The class MetricStatics defines the exported metrics such as counters for demo_http_requests_total. These changing metrics are observed in Prometheus as the services are requested.
Next, increase the scale value for each microservice pod to 3:
cd helm
helm upgrade --set authors.replicaCount=3 --namespace quotes ms charts/microservices
Grafana shows the scaling of the pods. Run the above command line observe the faster response times.
Next, increase the latency of the Authors service:
cd helm
helm upgrade --set authors.container.latency=599 --namespace quotes ms charts/microservices
Error metrics can be observed with
curl $(minikube service -n quotes ms-quotes --url)/forceError
The JVM metrics are present because of the line: DefaultExports.initialize();
Prometheus scrapes all these metrics with the relative /metrics call. If the service exposes a URL for metrics, then it listed in the Prometheus "Targets" catalog. The URL definition is defined in the spring-beans-config.xml Spring Boot file for each microservice project.
Reflect on what you have just explored. You stood up a personal cluster with a private Docker registry. You deployed and started three microservices wrapped in Docker containers. Next, you stood up a monitoring system and observed how the behaviors of the microservices are tracked with Prometheus metrics. Along the way you have seen examples of Spring Boot, Gradle, Helm and the helpful features Kubernetes.
My hope is this gives you a deeper appreciation of Kubernetes as a powerful software architecture movement. -- Jonathan Johnson
- VirtualBox 5.2.18
- Minikube 0.28.2 (Kubernetes 1.10.0 + Docker)
- Kubectl 1.11.2
- Helm 2.10.0
- Prometheus Operator
- Kube-Prometheus (Alertmanager + Grafana)
- Java 1.8
- Spring Boot 2.0.4-RELEASE
- Gradle 4.10 and a few helpful plugins for building and deploying containers
- Add YAML for Alertmanager rules.
- Add logging: fluentd, ElasticSearch, Kibana via the EFK addon in Minikube
- Add tracing: ZipKin and Sleuth for demonstration of tracing solutions
| Step | Command |
|---|---|
| Fresh Minikube | minikube delete |
| Initialize | ./minikube-start.sh |
| CLI env | . ./env.sh |
| Init Helm | cd helm && ./helm-init-rbac.sh |
| Tiller starts in ~2 min. | helm status |
| Start Docker UI | helm install charts/docker-registry-web --namespace kube-system --name ui |
| Change dir | cd ../configurations |
| Start Monitoring | ./prometheus-stack.sh |
| Publish microservices | cd microservices && ./pushImages |
| Start microservices | helm install charts/microservices --namespace quotes --name ms |
| Excersize microserices | curl $(minikube service -n quotes ms-quotes --url)/quote/full |
| Step | Command |
|---|---|
| Start microservices | cd ../helm && helm install charts/microservices --namespace quotes --name ms |
| Extract url | export SERVICE=$(minikube service -n quotes ms-quotes --url) && echo $SERVICE |
| Exercise services | curl -w " ms: %{time_total}\n" $SERVICE/quote/full |
| Loop services | while true; do curl -w " ms: %{time_total}\n" $SERVICE/quote/full; done |
| Loop and watch time | while true; do curl -w " ms: %{time_total}\n" $SERVICE/quote/full : head -c10; done |
Observe in Prometheus the statistic demo_http_requests_total growing.
- Visit the No Fluff Just Stuff tour and explore many ideas like this. Monitoring Clusters and Containers
- Installing the Prometheus Operator, a CoreOS open source contribution
- Spring Boot correlationIds
- Thank you Max Kuchin for the Docker Registry Web container.
- This solution is inspired from the Prometheus installation provided by CoreOS Tectonic.
- Chris Ricci, Solutions Engineer, CoreOS/Red Hat provides a helpful demonstration of Prometheus. Also, a talk on the advancements in version 2.
- Get Kubernetes Cluster Metrics with Prometheus in 5 Minutes