THIS REPOSITORY IS NO LONGER MAINTAINED AND ARCHIVED.

CMAK (prev. Kafka Manager) for Kubernetes

CMAK (prev. Kafka Manager) is a well-known mature tool for monitoring and managing Apache Kafka clusters.

CMAK operator is a Helm chart combining set of tools, that allows to install and configure CMAK (prev. Kafka Manager) into Kubernetes cluster.

Installation

CMAK operator could be installed only with Helm 3. Helm chart is published to public Helm repository, hosted on GitHub itself.

It’s recommended to install CMAK operator into a dedicated namespace.

Add Helm repository

$ helm repo add cmak https://eshepelyuk.github.io/cmak-operator
$ helm repo update

Install the latest version

$ helm install --create-namespace -n cmak-ns mycmak cmak/cmak-operator

Search for all available versions

$ helm search repo cmak-operator --versions
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
cmak/cmak-operator      0.2.1           3.0.0.5         CMAK operator for K8S.
cmak/cmak-operator      0.2.0           3.0.0.5         CMAK operator for K8S.

Install specific version

$ helm install --create-namespace -n cmak-ns --version 0.2.1 mycmak cmak/cmak-operator

Verify installation

❗	By default, CMAK operator doesn’t create neither Ingress nor any other K8S resources to expose UI via HTTP.

The simpliest test is to port forward CMAK UI HTTP port and access it from browser .

$ kubectl port-forward -n cmak-ns service/cmak 9000:9000

Then, open http://localhost:9000 in your browser.

Configuration

CMAK application

CMAK uses configuration file /cmak/conf/application.conf. Every parameter could be overriden via JVM system property, i.e. -DmyProp=myVal. Properties are passed to CMAK container via values.yaml.

For example to enable basic auth add following to values.yaml.

ui:
  extraArgs:
  - "-DbasicAuthentication.enabled=true"
  - "-DbasicAuthentication.username=admin"
  - "-DbasicAuthentication.password=password"

Kafka clusters

It’s extremely easy to configure multiple clusters in CMAK, starting from cluster setup, connection settings and ending with authorization using Helm values files.

Check CMAK operator values for all available options and their description.

Minimal values.yaml configuration for adding a several Kafka clusters to CMAK.

cmak:
  clusters:
    - name: "cluster-stage"
      curatorConfig:
        zkConnect: "kafka01.stage:2181,kafka02.stage:2181"
    - name: "cluster-prod"
      curatorConfig:
        zkConnect: "kafka01.prod:2181,kafka02.prod:2181,kafka03.prod:2181"

Connection settings could be configured for all clusters at once or per dedicated cluster.

cmak:
  clustersCommon:
    curatorConfig:
      zkMaxRetry: 100 # (1)
  clusters:
    - name: "cluster-stage"
      kafkaVersion: "2.4.0" # (2)
      curatorConfig:
        zkConnect: "kafka01.stage:2181,kafka02.stage:2181"
    - name: "cluster-prod"
      kafkaVersion: "2.1.0" # (3)
      enabled: false
      curatorConfig:
        zkConnect: "kafka01.prod:2181,kafka02.prod:2181,kafka03.prod:2181"

1. this setting is applied to both clusters.

2. applied only to cluster-stage.

3. applied only to cluster-prod.

Configuration should be passed to helm via command line during installation or upgrade.

$ helm install --create-namespace -n cmak-ns -f cmak-values.yaml mycmak cmak/cmak-operator

Architecture

CMAK operator comprises following components:

• CMAK, powered by Docker image from @hleb-albau.

• Apache ZooKeeper, powered by official Docker image.

• Custom cmak2zk tool, used to configure Kafka clusters in CMAK from YAML files.

Following desing choices were made.

Dedicated Zookeeper instance.

TO BE DEFINED.

Not using REST for configuring CMAK clusters.

TO BE DEFINED.

Reconciliation with CronJob.

TO BE DEFINED.

Standalone cmak2zk tool

cmak2zk was developed as a part of CMAK operator and actively used by the operator itself. But the same time this tool could be used on its own outside of Helm charts and Kubernetes.

Its purpose is to take Kafka cluster configuration for CMAK in YAML format and populate CMAK compatible config in Zookeeper. This allows to avoid manual configuration of CMAK and provides better possibilities to use CMAK in declarative configuration or GitOps based flows.

cmak2zk is distributed as docker image available at DockerHub.

To check out available options, run the image without parameters.

$ docker run eshepelyuk/cmak2zk:1.4.1

Example docker-compose and Kafka cluster configuration are located at cmak2zk/examples directory. One could run them using commands below.

$ curl -sLo clusters.yaml \
  https://raw.githubusercontent.com/eshepelyuk/cmak-operator/master/cmak2zk/examples/clusters.yaml

$ curl -sLo docker-compose-cmak2zk.yaml \
  https://raw.githubusercontent.com/eshepelyuk/cmak-operator/master/cmak2zk/examples/docker-compose-cmak2zk.yaml

$ docker-compose -f docker-compose-cmak2zk.yaml up

Wait for some time until components are stabilizing, it may take up to 5 mins. Then, open your browser at http://localhost:9000. There should be two pre-configured clusters, pointing to the same Kafka instance, running in Docker.

Alternatives

AKHQ project seems to be the most active open source tool for managing and monitoring Kafka clusters. It could be missing some functionality from CMAK, but their developers are open for feature requests and contributions.

Contribution

Your contributions like feature suggesstions, bug reports and pull requests are always welcomed. Please check CONTRIBUTING.adoc for details.