KafkaExLagExporter

This project will collect Kafka consumer lag and provide them via Prometheus.

Metrics

Prometheus is a standard way to represent metrics in a modern cross-platform manner. KafkaExLagExporter exposes several metrics as an HTTP endpoint that can be readily scraped by Prometheus.

kafka_consumergroup_group_topic_sum_lag

Labels: cluster_name, group, topic, consumer_id, member_host

The sum of the difference between the last produced offset and the last consumed offset of all partitions in this topic for this group.

kafka_consumergroup_group_lag

Labels: cluster_name, group, partition, topic, member_host, consumer_id

The difference between the last produced offset and the last consumed offset for this partition in this topic partition for this group.

Start

docker run -ti --net="host" -e KAFKA_BROKERS=redpanda:29092 -p 4000:4000 lechindianer/kafkaex_lag_exporter:0.2.0

Now you can check the exposed metrics at http://localhost:4000.

Configuration

KafkaExLagExporter uses 5 seconds as default interval to update the lags. If you want to configure it to use another value set KAFKA_EX_INTERVAL_MS, i.e.

docker run -ti --net="host" -e KAFKA_BROKERS=redpanda:29092 -e KAFKA_EX_INTERVAL_MS=10000 -p 4000:4000 \
  lechindianer/kafkaex_lag_exporter:0.2.0

Developing

To start the project locally:

KAFKA_BROKERS="localhost:9092" iex -S mix 

There is also a Docker compose file included which will start Kafka, serve Kowl (Web UI for Kafka) and start KafkaexLagExporter:

docker compose up --build

Kowl is served at http://localhost:8080.

Tests

MIX_ENV=test mix test --no-start

Code style

Don't forget to check credo for code violations:

mix credo

This project also leverages the use of typespecs in order to provide static code checking:

mix dialyzer

Links

Source is on Gitlab.

The initial project Kafka Lag Exporter was a huge inspiration for me creating my first real Elixir project. Thank you!