Apache Kafka Workshop

Table of Contents

1. Slides
2. Assignment
3. Kafka on Docker
4. Kafka on Host Machine
   1. Installation
   2. Start Zookeeper and Kafka
   3. Setup Zookeeper and Kafka Systemd Services for Linux
5. Create Kafka Topic
6. Produce messages to Kafka
7. Consume messages from Kafka
8. Produce/Consume using Kafka Clients
   1. Python-Client
9. Schema Registry and Schema Registry UI
10. Rest Proxy and Kafka Topics UI
11. Tools for Kafka and Zookeeper
    1. CMAK
    2. ZooNavigator
12. Reassigning Partitions

Slides

Slides are available here.

The instructions below are for a linux(optionally mac) OS. You can follow the steps in the Kafka Documentation for other OSs.

Assignment

1. Spin up dockerized containers for Kafka and Zookeeper using Kafka on Docker.
2. Install kafkacat.
3. Create a topic in the Kafka cluster using kafkacat.
4. Produce to and consume from the topic using kafkacat.

Additional steps:

5. Write a Java application to produce and consume from the Kafka topic using the kafka-clients directory in thie repo.

Kafka on Docker

There are several Kafka Docker images available. We are going to use wurstmeister/kafka-docker image here.

cd kafka

# Start zookeeper and kafka broker
docker-compose up -d

# Stop zookeeper and kafka containers
docker-compose down

Even if run Kafka on Docker, you would still want to follow the steps for Kafka on Host Machine if you want to use kafka-client-producer and kafka-client-consumer that is shipped along with Kafka.

Kafka on Host Machine

Installation

Download the latest Kafka binary from the Apache Kafka Download page.

wget <kafka tgz>

sudo tar -xvf <kafka tgz> -C /usr/local/

# Create a symbolic link to the kafka directory to refer to it easily 
sudo ln -s /usr/local/<kafka_dir> /usr/local/kafka

If you want to use Kafka commands directly without using , add the below export command to your .bashrc / .bash_profile and source the file:

# Add kafka bin to PATH
export PATH=/usr/local/kafka/bin:$PATH

source ~/.bashrc or source ~/.bash_profile

Start Zookeeper and Kafka

# Command options within [] are optional. Please make the relevant changes to your command before running them.
# -daemon runs the process in background
/usr/local/kafka/bin/zookeeper-server-start.sh [-daemon] /usr/local/kafka/config/zookeeper.properties

To stop zookeeper, /usr/local/kafka/bin/zookeeper-server-stop.sh

# Setting environment variables for Kafka
export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"

# -daemon runs the process in background
/usr/local/kafka/bin/kafka-server-start.sh [-daemon] /usr/local/kafka/config/server.properties

To stop kafka, /usr/local/kafka/bin/kafka-server-stop.sh

Setup Zookeeper and Kafka Systemd Services for Linux

Refer to the steps here to setup Systemd services for Kafka and Zookeeper to automate the start/stop commands and make your life easier.

Create Kafka Topic

/usr/local/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic topic1 --replication-factor 1 --partitions 2

Produce messages to Kafka

/usr/local/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic topic1

Open a new terminal window to start consuming while leaving this window untouched.

Consume messages from Kafka

/usr/local/kafka/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic topic1  [--from-beginning]

Experiment with producing in string messages using the console producer and viewing them back into the console consumer.

Produce/Consume using Kafka Clients

Python-Client

pip3 install kafka-python

python3 python-client/consumer.py

# In a new shell window:
python3 python-client/producer.py

Schema Registry and Schema Registry UI

Reference: Confluent Schema Registry

cd schema-registry

docker-compose up -d

This creates two services:

• Schema Registry at http://localhost:8081. You can then use curl or a Schema Registry UI to play with the registry.
• Schema Registry UI at http://localhost:8001

Rest Proxy and Kafka Topics UI

Reference: Confluent Rest Proxy

cd kafka-rest-proxy/

docker-compose up -d

This creates two services:

• Rest Proxy at http://localhost:8082.
• Kafka Topics UI at http://localhost:8002

Tools for Kafka and Zookeeper

CMAK - Cluster Manager for Apache Kafka

cd kafka-manager

docker-compose up -d

ZooNavigator

cd zoonavigator

docker-compose up -d

Zoonavigator should be available on http://localhost:9000

Reassigning Partitions

Documentation available here.

Create a topics-to-move.json file

{"topics":  [{"topic": "topic1"}],
 "version":1
}

kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file topics-to-move.json --broker-list "1001,1002,1003" --generate

Save the proposed partition reassignment configuration into a file reassignment.json

{
  "version":1,
  "partitions":[{"topic":"topic1","partition":0,"replicas":[1002],"log_dirs":["any"]},
                {"topic":"topic1","partition":2,"replicas":[1001],"log_dirs":["any"]},
                {"topic":"topic1","partition":1,"replicas":[1003],"log_dirs":["any"]}]
}

kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file reassignment.json --execute

kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file reassignment.json --verify