Kafka MirrorMaker 2 Demo

Kafka v2.4.0 introduced MirrorMaker 2.0 (MM2); a new multi-cluster, cross-datacenter replication engine. MirrorMaker 2.0 is well documented in the KIP-382 proposal and the MirrorMaker 2.0 README.md, but I learn-by-doing.

This demo sets up 2 local, single-broker Kafka clusters and one MirrorMaker 2.0 instance to replicate topics from cluster A to cluster B. Once you have this environment set up, it's quick and easy to play with different configuration options.

Clusters	Topic	Kafka Port	ZooKeeper Port
A	t1	9091	2181
B	t2	9092	2182

Set up

1. Clone this repository and cd into the directory.

2. Download a Kafka release and extract it to a ./kafka/.

3. Clean up, if you've run this before.

   rm -rf /tmp/zk1
   rm -rf /tmp/zk2
   rm -rf /tmp/kafka-logs-1
   rm -rf /tmp/kafka-logs-2

4. Run each of the following commands in a separate shell to start your two single-node Kafka clusters (each with their own single-node ZooKeeper instance).

   kafka/bin/zookeeper-server-start.sh zk1.properties
   kafka/bin/kafka-server-start.sh server1.properties
   kafka/bin/zookeeper-server-start.sh zk2.properties
   kafka/bin/kafka-server-start.sh server2.properties

5. Create topics.

   kafka/bin/kafka-topics.sh --create --topic t1 --bootstrap-server localhost:9091 --partitions 1 --replication-factor 1
   kafka/bin/kafka-topics.sh --create --topic t2 --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

6. Start MirrorMaker.

   kafka/bin/connect-mirror-maker.sh mm2.properties

Common tasks

# List topics
kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9091
kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092

# Produce
kafka/bin/kafka-console-producer.sh --topic t1 --bootstrap-server localhost:9091
kafka/bin/kafka-console-producer.sh --topic t2 --bootstrap-server localhost:9092

# Consume
kafka/bin/kafka-console-consumer.sh --topic t1 --bootstrap-server localhost:9091
kafka/bin/kafka-console-consumer.sh --topic t2 --bootstrap-server localhost:9092

# Consumer replicated topic
kafka/bin/kafka-console-consumer.sh --topic A.t1 --bootstrap-server localhost:9092