This module demonstrates the following:
- The usage of the Kafka Streams DSL, including
aggregate()
,groupByKey()
,selectKey()
,toStream()
andpeek()
. - Unit testing using the Topology Test Driver.
In this module, records of type <String, KafkaPerson>
are streamed from a topic named PERSON_TOPIC
.
The following tasks are performed:
- Group the stream by last name using
groupByKey()
operation. - Apply an aggregator that combines each
KafkaPerson
record with the same last name into aKafkaPersonGroup
object and aggregates the first names by last name. - Write the resulting records to a new topic named
PERSON_AGGREGATE_TOPIC
.
The output records will be in the following format:
{"firstNameByLastName":{"Last name 1":{"First name 1", "First name 2", "First name 3"}}}
{"firstNameByLastName":{"Last name 2":{"First name 4", "First name 5", "First name 6"}}}
{"firstNameByLastName":{"Last name 3":{"First name 7", "First name 8", "First name 9"}}}
To compile and run this demo, you will need the following:
- Java 21
- Maven
- Docker
To run the application manually, please follow the steps below:
- Start a Confluent Platform in a Docker environment.
- Produce records of type
<String, KafkaPerson>
to a topic namedPERSON_TOPIC
. You can use the producer person to do this. - Start the Kafka Streams.
To run the application in Docker, please use the following command:
docker-compose up -d
This command will start the following services in Docker:
- 1 Zookeeper
- 1 Kafka broker
- 1 Schema registry
- 1 Control Center
- 1 producer person
- 1 Kafka Streams aggregate