Skip to content

A simple proxy built atop Akka HTTP for Apache Kafka

License

Notifications You must be signed in to change notification settings

stevenchen3/kafka-proxy

Repository files navigation

Kafka Proxy

Inspired by Confluent REST Proxy and Kafka-Pixy. Give it a try to implement a simple REST/gRPC proxy built atop Akka HTTP for Apache Kafka using Scala and carry out some benchmarks. The codes are for quick prototyping purpose, and have not yet been covered by unit tests.

Building the source on bare-metal

Prerequisites:

  • JDK 1.8
  • SBT 0.13.x or 1.0.x
  • Protobuf 3.0.0 or above
  • grpc-java codegen compiler plugin

Compiling protoc-gen-grpc-java from source

The protoc-gen-grpc-java binary is required by sbt-protobuf plugin in order to automatically generate gRPC related Java source files in SBT Compile scope. Below is one way to get this protoc-gen-grpc-java binary.

git clone --recursive git@github.com:grpc/grpc-java.git
cd grpc-java/compiler
../gradlew java_pluginExecutable
../gradlew test
cp build/exe/java_plugin/protoc-gen-grpc-java /usr/local/bin/

See here for details.

Building

Compile the source codes and run the tests:

sbt compile
sbt test
sbt assembly

# Run scalafix check
sbt fixCheck

Building the source inside Docker

Alternatively, we can build the source using Docker in that we could possibly avoid complex dependencies on bare-metal environment as they have been well configured inside Docker image.

Prerequisites:

  • Docker 18 or above
  • Docker Compose 1.23 or above

Building

Compile the source codes and run the tests (it may take some time for the first time):

docker-compose run test

Building the Docker image

Below will create a Docker image tagged as kafka-proxy:0.1.0-rc1.

docker-compose build

Gotcha

  • dyld: Library not loaded error during compilation
dyld: Library not loaded: /usr/local/opt/protobuf/lib/libprotoc.13.dylib

Reinstall grpc-java to fix it.