You need to setup & run quick a bunch of Kafka brokers ? You are afraid of monkeying with the underlying Zookeeper ensemble ? You are cool and love Kubernetes ?
Good news, you can easily leverage Ochonetes and get everything running in no time !
We propose two simple containers that will do all the magic for you. You can look at the code in the images/
folder. We basically have a Zookeeper container that will self-cluster into an
ensemble and a Kafka container which relies on that ensemble (e.g it will actively
wait on it and use the IPs in its configuration file).
How do they know about each other ? Go look at Ochopod for details about that magical process. What you want is to deploy these two guys in the same clustering scope and they will find each other. I said it was easy right ?
If you are curious try deploying less than 3 Zookeeper containers. The
Kafka containers will then boot but refuse to configure until this condition is met.
Please note I included a fairly simple Kafka container which will use a local
/var/lib/kafka to store its logs. Feel free to mount something or even allocate a volume on the fly.
Easy, look at the README in Ochonetes if you are wondering what I am talking about.
If have a specific broker configuration in mind first edit kafka.yml and specify your own settings. You want 4
brokers ? No problemo, use our cool CLI and do the following:
$ ./cli.py
welcome to the ocho CLI ! (CTRL-C to exit)
> deploy zookeeper -p 3 -n test
100% success / spawned 3 pod(s)
> deploy kafka -p 4 -n test
100% success / spawned 4 pod(s)
Wait a bit and you should see your brokers. For instance:
> grep test.kafka
<test.kafka> -> 100% replies (4 pods total) ->
cluster | pod IP | process | state
| | |
test.kafka #1 | 10.244.2.8 | running | leader
test.kafka #2 | 10.244.2.9 | running | follower
test.kafka #3 | 10.244.2.10 | running | follower
test.kafka #4 | 10.244.1.8 | running | follower
Your brokers are now functional. Note their IP and use them from any other pod on port TCP 9092. When you are tired playing with them you can tear the whole thing down in one shot :
> kill "test.*"
$ 100% success / killed 7 pod(s)
Contact autodesk.cloud.opensource@autodesk.com for more information about this project.
© 2015 Autodesk Inc. All rights reserved
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.