Install Clustered (4 nodes) Apache Kafka and Zookeeper on Ubuntu 10.04

http://www.jonzobrist.com/2012/04/17/install-apache-kafka-and-zookeeper-on-ubuntu-10-04/

Roughly followed quickstart guide
http://incubator.apache.org/kafka/quickstart.html

Need Java? Pissed that Ubuntu dropped official Sun JDK from their Partner Repository?
Use
https://launchpad.net/~ferramroberto/+archive/java
Need this to add-apt-repository
apt-get install -y python-software-properties
sudo add-apt-repository ppa:ferramroberto/java
apt-get update
apt-get -y install sun-java6-jdk
java -version
#Should return something like
java version “1.6.0_26″
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

Note that repository is yet to have a JDK for Ubuntu 12.04

I worked around this with a binary install of the latest 1.6 JDK to /usr/java

#Setup the Kafka user, install Kafka

I’m using 4 nodes, their IP’s are 192.168.1.50,51,52,53, all run Kafka and Zookeeper. The Zookeeper Admin guide recommends an odd number for the ensemble so majority rule is less likely to be split evenly/split brain style. although I did get an error from one of them urging me to run either Kafka or Zookeeper in an odd number of nodes, which I’m not sure the reasoning behind.

groupadd -g 3320 kafka
useradd -m -d /usr/local/kafka -s /bin/bash -u 3320 -g 3320 kafka
wget http://people.apache.org/~nehanarkhede/kafka-0.7.0-incubating/kafka-0.7.0-incubating-src.tar.gz
tar -zxvf kafka-0.7.0-incubating-src.tar.gz
cd kafka-0.7.0-incubating-src
mv * ~kafka/
chown -R kafka:kafka ~kafka
chown -R kafka:kafka ~kafka/*

su kafka
cd
./sbt update
./sbt package
#Now edit ~kafka/.bashrc and add these lines, you may have to change the quotes, since web clipboards will likely paste in non working versions of fancy quotes
declare -x JAVA_OPTS=”-Xmx3600M -Xms256M”
declare -x PATH=”${HOME}/bin:${PATH}”
mkdir -p ~/zookeeper ~/logs ~/run/logs
#Configure Kafka
#setup myid file, myid has to match the server’s setting in ~kafka/config/zookeeper.properties
#This will be different on each server (1,2,3,4)
echo “2” > /usr/local/kafka/zookeeper/myid
#Setup Kafka’s config file, server.properties, it needs direct pointers to all servers in the Kafka cluster on the zk.connect setting

 brokerid=2
 port=9092
 num.threads=8
 socket.send.buffer=1048576
 socket.receive.buffer=1048576
 max.socket.request.bytes=104857600
 log.dir=/usr/local/kafka/logs
 num.partitions=1
 log.flush.interval=10000
 log.default.flush.interval.ms=1000
 log.default.flush.scheduler.interval.ms=1000
 log.retention.hours=168
 log.file.size=536870912
 log.cleanup.interval.mins=1
 enable.zookeeper=true
 zk.connect=192.168.1.50:2181,192.168.1.51:2181,192.168.1.52:2181,192.168.1.53:2181
 zk.connectiontimeout.ms=1000000

#Setup Zookeeper’s config file, needs server.X for each node along with IP:port:port
vi config/zookeeper.properties
I ran this bash one liner on each node to get the server.X lines

echo "server.$(grep brokerid server.properties  | sed 's/=/ /' | awk '{ print $2 }')=$(hostname -I):2888:3888"

Then removed the spaces and put them into zookeeper.properties

dataDir=/tmp/zookeeper
 clientPort=2181
 maxClientCnxns=0
 tickTime=2000
 dataDir=/usr/local/kafka/zookeeper/
 initLimit=5
 syncLimit=2
 server.1=192.168.1.50:2888:3888
 server.2=192.168.1.51:2888:3888
 server.3=192.168.1.52:2888:3888
 server.4=192.168.1.53:2888:3888

#Setup Kafka’s producer.properties

 broker.list=1:192.168.1.50:9092,2:192.168.1.51:9092,3:192.168.1.52:9092,4:192.168.1.53:9092
 zk.connect=192.168.1.50:2181,192.168.1.51:2181,192.168.1.52:2181,192.168.1.53:2181
 producer.type=sync
 compression.codec=0
 serializer.class=kafka.serializer.StringEncoder

#Setup Kafka’s consumer.properties

 zk.connect=192.168.1.50:2181,192.168.1.51:2181,192.168.1.52:2181,192.168.1.53:2181
 zk.connectiontimeout.ms=1000000
 groupid=test-consumer-group

#Start it all, the docs say to use something like djb’s daemon tools, but for now I just run these in a screen, as user kafka

screen -R zookeeper
cd
~/bin/zookeeper-server-start.sh ~/config/zookeeper.properties
CTRL+A D

screen -R kafka
cd
~/bin/kafka-server-start.sh ~/config/server.properties
CTRL+A D

Check zookeeper

telnet localhost 2181
ruok

#expect back imok

#Send some Kafka messages
#Start Producer

bin/kafka-console-producer.sh --zookeeper localhost:2181 --topic test
This is a test
This is more testing

#Shutdown producer
CTRL+D
#Start consumer
#Receive the messages

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

#Expect to see
This is a test
This is more testing
References
http://incubator.apache.org/kafka/quickstart.html
http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html
http://zookeeper.apache.org/doc/r3.4.3/recipes.html
http://zookeeper.apache.org/doc/trunk/zookeeperJMX.html

Download Kafka from
http://incubator.apache.org/kafka/downloads.html