Setting up Kafka

1. Download the latest release package from kafka site. Un-tar it and move extracted files to where you want to put them, like $HOME/usr or $HOME/bin.

2. Change the config.
   For some scenario, we only need to update server.properties, reference to: brokerconfigs:

   • broker.id: the unique broker id in integer.
   • port: default is 9092. The port the socket server listens on.
   • log.dirs: a comma-separated list of one or more directories in which Kafka data is stored. It's better to change this directory to local dir, such as ${HOME}/data/kafka. Default dir is /tmp/kafka-logs.
   • zookeeper.connect: zookeeper connection string, a comma separated host:port pairs, each corresponding to a zk server. ZooKeeper also allows you to add a "chroot" path which will make all kafka data for this cluster appear under a particular path. For example, ${zk_host1}:2181,${zk_host2}:2181,${zk_host3}:2181/kafka

3. Start kafka server.
   After setting config, in kafka install dir, run following command to start kafka cluster server:

   bin/kafka-server-start.sh config/server.properties

---

Kafka Utility

1. Create topic

   bin/kafka-topics.sh --create --zookeeper ${zk_host}:${zk_port}[${zk_chroot}] --replication-factor ${replication_factor} --partitions ${partition_count} --topic ${topic_name}
   

   Note: for zookeeper connect string ${zk_host}:${zk_port}[${zk_chroot}], ${zk_chroot} is optional

2. Describe topic

   • Describe topic all topics

     bin/kafka-topics.sh --describe --zookeeper ${zk_host}:${zk_port}[${zk_chroot}]
     

   • Describe specified topic

     bin/kafka-topics.sh --describe --zookeeper ${zk_host}:${zk_port}[${zk_chroot}] --topic ${topic_name}
     

3. Publish messages. You can use this utility to test you topic.

   bin/kafka-console-producer.sh --broker-list ${kafka_server}:{kafka_port} --topic ${topic_name}
   

   Note: if you local kafka server is started, you can use localhost:9092 as broker-list.

4. Consume messages

   bin/kafka-console-consumer.sh --zookeeper ${zk_host}:${zk_port}[${zk_chroot}] [--from-beginning] --topic ${topic_name}
   

5. Modifying topics
   Currently, support increase partitions well:

   bin/kafka-topics.sh --zookeeper ${zk_host}:${zk_port}[${zk_chroot}] --alter --topic ${topic_name} --partitions ${partition_count}
   

   Kafka does not currently support reducing the number of partitions for a topic or changing the replication factor.

6. Delete topic

   bin/kafka-topics.sh --zookeeper ${zk_host}:${zk_port}[${zk_chroot}] --delete --topic ${topic_name}
   

   Topic deletion option is disabled by default. To enable it set the server config:

   delete.topic.enable=true