1.Download and install kafka
wget https://www.apache.org/dyn/closer.cgi?path=/kafka/0.8.1.1/kafka_2.10-0.8.1.1.tgz
tar zvxf kafka_2.10-0.8.1.1.tgz
cd kafka_2.10-0.8.1.1
2.Start zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
3.Start kafka
bin/kafka-server-start.sh config/server.properties
4.Create topic
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partition 1 --topics test
5.Open producer,send message
bin/kafka-console-producer.sh --broler-list localhost:9092 --topic test
6.Open consumer, receive message
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
7.Build a multi broker cluster
7.1 Copy setting files from the first broker
cp config/server.properties config/server-1.properties
cp config/server.properties config/server-2.properties
7.2 Add code below to every new broker
for broker1
broker.id = 1
port = 9093
log.dir = /tmp/kafka-log-1
for broker2
broker.id = 2
port = 9094
log.dir = /tmp/kafka-log-2
==++In the cluster, broker.id pointed to only one certain node. In case overwrite the data, we have to set different port and log file for each broker in the same mechine.++==
7.3 Now we have three node and one already be started, we just need to start the tow new nodes.
bin/kafka-server-start.sh config/server-1.properties
bin/kafka-server-start.sh config/server-2.properties
7.4 Create a topic which has three copies
bin/kafka-topics.sh --create --zookeeper lcoalhost:2181 --replication-factor 3 --partition 1 --topic my-replicated-topic
8. Now we have a cluster with three nodes, but how do we know information about each node? Run describe topics command and you will see.
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
Topic:my-replication-topic PartitionCount:1 Configs:
Topic:my-replication-topic Partition:1 Leader:0 Replicas:2,0,1 Isr:0,1,2
Notification: The first row show information of all the partitions. Below the first row, every partition has a row to show detail of its own information. Since there is only one partition for the topic, there is only one row.
Leader: Manage meaage reading and meaage writing
Replicas:List all broker
Isr: List all the broker which is in servicing
9. Test the fault-durable capbility of kafka
From information above we know that broker 0 is the leader, then we open this broker and send messages
bin/kafka-console-producer.sh --broler-list localhost:9092 --topic my-replicated-topic
And we open a consumer to receive the message
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic my-replicated-topic --from-beginning
Then we kill broker 0
netstat -tnlp
find broker 0(localhost:9092),remember PID, and kill it
kill -9 PID
And use command ‘describe’ we can see kafka chose another broker as the new leader and we use this new leader to send massege , the consumer can receive the message normally.