Kafka的整体架构:
本文中的配置:
在两台机器Node1和Node2上,分别部署了两个broker,Zookeeper使用的是单独的ZK集群。
在每个机器上下载并解压kafka_2.10-0.8.2.1
http://kafka.apache.org/downloads.html
Kafka配置
Node1:
ip为 172.16.212.17
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
cd
$KAFKA_HOME
/config
cp
server.properties server1.properties
cp
server.properties server2.properties
vi
server1.properties 修改以下参数:
broker.
id
=1
port=9091
host.name=172.16.212.17
log.
dirs
=
/tmp/kafka-logs/broker1/
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
vi
server2.properties 修改以下参数:
broker.
id
=2
port=9092
host.name=172.16.212.17
log.
dirs
=
/tmp/kafka-logs/broker2/
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
|
Node2:
ip为 172.16.212.102
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
cd
$KAFKA_HOME
/config
cp
server.properties server3.properties
cp
server.properties server4.properties
vi
server1.properties 修改以下参数:
broker.
id
=3port=9091
host.name=172.16.212.102
log.
dirs
=
/tmp/kafka-logs/broker3/
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
vi
server2.properties 修改以下参数:
broker.
id
=4
port=9092
host.name=172.16.212.102
log.
dirs
=
/tmp/kafka-logs/broker4/
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
|
Kafka启动
Node1:
1
2
3
|
cd
$KAFKA_HOME
/bin
nohup
.
/kafka-server-start
.sh $KAFKA_HOME
/config/server1
.properties &
nohup
.
/kafka-server-start
.sh $KAFKA_HOME
/config/server2
.properties &
|
Node2:
1
2
3
|
cd
$KAFKA_HOME
/bin
nohup
.
/kafka-server-start
.sh $KAFKA_HOME
/config/server3
.properties &
nohup
.
/kafka-server-start
.sh $KAFKA_HOME
/config/server4
.properties &
|
启动后,可以在Zookeeper中看到4个brokers:
[zk: localhost:2181(CONNECTED) 4] ls /brokers/ids
[3, 2, 1, 4]
创建topic
在任意Node上,
1
2
|
cd
$KAFKA_HOME
/bin
.
/kafka-topics
.sh --create --zookeeper zk1:2181,zk2:2181,zk3:2181 --replication-factor 2 --partitions 2 --topic lxw1234.com
|
创建一个名为lxw1234.com的topic.
查看topic
1
2
3
4
5
|
cd
$KAFKA_HOME
/bin
.
/kafka-topics
.sh --describe --zookeeper zk1:2181,zk2:2181,zk3:2181 --topic lxw1234.com
Topic:lxw1234.com PartitionCount:2 ReplicationFactor:2 Configs:
Topic: lxw1234.com Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2
Topic: lxw1234.com Partition: 1 Leader: 2 Replicas: 2,3 Isr: 2,3
|
模拟producer发送消息
1
2
|
cd
$KAFKA_HOME
/bin
.
/kafka-console-producer
.sh --broker-list 172.16.212.17:9091,172.16.212.17:9092,172.16.212.102:9091,172.16.212.102:9092 --topic lxw1234.com
|
启动之后,在控制台上可以先输入一些消息:
1
2
3
4
|
[root@dev bin]
# ./kafka-console-producer.sh --broker-list 172.16.212.17:9091,172.16.212.17:9092,172.16.212.102:9091,172.16.212.102:9092 --topic lxw1234.com
[2015-09-24 14:03:24,616] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
This is Kafka producer.
Hello, lxw1234.com.
|
模拟consumer接收消息
1
2
3
4
|
cd
$KAFKA_HOME
/bin
.
/kafka-console-consumer
.sh --zookeeper zk1:2181,zk2:2181,zk3:2181 --topic lxw1234.com --from-beginning
This is Kafka producer.
Hello, lxw1234.com.
|
接下来可以在producer的控制台中输入其他消息,看看consumer的控制台是否能正常打印出来。
删除topic
1
2
|
cd
$KAFKA_HOME
/bin
.
/kafka-topics
.sh --delete --zookeeper zk1:2181,zk2:2181,zk3:2181 --topic lxw1234.com
|
执行后,该topic会被kafka标记为删除,还需要在zookeeper中手动删除相关的节点:
[zk: localhost:2181(CONNECTED) 5] rmr /brokers/topics/lxw1234.com
停止Kafka
cd $KAFKA_HOME/bin
./kafka-server-stop.sh
或者找到kafka的进程,直接kill掉即可。