Linux搭建kafka集群并测试
1 准备3台机器
hostname | ip |
---|---|
server1 | 192.168.16.151 |
server2 | 192.168.16.152 |
server3 | 192.168.16.153 |
1.1host配置
#以server1为例
127.0.0.1 server1
192.168.16.151 server1
192.168.16.152 server2
192.168.16.153 server3
2 zookeeper配置
参考上篇博客 安装Zookeeper集群
3 安装kafka
3.1 设置环境变量
解压kafka后添加环境变量
#修改环境变量
sudo vim /etc/profile
export JAVA_HOME=/usr/lib/java/jdk1.8.0_181
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
export SCALA_HOME=/usr/lib/scala/scala-2.11.8
export PATH=${SCALA_HOME}/bin:$PATH
export ZOOKEEPER_HOME=/usr/local/programs/zookeeper-3.4.13
export PATH=${ZOOKEEPER_HOME}/bin:$PATH
export KAFKA_HOME=/usr/local/programs/kafka_2.11-2.1.0
export PATH=${KAFKA_HOME}/bin:$PATH
#让环境变量生效
source /etc/profile
3.2 修改改$KAFKA_HOME/config 目录下的server.properties文件
在三台机器上分别修改,这里用server1举例
#这个1就是和zookeeper的myid文件的1对应
broker.id=1
#填写3台机器的地址,中间用逗号隔开
zookeeper.connect=server1:2181,server2:2181,server3:2181
#host.name是当前机器的地址
host.name=192.168.16.151
3.4 启动zookeeper和kafka
在3台机器上分别启动
#启动zookeeper
zkServer.sh start
#启动kafka
kafka-server-start.sh -daemon /usr/local/programs/kafka_2.11-2.1.0/config/server.properties
#查看进程
jps
3.5 测试
#在leader上创建主题
kafka-topics.sh --create --zookeeper server1:2181,server2:2181,server3:2181 --replication-factor 3 --partitions 3 --topic WinterTop
#查看主题List
kafka-topics.sh --list --zookeeper server1:2181,server2:2181,server3:2181
#在leader上发送消息
kafka-console-producer.sh --broker-list server1:9092,server2:9092,server3:9092 --topic WinterTop
#在follower上消费消息
kafka-console-consumer.sh --bootstrap-server server1:9092,server2:9092,server3:9092 --from-beginning --topic WinterTop