这里3台机器集群ip如下
192.168.0.101
192.168.0.102
192.168.0.103
kafka安装需要依赖zookeeper,我们这里已经在这3台机器上安装了zookeeper集群。
1、下载解压kafka
cd /usr/local/soft;
wget https://downloads.apache.org/kafka/2.8.0/kafka_2.13-2.8.0.tgz;
tar -xzvf kafka_2.13-2.8.0.tgz;
mv kafka_2.13-2.5.0 kafka;
cd kafka;
2、同步到其他2台机器
#分别在102和103上执行
scp -r root@192.168.0.101:/usr/local/soft/kafka /usr/local/soft/
3、修改配置文件
101机器server.properties文件修改:
broker.id=1
listeners=PLAINTEXT://192.168.0.101:9092
# 配置zk地址
zookeeper.connect=192.168.0.101:2181,192.168.0.102:2181,192.168.0.103:2181
102机器server.properties文件修改:
broker.id=2
listeners=PLAINTEXT://192.168.0.102:9092
# 配置zk地址
zookeeper.connect=192.168.0.101:2181,192.168.0.102:2181,192.168.0.103:2181
103机器server.properties文件修改:
broker.id=3
listeners=PLAINTEXT://192.168.0.103:9092
# 配置zk地址
zookeeper.connect=192.168.0.101:2181,192.168.0.102:2181,192.168.0.103:2181
4、启动kafka
cd /usr/local/soft/kafka/bin
./kafka-server-start.sh -daemon ../config/server.properties
集群启动脚本
#!/bin/bash
#set -x
brokers="172.23.163.127 172.23.163.128 172.23.163.129"
kafka_home="/home/software/kafka/kafka_2.11-2.4.1/"
start() {
for i in $brokers
do
echo "Starting kafka on ${i} ... "
ssh root@$i "source /etc/profile; sh ${kafka_home}/bin/kafka-server-start.sh -daemon ${kafka_home}/config/server.properties"
if [[ $? -ne 0 ]]; then
echo "Start kafka on ${i} is OK !"
fi
done
echo kafka kafka are started !
}
stop() {
for i in $brokers
do
echo "Stopping kafka on ${i} ..."
ssh root@$i "source /etc/profile;sh ${kafka_home}/bin/kafka-server-stop.sh ${kafka_home}/config/server.properties"
if [[ $? -ne 0 ]]; then
echo "Stopping ${kafka_home} on ${i} is down"
fi
done
echo all kafka are stopped !
}
case "$1" in
start)
start
;;
stop)
stop
;;
*)
echo "Usage: start|stop"
;;
esac
5、集群下创建topic
cd /usr/local/soft/kafka/bin;
sh kafka-topics.sh --create --zookeeper 192.168.0.101:2181,192.168.0.102:2181,192.168.0.103:2181 --replication-factor 2 --partitions 3 --topic test1;
查看已经创建的 topic:
sh kafka-topics.sh --list --zookeeper 192.168.0.101:2181,192.168.0.102:2181,192.168.0.103:2181
6、集群下启动Consumer
在一个新的远程窗口中:
sh kafka-console-consumer.sh --bootstrap-server 192.168.0.101:9092,192.168.0.102:9092,192.168.0.103:9092 --topic test1 --from-beginning
7、集群下启动Producer
打开一个新的窗口:
sh kafka-console-producer.sh --broker-list 192.168.0.101:9092,192.168.0.102:9092,192.168.0.103:9092 --topic test1
8、测试发送消息
参考博客:https://gper.club/articles/7e7e7f7ff4g57gcag69