kafka集群部署

这里3台机器集群ip如下

192.168.0.101

192.168.0.102

192.168.0.103

kafka安装需要依赖zookeeper，我们这里已经在这3台机器上安装了zookeeper集群。

1、下载解压kafka

cd /usr/local/soft;
wget https://downloads.apache.org/kafka/2.8.0/kafka_2.13-2.8.0.tgz;
tar -xzvf kafka_2.13-2.8.0.tgz;
mv kafka_2.13-2.5.0 kafka;
cd kafka;

2、同步到其他2台机器

#分别在102和103上执行
scp -r root@192.168.0.101:/usr/local/soft/kafka /usr/local/soft/

3、修改配置文件

101机器server.properties文件修改：

broker.id=1
listeners=PLAINTEXT://192.168.0.101:9092
# 配置zk地址
zookeeper.connect=192.168.0.101:2181,192.168.0.102:2181,192.168.0.103:2181

102机器server.properties文件修改：

broker.id=2
listeners=PLAINTEXT://192.168.0.102:9092
# 配置zk地址
zookeeper.connect=192.168.0.101:2181,192.168.0.102:2181,192.168.0.103:2181

103机器server.properties文件修改：

broker.id=3
listeners=PLAINTEXT://192.168.0.103:9092
# 配置zk地址
zookeeper.connect=192.168.0.101:2181,192.168.0.102:2181,192.168.0.103:2181

4、启动kafka

cd /usr/local/soft/kafka/bin
./kafka-server-start.sh -daemon ../config/server.properties

集群启动脚本

#!/bin/bash                                                                                                                                                                                                     
#set -x                                                                                                                                                                                                         
brokers="172.23.163.127 172.23.163.128 172.23.163.129"
kafka_home="/home/software/kafka/kafka_2.11-2.4.1/"

start() {
  for i in $brokers
  do
      echo "Starting kafka on ${i} ... "
      ssh root@$i "source /etc/profile; sh ${kafka_home}/bin/kafka-server-start.sh -daemon ${kafka_home}/config/server.properties"
      if [[ $? -ne 0 ]]; then
          echo "Start kafka on ${i} is OK !"
      fi
  done
  echo kafka kafka are started !
}

stop() {
  for i in $brokers
  do
      echo "Stopping kafka on ${i} ..."
      ssh root@$i "source /etc/profile;sh ${kafka_home}/bin/kafka-server-stop.sh ${kafka_home}/config/server.properties"
      if [[ $? -ne 0 ]]; then
          echo "Stopping ${kafka_home} on ${i} is down"
      fi
  done

  echo all kafka  are stopped !
}

case "$1" in

  start)
    start
  ;;

  stop)
    stop
  ;;

  *)
    echo "Usage: start|stop"
  ;;

esac

5、集群下创建topic

cd /usr/local/soft/kafka/bin;
sh kafka-topics.sh --create --zookeeper 192.168.0.101:2181,192.168.0.102:2181,192.168.0.103:2181 --replication-factor 2 --partitions 3 --topic test1;

查看已经创建的 topic：

sh kafka-topics.sh --list --zookeeper 192.168.0.101:2181,192.168.0.102:2181,192.168.0.103:2181

6、集群下启动Consumer

在一个新的远程窗口中：

sh kafka-console-consumer.sh --bootstrap-server 192.168.0.101:9092,192.168.0.102:9092,192.168.0.103:9092 --topic test1 --from-beginning

7、集群下启动Producer

打开一个新的窗口：

sh kafka-console-producer.sh --broker-list 192.168.0.101:9092,192.168.0.102:9092,192.168.0.103:9092 --topic test1

8、测试发送消息

参考博客：https://gper.club/articles/7e7e7f7ff4g57gcag69