Kafka集群搭建实战
集群结构:
集群环境信息:
192.168.157.130
192.168.157.131
192.168.157.129
Zookeeper集群搭建
-
三台机器上均安装JDK(这里采用压缩包的安装方式
jdk-8u301-linux-x64.tar.gz
)xxx:~#tar -zxvf jdk-8u301-linux-x64.tar.gz # 解压文件 xxx:~#mv jdk1.8.0_301 /opt # 移动文件至 /opt目录下 xxx:~#vim /etc/profile # 编辑 /etc/profile 文件 # jdk export JAVA_HOME=/opt/jdk1.8.0_301 export JRE_HOME=${JAVA_HOME}/jre export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=.:${JAVA_HOME}/bin:$PATH xxx:~#source /etc/profile # 用source命令使新加的配置生效 xxx:~#java -version # java -version 验证java是否配置成功 java version "1.8.0_301" Java(TM) SE Runtime Environment (build 1.8.0_301-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.301-b09, mixed mode)
-
三台机器上均安装Zookeeper用来搭建集群(这里采用压缩包的安装方式
zookeeper-3.4.14.tar.gz
)-
第一台机器上配置
xxx:~#tar -zxf zookeeper-3.4.14.tar.gz # 解压文件 xxx:~#mv zookeeper-3.4.14 /opt # 移动文件至 /opt 目录下 xxx:~#cd /opt/zookeeper-3.4.14/conf # 切换目录至 /opt/zookeeper-3.4.14/conf xxx:/opt/zookeeper-3.4.14/conf#cp zoo_sample.cfg zoo.cfg # 复制 zoo_sample.cfg 配置文件为 zoo.cfg xxx:/opt/zookeeper-3.4.14/conf#vim zoo.cfg # 编辑 zoo.cfg # 设置 dataDir=/var/software/zookeeper/data # 添加 server.1=192.168.157.130:2881:3881 server.2=192.168.157.131:2881:3881 server.3=192.168.157.129:2881:3881 xxx:/opt/zookeeper-3.4.14/conf#mkdir -p /var/software/zookeeper/data # 创建目录/var/software/zookeeper/data xxx:/opt/zookeeper-3.4.14/conf#echo 1 > /var/software/zookeeper/data/myid # 写入 1 至 文件 xxx:~#vim /etc/profile # 编辑 /etc/profile 文件 # zookeeper export ZOOKEEPER_PREFIX=/opt/zookeeper-3.4.14 export PATH=$PATH:$ZOOKEEPER_PREFIX/bin export ZOO_LOG_DIR=/var/software/zookeeper/log xxx:~#source /etc/profile # 用 source 命令使配置生效 xxx:~#scp -r /opt/zookeeper-3.4.14/ 192.168.157.131:/opt # 用 scp 命令远程传输文件至指定主机下的指定目录 xxx:~#scp -r /opt/zookeeper-3.4.14/ 192.168.157.129:/opt # 同上
-
第二台机器上配置
xxx:~#vim /etc/profile # zookeeper export ZOOKEEPER_PREFIX=/opt/zookeeper-3.4.14 export PATH=$PATH:$ZOOKEEPER_PREFIX/bin export ZOO_LOG_DIR=/var/software/zookeeper/log xxx:~#source /etc/profile xxx:~#mkdir -p /var/software/zookeeper/data xxx:~#echo 2 > /var/software/zookeeper/data/myid
-
第三台机器上配置
xxx:~#vim /etc/profile # zookeeper export ZOOKEEPER_PREFIX=/opt/zookeeper-3.4.14 export PATH=$PATH:$ZOOKEEPER_PREFIX/bin export ZOO_LOG_DIR=/var/software/zookeeper/log xxx:~#source /etc/profile xxx:~#mkdir -p /var/software/zookeeper/data xxx:~#echo 3 > /var/software/zookeeper/data/myid
-
启动zookeeper(三台机器均按如下操作)
xxx:~#zkServer.sh start xxx:~#zkServer.sh status
-
Kafka集群搭建
-
三台机器上安装Kafka
-
第一台机器上配置
xxx:~#tar -zvxf kafka_2.12-1.0.2.tgz xxx:~#mv kafka_2.12-1.0.2 /opt xxx:~#scp -r /opt/kafka_2.12-1.0.2/ 192.168.157.131:/opt xxx:~#scp -r /opt/kafka_2.12-1.0.2/ 192.168.157.129:/opt xxx:~#vim /etc/profile # kafka export KAFKA_HOME=/opt/kafka_2.12-1.0.2 export PATH=$PATH:$KAFKA_HOME/bin xxx:~#source /etc/profile xxx:~#vim /opt/kafka_2.12-1.0.2/config/server.properties broker.id=0 listeners=PLAINTEXT://:9092 advertised.listeners=PLAINTEXT://192.168.157.130:9092 log.dirs=/var/software/kafka/log zookeeper.connect=192.168.157.130:2181,192.168.157.131:2181,192.168.157.129:2181/myKafka
-
第二台机器上配置
xxx:~#vim /etc/profile # kafka export KAFKA_HOME=/opt/kafka_2.12-1.0.2 export PATH=$PATH:$KAFKA_HOME/bin xxx:~#source /etc/profile xxx:~#vim /opt/kafka_2.12-1.0.2/config/server.properties broker.id=1 listeners=PLAINTEXT://:9092 advertised.listeners=PLAINTEXT://192.168.157.131:9092 log.dirs=/var/software/kafka/log zookeeper.connect=192.168.157.130:2181,192.168.157.131:2181,192.168.157.129:2181/myKafka
-
第三台机器上配置
xxx:~#vim /etc/profile # kafka export KAFKA_HOME=/opt/kafka_2.12-1.0.2 export PATH=$PATH:$KAFKA_HOME/bin xxx:~#source /etc/profile xxx:~#vim /opt/kafka_2.12-1.0.2/config/server.properties broker.id=2 listeners=PLAINTEXT://:9092 advertised.listeners=PLAINTEXT://192.168.157.129:9092 log.dirs=/var/software/kafka/log zookeeper.connect=192.168.157.130:2181,192.168.157.131:2181,192.168.157.129:2181/myKafka
-
-
启动Kafka(三台机器均按如下操作)
xxx:~#kafka-server-start.sh /opt/kafka_2.12-1.0.2/config/server.properties
-
验证Kafka(查看三台机器的Kafka启动过程中的
Cluster ID
是否一致)-
Cluster Id是一个唯一的不可变的标志符,用于唯一标志一个Kafka集群。
-
该Id最多可以有22个字符组成,字符对应于URL-safe Base64。
-
Kafka 0.10.1版本及之后的版本中,在集群第一次启动的时候,Broker从Zookeeper
的<Kafka_ROOT>/cluster/id节点获取。如果该Id不存在,就自动生成一个新的。
xxx:~#zkCli.sh [zk: localhost:2181(CONNECTED) 2] ls /myKafka/brokers/ids [0, 1, 2] [zk: localhost:2181(CONNECTED) 3] get /myKafka/brokers/ids/0 {"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://192.168.157.130:9092"],"jmx_port":-1,"host":"192.168.157.130","timestamp":"1629859078697","port":9092,"version":4} cZxid = 0x200000097 ctime = Wed Aug 25 10:37:58 CST 2021 mZxid = 0x200000097 mtime = Wed Aug 25 10:37:58 CST 2021 pZxid = 0x200000097 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x200004838ed000a dataLength = 200 numChildren = 0 [zk: localhost:2181(CONNECTED) 4] get /myKafka/brokers/ids/1 {"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://192.168.157.131:9092"],"jmx_port":-1,"host":"192.168.157.131","timestamp":"1629859267652","port":9092,"version":4} cZxid = 0x20000010d ctime = Wed Aug 25 10:41:07 CST 2021 mZxid = 0x20000010d mtime = Wed Aug 25 10:41:07 CST 2021 pZxid = 0x20000010d cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x300004845880007 dataLength = 200 numChildren = 0 [zk: localhost:2181(CONNECTED) 5] get /myKafka/brokers/ids/2 {"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://192.168.157.129:9092"],"jmx_port":-1,"host":"192.168.157.129","timestamp":"1629859117629","port":9092,"version":4} cZxid = 0x2000000f8 ctime = Wed Aug 25 10:38:37 CST 2021 mZxid = 0x2000000f8 mtime = Wed Aug 25 10:38:37 CST 2021 pZxid = 0x2000000f8 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x10000751a660004 dataLength = 200 numChildren = 0
-