kafka3.3.1集群安装(KRaft模式|Zookeeper模式)

版本：Centos7、kafka_2.13-3.3.1
节点：
27.196.38.200 kafka1
27.196.38.201 kafka2
27.196.38.202 kafka3

基础配置

0、安装JDK1.8 (略)
1、设置主机名和安全配置
在所有节点操作

[root@localhost ~]# hostnamectl set-hostname kafka1（每个节点单独操作）
[root@localhost ~]# cat >> /etc/hosts << EOF
> 27.196.38.200 kafka1
> 27.196.38.201 kafka2
> 27.196.38.202 kafka3
> EOF

[root@localhost ~]# systemctl stop firewalld
[root@localhost ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@localhost ~]# setenforce 0
[root@localhost ~]# sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

2、设置免密通信
在kafaka1上操作：

ssh-keygen -t rsa
ssh-copy-id -i kafka2
ssh-copy-id -i kafka3

在kafka2上操作：

ssh-keygen -t rsa
ssh-copy-id -i kafka1
ssh-copy-id -i kafka3

在kafka3上操作：

ssh-keygen -t rsa
ssh-copy-id -i kafka1
ssh-copy-id -i kafka2

Kafka配置（KRaft模式）

1、安装

tar -zxvf kafka_2.13-3.3.1.tar
mv kafka_2.13-3.3.1 /usr/local/kafka
mkdir -p /data/kafka3

2、配置
修改config/kraft/server.properties配置文件

vi /usr/local/kafka/config/kraft/server.properties

kafka1配置：

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@kafka1:9093,2@kafka2:9093,3@kafka3:9093
listeners=PLAINTEXT://kafka1:9092,CONTROLLER://kafka1:9093
advertised.listeners=PLAINTEXT://kafka1:9092
log.dirs=/data/kafka3

kafka2配置：

process.roles=broker,controller
node.id=2
controller.quorum.voters=1@kafka1:9093,2@kafka2:9093,3@kafka3:9093
listeners=PLAINTEXT://kafka2:9092,CONTROLLER://kafka2:9093
advertised.listeners=PLAINTEXT://kafka2:9092
log.dirs=/data/kafka3

kafka3配置：

process.roles=broker,controller
node.id=3
controller.quorum.voters=1@kafka1:9093,2@kafka2:9093,3@kafka3:9093
listeners=PLAINTEXT://kafka3:9092,CONTROLLER://kafka3:9093
advertised.listeners=PLAINTEXT://kafka3:9092
log.dirs=/data/kafka3

3、生成集群UUID
在一个节点（如kafka1）执行：

[root@localhost kafka]# /usr/local/kafka/bin/kafka-storage.sh random-uuid
L55Y6-yYQhucxmWS7sHBlw

4、初始化元数据
在所有节点执行：

[root@localhost kafka]# /usr/local/kafka/bin/kafka-storage.sh format -t L55Y6-yYQhucxmWS7sHBlw -c /usr/local/kafka/config/kraft/server.properties 
Formatting /data/kafka3 with metadata.version 3.3-IV3.

5、启动kafka
在3个节点依次执行：

 /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/kraft/server.properties &

在/usr/local/kafka/logs/server.log可以看到启动成功的日志：

 INFO [KafkaRaftServer nodeId=1] Kafka Server started (kafka.server.KafkaRaftServer)

6、停止命令

bin/kafka-server-stop.sh

Kafka配置（zookeeper方式）

1、安装

tar -zxvf kafka_2.13-3.3.1.tar
mv kafka_2.13-3.3.1 /usr/local/kafka
mkdir -p /data/kafka
mkdir -p /data/zkdata

2、配置
1）修改config/zookeeper.properties配置文件

vi /usr/local/kafka/config/zookeeper.properties

kafka1、kafka2、kafka3配置相同：

dataDir=/data/zkdata
Time=2000
initLimit=10
syncLimit=5
server.1=27.196.38.200:2888:3888
server.2=27.196.38.201:2888:3888
server.3=27.196.38.202:2888:3888

新建对应不同的server的myid，和zookeeper.properties中的值相同:
kafka1:

echo 1 >/data/zkdata/myid

kafka2:

echo 2 >/data/zkdata/myid

kafka3:

echo 3 >/data/zkdata/myid

2）修改config/server.properties配置文件

vi /usr/local/kafka/config/server.properties

kafka1配置：

broker.id=0
listeners=PLAINTEXT://kafka1:9092
delete.topic.enable=true
log.dirs=/data/kafka
zookeeper.connect=kafka1:2181,kafka2:2181,kafka3:2181

kafka2配置：

broker.id=1
listeners=PLAINTEXT://kafka2:9092
delete.topic.enable=true
log.dirs=/data/kafka
zookeeper.connect=kafka1:2181,kafka2:2181,kafka3:2181

kafka3配置：

broker.id=2
listeners=PLAINTEXT://kafka3:9092
delete.topic.enable=true
log.dirs=/data/kafka
zookeeper.connect=kafka1:2181,kafka2:2181,kafka3:2181

3）启动zookeeper：

/usr/local/kafka/bin/zookeeper-server-start.sh -daemon /usr/local/kafka/config/zookeeper.properties &

4）启动kafka：

/usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties &

5）创建Topic：

/usr/local/kafka/bin/kafka-topics.sh --create --bootstrap-server kafka1:9092 --replication-factor 3 --partitions 3 --topic cicdlog

kafka停止命令：/usr/local/kafka/bin/kafka-server-stop.sh
zookeep停止命令：/usr/local/kafka/bin/zookeeper-server-stop.sh

kafka manager安装

27.196.38.220
目前kafka manager不支持kraft方式的kafka集群

1、安装JDK11，cmak3.0.0.6依赖JDK11。

tar -zxvf jdk-11.0.16.1_linux-x64_bin.tar.gz
mv jdk-11.0.16.1 /usr/local/

不配置环境环境，以免影响其它应用。
2、安装cmak
解压cmak-3.0.0.6.zip，拷贝到/usr/local/目录下
修改配置文件

vi /usr/local/cmak-3.0.0.6/conf/application.conf
application.home=/usr/local/cmak-3.0.0.6
cmak.zkhosts="27.196.38.200:2181,27.196.38.201:2181,27.196.38.202:2181"

后台启动

chmod -R 777 /usr/local/cmak-3.0.0.6
nohup /usr/local/cmak-3.0.0.6/bin/cmak -java-home /usr/local/jdk-11.0.16.1 > /tmp/cmaklog.txt 2>&1 &

重启需要删掉 /data/service/cmak-3.0.0.6/RUNNING_PID 文件
访问地址：http://27.196.38.220:9000