zookeeper、kafka集群配置

一、准备

查看系统版本

 cat /etc/redhat-release

准备3台机器A、B、C
确保安装jdk环境
选择合适版本并下载
kafka下载地址
zookeeper下载地址：

二、开始安装

1.zookeeper集群

1.解压文件

# 创建目录
mkdir -p /home/kafka-zookeeper
# 解压
tar -zxvf apache-zookeeper-3.6.1-bin.tar.gz
# 重命名
mv apache-zookeeper-3.6.1-bin zookeeper-3.6.1

2.修改配置文件（3台）

# 1.创建目录
mkdir -p /home/kafka-zookeeper/zookeeper-3.6.1/{data,logs}
# 修改配置文件名
cd /home/kafka-zookeeper/zookeeper-3.6.1/conf
mv zoo_sample.cfg zoo.cfg

3.修改数据路径、端口及集群列表

$ vim zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/kafka-zookeeper/zookeeper-3.6.1/data
dataLogDir=/home/kafka-zookeeper/zookeeper-3.6.1/logs
clientPort=2181
server.1=Aip:2888:3888
server.2=Bip:2888:3888
server.3=Cip:2888:3888
##集群服务器列表，几台服务器集群就配几个，此处的1,2,3(使用ip尾号也可)是服务器对应的id

4.创建myid

cd /home/kafka-zookeeper/zookeeper-3.6.1/data
touch myid
# 将server.1对应的1写入myid中（server.2就是2），每台服务器都执行这个操作，id和服务器不能对应错，分别执行。
echo 1 > /home/kafka-zookeeper/zookeeper-3.6.1/data/myid
# 其他服务器依次：
echo 2 > /home/kafka-zookeeper/zookeeper-3.6.1/data/myid
echo 3 > /home/kafka-zookeeper/zookeeper-3.6.1/data/myid

5.zookeeper启动、停止

# 命令
/home/kafka-zookeeper/zookeeper-3.6.1/bin/zkServer.sh [--config <conf-dir>] {start|start-foreground|stop|restart|status|print-cmd}
# 启动
/home/kafka-zookeeper/zookeeper-3.6.1/bin/zkServer.sh start
# 停止
/home/kafka-zookeeper/zookeeper-3.6.1/bin/zkServer.sh stop
# 查看状态
/home/kafka-zookeeper/zookeeper-3.6.1/bin/zkServer.sh status
# 3台机器状态
mode: leader	# 主
mode: follower	# 从
mode: follower	# 从
# 运行客户端 (后面介绍查看kafka注册)
/home/kafka-zookeeper/zookeeper-3.6.1/bin/zkCli.sh -server Aip:2181

到这里就配置完成了，如果查看zookeeper状态没有启动，查看logs下日志文件-失败原因

2.kafka集群

1.解压

cd /home/kafka-zookeeper
tar -zxvf kafka_2.12-2.5.0.tgz

2.修改配置（3台）

cd /home/kafka-zookeeper/kafka_2.12-2.5.0
mkdir -p {data,logs}

vim config/server.properties
# A
broker.id=1
listeners=PLAINTEXT://Aip:9092
num.network.threads=12
num.io.threads=24
log.dirs=/home/kafka-zookeeper/kafka_2.12-2.5.0/logs
num.partitions=3
num.recovery.threads.per.data.dir=12
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3
zookeeper.connect=Aip:2181,Bip:2181,Cip:2181
# 其他两台配置
# B
broker.id=2
listeners=PLAINTEXT://Bip:9092
log.dirs=/home/kafka-zookeeper/kafka_2.12-2.5.0/logs
num.partitions=3
zookeeper.connect=Aip:2181,Bip:2181,Cip:2181
# C
broker.id=3
listeners=PLAINTEXT://Cip:9092
log.dirs=/home/kafka-zookeeper/kafka_2.12-2.5.0/logs
num.partitions=3
zookeeper.connect=Aip:2181,Bip:2181,Cip:2181

3.启动kafka集群

首先确保zookeeper是启动的

# 启动kafka
/home/kafka-zookeeper/kafka_2.12-2.5.0/bin/kafka-server-start.sh /home/kafka-zookeeper/kafka_2.12-2.5.0/config/server.properties &
或者：
nohup /home/kafka-zookeeper/kafka_2.12-2.5.0/bin/kafka-server-start.sh /home/kafka-zookeeper/kafka_2.12-2.5.0/config/server.properties 1>/dev/null 2>&1 &

三、验证集群

验证kafka

cd /home/kafka-zookeeper/kafka_2.12-2.5.0
# 创建kafka topic test
bin/kafka-topics.sh --create --bootstrap-server Aip:9092 --replication-factor 3 --partitions 3 --topic test
# 列出所有topic
bin/kafka-topics.sh --list --bootstrap-server Aip:9092

# 发布消息
bin/kafka-console-producer.sh --broker-list Aip:9092 --topic test
>1111
>2222

# 在B机器消费
bin/kafka-console-consumer.sh --bootstrap-server Bip:9092 --topic test --from-beginning
1111
2222

# 查看topic信息
bin/kafka-topics.sh --describe --zookeeper Aip:22181 --topic test

# 删除topic
bin/kafka-topics.sh --bootstrap-server Aip:9092 --delete --topic test

验证zookeeper

cd /home/kafka-zookeeper/zookeeper-3.6.1
# 进入zookeeper客户端，如果自定义端口 一定要-server 指定ip端口，否则进入默认2181
bin/zkCli.sh -server Aip:2181
# 查看服务
ls /brokers/ids
[1,2,3]
ls /brokers/topics
[__consumer_offsets, test]

为了以后的自己-好好学习