kafka集群部署教程

kafka集群部署文档

1.文件清单
文件
说明

apache-zookeeper-3.5.10-bin.tar.gz
zookeeper安装包

kafka_2.12-2.8.2.tgz
kafka安装包

jdk-8u171-linux-x64.tar.gz
jkd8

2.机器资源分配3台 4CPU+8G+ 200GB配置组成kafka高可用集群
生产环境：建议3或者大于3台基数的虚拟机，如果后续是作为公共集群使用，需要更好的配置和更多的机器数量
假如3台机器的IP分别为
10.206.52.10
10.206.21.74
10.206.52.109
JDK的安装不说明
将文件清单上的文件上传到每台服务器的/app目录
3.zookeeper集群部署
在3台服务器下分别按照如果步骤按照启动。
进入/app目录，解压zookeeper，执行：

tar -zxvf apache-zookeeper-3.5.10-bin.tar.gz 

菜单进入conf目录下面,将zoo_sample.cfg复制一份到本目录并改名为zoo.cfg
执行 cp zoo_sample.cfg zoo.cfg
在/app/apache-zookeeper-3.5.10-bin/目录下创建zkdatas文件夹
执行 mkdir zkdatas
执行 echo 1 > zkdatas/myid
注意：myid下的值在3台服务器下需唯一，另外2台则分别写入 2 ，3
编辑zoo.cfg文件

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
# dataDir=/tmp/zookeeper
dataDir=/app/apache-zookeeper-3.5.10-bin/zkdatas
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=1

server.1=10.206.52.10:2888:3888
server.2=10.206.21.74:2888:3888
server.3=10.206.52.109:2888:3888

注意：
①修改红色的位置
②server.后面的1 2 3对应各自服务器下myid的值
启动/停止
分别在3个节点启动/停止：

./bin/zkServer.sh start/stop

#查看zookeeper集群状态

./bin/zkServer.sh status

链接ZK

/app/apache-zookeeper-3.5.10-bin/bin/zkCli.sh -server 10.206.52.10:2181,10.206.21.74:2181,10.206.52.109:2181

4.部署kafka集群
在3个节点分别按如下步骤执行

进入/app目录，解压kafka，执行
tar -zxvf kafka_2.12-2.8.2.tgz
编辑config目录下的server.properties文件,修改红色位置

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
port=9092
host.name=10.206.52.10

############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/app/kafka_2.12-2.8.2/kafkadata/kafka-logs
############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=10.206.52.10:2181,10.206.21.74:2181,10.206.52.109:2181

注意：
①broker.id 是节点唯一ID，3个节点需保持唯一
②host.name是本机IP地址
③zookeeper.connect是zk的链接地址
创建数据存储目录

mkdir -p /app/kafka_2.12-2.8.2/kafkadata/kafka-logs

启动kafka
##启动kafka集群-daemon(以后台服务方式启动) 后面跟的是以配置文件启动
进入bin目录，执行

./kafka-server-start.sh -daemon ../config/server.properties

停止kafka
进入bin目录，执行 ./kafka-server-stop.sh
kafka集群测试验证
①创建topic

/app/kafka_2.12-2.8.2/bin/kafka-topics.sh --create --zookeeper 10.206.52.10:2181,10.206.21.74:2181,10.206.52.109:2181 --replication-factor 1 --partitions 1 --topic my_first_topic

②查看topic 列表：

/app/kafka_2.12-2.8.2/bin/kafka-topics.sh --list --zookeeper 10.206.52.10:2181,10.206.21.74:2181,10.206.52.109:2181

③查看指定topic：

/app/kafka_2.12-2.8.2/bin/kafka-topics.sh  --describe --zookeeper 10.206.52.10:2181,10.206.21.74:2181,10.206.52.109:2181 --topic my_first_topic

④删除topic

/app/kafka_2.12-2.8.2/bin/kafka-topics.sh --delete --zookeeper 10.206.52.10:2181,10.206.21.74:2181,10.206.52.109:2181 --topic my_first_topic