kafka集群和efak图形化监控页面
安装
zookeeper安装
此篇介绍的是kafka2.6.0版本,需要依赖zookeeper
- zookeeper下载
点此下载3.5.10 - 创建zookeeper文件夹,将压缩包解压出来,复制三份到zookeeper下,分别命名为zk1、zk2、zk3。进入conf文件夹下,查看是否有
zoo.cfg
,如没有,复制zoo_sample.cfg文件即可。 - 修改配置文件如下,依次修改三份配置文件:
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
#zookeeper数据以及日志文件目录
dataDir=D:/zookeeper/data/zk1
dataLogDir=D:/zookeeper/log/zk1
# the port at which the clients will connect
#此处为zookeeper对外暴露端口
clientPort=2187
#此处为zookeeper集群选举以及内部通信端口
server.1=localhost:2887:3887
server.2=localhost:2888:3888
server.3=localhost:2889:3889
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
- 在上述三个zookeeper的data数据文件夹下增加myid文件(注意不带后缀名)依次添加1、2、3
- 进入bin目录下,依次执行zkServer.cmd
kafka安装
- kafka下载
点此下载kafka_2.13-2.6.0 - 创建kafka文件夹,将压缩包解压出来,复制三份到kafka下,分别命名为k1、k2、k3。进入config文件夹下。
- 修改server.properties配置文件如下:
#每个集群服务的id
broker.id=0
broker.list=192.168.1.2:9097,192.168.1.2:9098,192.168.1.2:9099
#监听的端口,外部连接的时候需要与此ip地址一致
listeners=PLAINTEXT://192.168.1.2:9097
advertised.listeners=PLAINTEXT://192.168.1.2:9097
host.name=192.168.1.2
#每个服务对外暴露的端口
port=9097
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=D:\kafka\log\k1
#副本数需要进行设置,默认为1的话,会存在单点故障的问题
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
#zookeeper集群服务地址
zookeeper.connect=localhost:2187,localhost:2188,localhost:2189
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0
- 依次在bin/windows文件夹下进入cmd黑窗口模式执行
kafka-server-start.bat ../../config/server.properties
efak下载安装
efak最新版本3.0.1不知道为什么一直在windows上运行出问题,在linux系统上运行没问题。所以这里介绍的是在linux上部署efak
-
下载
官网地址 -
需要解压两次,进入config文件夹下
-
需要修改的配置如下:
#配置zookeeper服务地址
efak.zk.cluster.alias=cluster1
cluster1.zk.list=192.168.1.2:2187,192.168.1.2:2188,192.168.1.2:2189
cluster1.efak.offset.storage=kafka
#cluster2.efak.offset.storage=zk
#配置数据库
efak.driver=com.mysql.cj.jdbc.Driver
efak.url=jdbc:mysql://192.168.1.2:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
efak.username=root
efak.password=123456
- 修改kafka启动参数kafka-server-start.bat
IF ["%KAFKA_HEAP_OPTS%"] EQU [""] (
rem detect OS architecture
wmic os get osarchitecture | find /i "32-bit" >nul 2>&1
IF NOT ERRORLEVEL 1 (
rem 32-bit OS
set KAFKA_HEAP_OPTS=-Xmx512M -Xms512M
) ELSE (
rem 64-bit OS
rem set KAFKA_HEAP_OPTS=-Xmx1G -Xms1G
set KAFKA_HEAP_OPTS=-Xms2G -Xmx2G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=70
set JMX_PORT=9987
)
)
注意这里是windows版本,linux需要在kafka-server-start.sh里进行修改
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
export KAFKA_HEAP_OPTS="-server -Xms2G -Xmx2G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=70"
export JMX_PORT="9999"
fi
- 进入bin目录下启动efak
./ke.sh start