Kafka 2.5.0 集群安装(单机伪集群)
一、安装Zookeeper 3.6.3单节点
1.1、下载解压Zookeeper
-
下载地址
https://zookeeper.apache.org/releases.html
这里下载二进制版本,不需要编译 apache-zookeeper-3.6.3-bin.tar.gz -
在/路径下建立文件夹zookeeper并进入目录进行解压
[root@localhost zookeeper]# tar -zxvf apache-zookeeper-3.6.3-bin.tar.gz
-
重命名解压的目录为zookeeper-3.6.3
[root@localhost zookeeper]# mv apache-zookeeper-3.6.3-bin zookeeper-3.6.3
1.2、建立数据存放目录及日志存放目录
-
存放在CentOS路径参考:
/zookeeper/zookeeper-3.6.3/tmp/data /zookeeper/zookeeper-3.6.3/tmp/logs(酌情选择是否单独建立此路径)
1.3、修改配置文件
-
进入到conf下,复制并重命名配置文件
[root@localhost conf]# cp zoo_sample.cfg zoo.cfg
-
修改 zoo.cfg,修改数据存放目录及新增日志存放目录,及端口号由2181改为2188(非必须)
dataDir=/zookeeper/zookeeper-3.6.3/tmp/data #(酌情选择是否单独建立此路径) dataLogDir=/zookeeper/zookeeper-3.6.3/tmp/logs #可按需选择zk的端口号,默认为2181 clientPort=2188 #可按需选择admin服务的端口号,默认为8080(一般会重复) admin.serverPort=8001
-
如配置zk多点集群,则需要在此配置文件下端写上对应的服务ip地址例如
server.1=192.168.174.25:8880:7770 server.2=192.168.174.33:8880:7770 server.3=192.168.174.36:8880:7770
-
如配置zk多点集群,启动前在data数据路径下执行如下操作,同理在另外两个节点分别设置为’2’和’3’,即分别在三个节点的/zookeeper/zookeeper-3.6.3/tmp/data 目录下创建myid文件,内容为:
第一个节点为 1,第二个节点为 2,第三个节点为 3(根据对应的server.数字配置而定)。[root@local data] echo '1' > /myid
1.4、启动Zookeeper
-
进入到bin目录下,执行启动命令 ./zkServer.sh start
[root@localhost zookeeper-3.6.3]# cd bin/ [root@localhost bin]# ./zkServer.sh start ZooKeeper JMX enabled by default Using config: /zookeeper/zookeeper-3.6.3/bin/../conf/zoo.cfg Starting zookeeper ... STARTED
1.5、查看是否启动成功
-
进入到bin目录下,执行查看状态命令 ./zkServer.sh status
[root@localhost bin]# ./zkServer.sh status ZooKeeper JMX enabled by default Using config: /zookeeper/zookeeper-3.6.3/bin/../conf/zoo.cfg Client port found: 2188. Client address: localhost. Client SSL: false. Mode: standalone
-
或者查看对应zookeeper进程信息
[root@localhost bin]# ps -ef | grep zookeeper
-
或者查看上面配置的2188端口是否被占用
netstart -nplt | grep 2188
1.6、停止Zookeeper
-
进入到bin目录下,执行启动命令
[root@localhost zookeeper-3.6.3]# cd bin/ [root@localhost bin]# ./zkServer.sh stop
1.7、可能遇到的问题
- 启动Zookeeper服务,查看是否启动成功时,如提示如下,则启动不成功
[root@localhost bin]# ps -ef | grep zookeeper
[root@localhost bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /zookeeper/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2188. Client address: localhost. Client SSL: false.
Error contacting service. It is probably not running.
- 去Zookeeper的日志查看失败原因
[root@localhost logs]# cat zookeeper-root-server-localhost.out
org.apache.zookeeper.server.admin.AdminServer$AdminServerException: Problem starting AdminServer on address 0.0.0.0, port 8080 and command URL /commands
at org.apache.zookeeper.server.admin.JettyAdminServer.start(JettyAdminServer.java:189)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:153)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:112)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:67)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:140)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:90)
Caused by: java.io.IOException: Failed to bind to /0.0.0.0:8080
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.server.Server.doStart(Server.java:401)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.zookeeper.server.admin.JettyAdminServer.start(JettyAdminServer.java:180)
... 5 more
Caused by: java.net.BindException: 地址已在使用
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:225)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
... 12 more
Unable to start AdminServer, exiting abnormally
2021-07-06 14:28:48,019 [myid:] - INFO [main:ZKAuditProvider@42] - ZooKeeper audit is disabled.
2021-07-06 14:28:48,021 [myid:] - ERROR [main:ServiceUtils@42] - Exiting JVM with code 4
- 根据错误信息,查看8080端口被占用的情况
[root@localhost logs]# netstat -lnp|grep 8080
tcp6 0 0 :::8080 :::* LISTEN 14485/java
[root@localhost logs]# ps 14485
PID TTY STAT TIME COMMAND
14485 ? Sl 1:13 /usr/local/src/jdk8/bin/java -cp /spark/spark2.4.8/conf/:/spark/spark2.4.8/jars/* -Xmx1g org.apache.spark.deploy.master.Master
- 可以将zoo.cfg中添加了一行配置:admin.serverPort=8001,保存后再重启Zookeeper即可
二、安装Kafka单机多节点伪集群
2.1、环境依赖
-
CentOS7,JDK8,kafka_2.13-2.5.0.tgz,单机启动3个Kafka节点
-
关闭防火墙
#关闭命令 service firewalld stop chkconfig firewalld off
2.2、下载并解压Kafka
-
获取下载地址(点开具体版本):
http://kafka.apache.org/downloads 下载Binary 二进制版本而不是源码 -
在/路径下建立文件夹kafka/cluster并进入目录
[root@localhost /]# mkdir kafka [root@localhost /]# cd kafka [root@localhost kafka]# mkdir cluster [root@localhost kafka]# cd cluster
-
解压在上级kafka目录下的压缩文件到当前目录下
[root@localhost cluster]# tar -zxvf /kafka/kafka_2.13-2.5.0.tgz
-
修改解压后的文件名为kafka2.5
[root@localhost cluster]# mv kafka_2.13-2.5.0 kafka2.5
2.3、创建三个节点的日志文件路径
-
在cluster文件夹下创建三个日志文件目录kafka-logs9093、kafka-logs9094、kafka-logs9095
[root@localhost cluster]# mkdir kafka-logs9093 [root@localhost cluster]# mkdir kafka-logs9094 [root@localhost cluster]# mkdir kafka-logs9095 [root@localhost cluster]# ls kafka2.5 kafka-logs9093 kafka-logs9094 kafka-logs9095
2.4、修改配置文件
-
将server.properties复制三份server9093.properties、server9094.properties、server9095.properties
[root@localhost config]# cp server.properties server9093.properties [root@localhost config]# cp server.properties server9094.properties [root@localhost config]# cp server.properties server9095.properties
-
server9093.properties 的配置修改的地方:
broker.id=3 listeners=PLAINTEXT://192.168.230.132:9093 log.dirs=/kafka/cluster/kafka-logs9093 zookeeper.connect=192.168.230.132:2188
-
server9094.properties 的配置修改的地方:
broker.id=4 listeners=PLAINTEXT://192.168.230.132:9094 log.dirs=/kafka/cluster/kafka-logs9094 zookeeper.connect=192.168.230.132:2188
-
server9095.properties 的配置修改的地方:
broker.id=5 listeners=PLAINTEXT://192.168.230.132:9095 log.dirs=/kafka/cluster/kafka-logs9095 zookeeper.connect=192.168.230.132:2188
2.5、启动3个Kafka服务
-
第一步:启动ZK,再启动kafka。
-
进入到bin路径,启动3个kakfa服务
[root@localhost bin]# ./kafka-server-start.sh -daemon ../config/server9093.properties [root@localhost bin]# ./kafka-server-start.sh -daemon ../config/server9094.properties [root@localhost bin]# ./kafka-server-start.sh -daemon ../config/server9095.properties
-
或者如下命令:
[root@localhost bin]# ./kafka-server-start.sh config/server9093.properties & [root@localhost bin]# ./kafka-server-start.sh config/server9094.properties & [root@localhost bin]# ./kafka-server-start.sh config/server9095.properties &
2.6、集群下创建Topic
-
在bin目录下,利用Zookeeper地址创建一个名为clustertest的topic,只有一个副本,一个分区:
[root@localhost bin]# sh kafka-topics.sh --create --zookeeper 192.168.230.132:2188 --replication-factor 1 --partitions 1 --topic clustertest
-
在bin目录下,利用Kafka地址创建一个名为clustertest01的topic,只有一个副本,一个分区:
[root@localhost bin]# ./kafka-topics.sh --create --topic clustertest01 --bootstrap-server 192.168.230.132:9093,192.168.230.132:9094,192.168.230.132:9095 --partitions 1 --replication-factor 3
-
查看已经创建的 topic: (利用Zookeeper地址)
[root@localhost bin]# sh kafka-topics.sh -list -zookeeper 192.168.230.132:2188
-
查看已经创建的 topic: (利用Kafka地址)
[root@localhost bin]# ./kafka-topics.sh -list --bootstrap-server 192.168.230.132:9093
-
查看具体的某条 topic信息:(利用Kafka地址)
[root@localhost bin]# ./kafka-topics.sh --describe --topic clustertest01 --bootstrap-server 192.168.230.132:9093,192.168.230.132:9094,192.168.230.132:9095
2.7、集群下启动Consumer
-
在bin目录下
[root@localhost bin]# ./kafka-console-consumer.sh --bootstrap-server 192.168.230.132:9093,192.168.230.132:9094,192.168.230.132:9095 --topic clustertest01 --from-beginning
2.8、集群下启动Producer
-
在bin目录下
[root@localhost bin]# ./kafka-console-producer.sh --broker-list 192.168.230.132:9093,192.168.230.132:9094,192.168.230.132:9095 --topic clustertest01
2.9、集群下Producer窗口发送消息,在Consumer窗口验证接收消息
三、示例
3.1、创建一个3个副本的Topic
- 利用zookeeper地址及端口方式 --zookeeper
[root@localhost bin]# sh kafka-topics.sh --create --zookeeper 192.168.230.132:2188 --replication-factor 3 --partitions 1 --topic test-alix
- 利用kafka地址及端口方式 --bootstrap-server
[root@localhost bin]# ./kafka-topics.sh --create --topic test-alix01 --bootstrap-server 192.168.230.132:9093,192.168.230.132:9094,192.168.230.132:9095 --partitions 1 --replication-factor 3
3.2、查看对应的Topic主题信息
[root@localhost bin]# ./kafka-topics.sh --describe --topic test-alix01 --bootstrap-server
192.168.230.132:9093,192.168.230.132:9094,192.168.230.132:9095
3.3、用集群Producer窗口进行消息发送
[root@localhost bin]# sh kafka-console-producer.sh --broker-list 192.168.237.128:9093,192.168.237.128:9094,192.168.237.128:9095 --topic test-alix
>test partitions
>test replication
3.4、分别用3个broker进行Consumer窗口的验证接收消息
[root@localhost bin]# sh kafka-console-consumer.sh --bootstrap-server 192.168.230.132:9093 --topic test-alix --from-beginning
test partitions
test replication
[root@localhost bin]# sh kafka-console-consumer.sh --bootstrap-server 192.168.230.132:9094 --topic test-alix --from-beginning
test partitions
test replication
[root@localhost bin]# sh kafka-console-consumer.sh --bootstrap-server 192.168.230.132:9095 --topic test-alix --from-beginning
test partitions
test replication
3.5、运行kafka自带的wordcount程序
- 创建输入及输出topic
[root@localhost bin]# ./kafka-topics.sh --create --bootstrap-server 192.168.230.132:9093 --topic stream-plaintext-input --partitions 1 --replication-factor 1
[root@localhost bin]# ./kafka-topics.sh --create --bootstrap-server 192.168.230.132:9093 --topic stream-wordcount-output --partitions 1 --replication-factor 1 --config cleanup.policy=compact
-
配置启动对应的WordCountDemo源码
-
分别启动生产者及消费者
[root@localhost bin]# ./kafka-console-producer.sh --bootstrap-server 192.168.230.132:9093 --topic stream-plaintext-input
[root@localhost bin]# ./kafka-console-consumer.sh --bootstrap-server 192.168.230.132:9093 --topic stream-wordcount-output --from-beginning --formatter kafka.tools.DefaultMessageFormatter --property print.key=true --property print.value=true --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer --property value.deserializer=org.apache.kafka.common.serialization.LongDeserializer
- 生产者窗口数据输入相应的字符串即可
四、删除Kafka全部数据
- 停止每台机器上的kafka;
- 删除kafka存储目录 (config/server.properties文件log.dirs配置,默认为"/tmp/kafka-logs")全部topic的数据目录;
- 删除zookeeper上与kafka相关的znode节点;除了/zookeeper;
- 重启kafka。
五、常见问题
-
vim config/server.properties listeners=PLAINTEXT://192.168.237.128:9092 如用了阿里云的云服务器 发现不管本机IP配置内网IP还是外网IP,外网还是无法访问,可以连接,但是监听不了topic 可以改动为: host.name=阿里云内网IP port=9092 advertised.host.name=阿里云外网IP
-
关闭kafka时提示No kafka server to stop
[root@localhost kafka_2.13-2.5.0]# ./bin/kafka-server-stop.sh No kafka server to stop
[root@localhost bin]# vim kafka-server-stop.sh
可以看到PID为
PIDS=$(ps ax | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk '{print $1}')
将PID改为
PIDS=$(jps -lm | grep -i 'kafka.Kafka'| awk '{print $1}')
再执行关闭命令即可。
-
关闭Zookeeper时提示No zookeeper server to stop
在关闭kafka自带的zk时,报如下错误,jps里一直有zk进行
[root@localhost bin]# ./zookeeper-server-stop.sh No zookeeper server to stop [root@localhost kafka_2.13-2.5.0]# jps 4631 QuorumPeerMain 12445 Jps
输入jps -m命令时终于发现了这个QuorumPeerMain线程的来源
也可直接kill掉
kill 9 4631
-
ZooViewer连接不上CentOS中的ZK服务
关闭Centos的防火墙
#关闭命令 service firewalld stop chkconfig firewalld off