Kafka-3.2.0-单体踩坑笔记

inch-whf

已于 2022-07-02 22:44:24 修改

阅读量1.3k

点赞数 1

分类专栏： Java 文章标签： kafka 分布式

于 2022-06-23 21:24:48 首次发布

本文链接：https://blog.csdn.net/iku_whf/article/details/125432794

版权

Java 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

Kafka-3.2.0-单体踩坑笔记

安装与使用
报错

Kafka-3.2.0 踩坑笔记
官网

参考：https://blog.csdn.net/shangyupeng_/article/details/124724774
使用：https://www.jianshu.com/p/39d6ed6b63c8

备注：单体安装，使用自带zookeeper

安装与使用

下载并配置环境变量

下载压缩文件

[root@localhost ~]# wget https://dlcdn.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz --no-check-certificate
--2022-06-23 18:20:48--  https://dlcdn.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz
正在解析主机 dlcdn.apache.org (dlcdn.apache.org)... 151.101.2.132, 2a04:4e42::644
正在连接 dlcdn.apache.org (dlcdn.apache.org)|151.101.2.132|:443... 已连接。
警告: 无法验证 dlcdn.apache.org 的由 “/C=US/O=Let's Encrypt/CN=R3” 颁发的证书:
  颁发的证书已经过期。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：103955943 (99M) [application/x-gzip]
正在保存至: “kafka_2.13-3.2.0.tgz”

 6% [========>                                                                                                                                     ] 6,634,936   74.8KB/s 剩余 41m 36s

在这里插入图片描述
2. 解压

[root@localhost ~]# tar -zxvf kafka_2.13-3.2.0.tgz 
kafka_2.13-3.2.0/
kafka_2.13-3.2.0/LICENSE
kafka_2.13-3.2.0/NOTICE
...

在这里插入图片描述
3. 移动到/usr/local/kafka

[root@localhost ~]# mv kafka_2.13-3.2.0 /usr/local/kafka
[root@localhost ~]# cd  /usr/local/kafka
[root@localhost kafka]# cd  /usr/local
[root@localhost local]# ls
aegis  bin  etc  games  hadoop  hbase  hive  include  kafka  lib  lib64  libexec  sbin  share  src
[root@localhost local]# cd kafka/
[root@localhost kafka]# ls
bin  config  libs  LICENSE  licenses  NOTICE  site-docs

在这里插入图片描述
4. 配置环境变量，/etc/.profile新增

export KAFKA_HOME=/usr/local/kafka
export PATH=$PATH:$KAFKA_HOME/bin

在这里插入图片描述

修改配置文件

修改配置文件

config/zookeeper.properties

dataDir=/usr/local/kafka/zookeeper/data/dataDir
dataLogDir=/usr/local/kafka/zookeeper/data/dataLogDir
clientPort=2181
maxClientCnxns=100
tickTime=2000
initLimit=10

config/server.properties修改

log.dirs=/usr/local/kafka/log

在这里插入图片描述

6. 新建上两步的文件夹，包含一些前面的步骤

[root@localhost local    ]# cd kafka/
[root@localhost kafka    ]# ls
bin  config  libs  LICENSE  licenses  NOTICE  site-docs
[root@localhost kafka    ]# source /etc/profile
[root@localhost kafka    ]# cd config/
[root@localhost config   ]# cp zookeeper.properties zookeeper.properties.bak
[root@localhost config   ]# cd ..
[root@localhost kafka    ]# ls
bin  config  libs  LICENSE  licenses  NOTICE  site-docs
[root@localhost kafka    ]# mkdir log
[root@localhost kafka    ]# mkdir zookeeper
[root@localhost kafka    ]# cd zookeeper/
[root@localhost zookeeper]# mkdir data
[root@localhost zookeeper]# cd data/
[root@localhost data     ]# mkdir dataDir
[root@localhost data     ]# mkdir dataLogDir
[root@localhost data     ]# cd ..
[root@localhost zookeeper]# cd ..

在这里插入图片描述

使用

# 启动（使用自带zookeeper）
kafkaStart.sh
# 启动（使用外面的zookeeper）
kafka-server-start.sh /usr/local/kafka/config/server.properties
# 新建topic
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
# 删除topic
kafka-topics.sh --delete --bootstrap-server localhost:9092 --topic test
# 发布消息
kafka-console-producer.sh --broker-list localhost:9092 --topic test
# 订阅消息
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test [--from-beginning]

-1. 创建启动文件kafka/kafkaStart.sh（启动（使用自带zookeeper）才需要）

#!/bin/bash
#启动zookeeper
/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties &
sleep 3  #默默等3秒后执行
#启动kafka
/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &

报错

运行kafka-topics.sh --create --zookeeper
解决方法：--zookeeper改为 --bootstrap-server

参考：https://blog.csdn.net/qq_29974229/article/details/124277593

报错示例

[root@localhost ~]# kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Exception in thread "main" joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option
        at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
        at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
        at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
        at joptsimple.OptionParser.parse(OptionParser.java:396)
        at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:567)
        at kafka.admin.TopicCommand$.main(TopicCommand.scala:47)
        at kafka.admin.TopicCommand.main(TopicCommand.scala)

新建Topic时，启动进程报错报错：WARN Close of session 0x0 (org.apache.zookeeper.server.NIOServerCnxn)
解决方法：将命令新建Topic 的端口改为 9092
原因：按参考博客修改时，只改了--bootstrap-server，没改localhost:9092

参考：https://blog.csdn.net/qq_29974229/article/details/124277593

[2022-06-23 19:57:23,894] WARN Close of session 0x0 (org.apache.zookeeper.server.NIOServerCnxn)
java.io.IOException: Unreasonable length = 308375649
        at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:166)
        at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:127)
        at org.apache.zookeeper.proto.ConnectRequest.deserialize(ConnectRequest.java:91)
        at org.apache.zookeeper.server.ZooKeeperServer.processConnectRequest(ZooKeeperServer.java:1350)
        at org.apache.zookeeper.server.NIOServerCnxn.readConnectRequest(NIOServerCnxn.java:419)
        at org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:180)
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:339)
        at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
        at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)