Kafka-3.0.1-Docker+集群 踩坑笔记

Kafka-3.2.0-Docker 踩坑笔记

参考：
https://blog.csdn.net/aa2528877987/article/details/123589123
https://hub.docker.com/r/bitnami/kafka
https://blog.csdn.net/hunheidaode/article/details/121401183

备注：Docker+集群

安装与使用

1. 安装Docker

https://blog.csdn.net/iku_whf/article/details/116357886

2. 运行一个zookeeper

docker run -d \
--name zookeeper \
-p 2181:2181 \
-v ~/zookeeper/data:/data \
-v  ~/zookeeper/conf:/conf \
zookeeper:3.4.9

3. 运行以下命令，启动一个kafka，（修改参数后运行即为集群）

-e 环境变量=value 的一些参数，其他见最上面的两个参考

KAFKA_BROKER_ID：brokerID，同一个集群的不能相同
KAFKA_ZOOKEEPER_CONNECT：zookeeper地址
KAFKA_LISTENERS：主动监听的port
KAFKA_ADVERTISED_LISTENERS：外部程序访问 ip+port
ALLOW_PLAINTEXT_LISTENER：允许不加密监听
KAFKA_HEAP_OPTS：JVM参数

docker run -d \
--name kafka0 \
-p 9092:9092 \
-e KAFKA_BROKER_ID=0 \
--net kafka_network  \
-e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://whfc.cc:9092 \
-e ALLOW_PLAINTEXT_LISTENER=yes \
-e KAFKA_HEAP_OPTS="-Xms200m  -Xmx200m" \
bitnami/kafka

根据上一个稍作修改，就有了第二个

docker run -d \
--name kafka1 \
-p 9093:9092 \
-e KAFKA_BROKER_ID=1 \
--net kafka_network  \
-e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://whfc.cc:9093 \
-e ALLOW_PLAINTEXT_LISTENER=yes \
-e KAFKA_HEAP_OPTS="-Xms200m  -Xmx200m" \
bitnami/kafka

报错（各种原因容器没能启动）

1. docker容器创建语句没有ALLOW_PLAINTEXT_LISTENER
   解决：加上ALLOW_PLAINTEXT_LISTENER
   报错：

[root@iZbp18y7b5jm99960ajdloZ ~]# docker start kafka2 -i
kafka 11:42:15.46 
kafka 11:42:15.47 Welcome to the Bitnami kafka container
kafka 11:42:15.47 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-kafka
kafka 11:42:15.47 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-kafka/issues
kafka 11:42:15.47 
kafka 11:42:15.48 INFO  ==> ** Starting Kafka setup **
kafka 11:42:15.58 ERROR ==> The KAFKA_CFG_LISTENERS environment variable does not configure a secure listener. Set the environment variable ALLOW_PLAINTEXT_LISTENER=yes to allow the container to be started with a plaintext listener. This is only recommended for development.
kafka 11:42:15.58 ERROR ==> The KAFKA_ZOOKEEPER_PROTOCOL environment variable does not configure a secure protocol. Set the environment variable ALLOW_PLAINTEXT_LISTENER=yes to allow the container to be started with a plaintext listener. This is only recommended for development.
[root@iZbp18y7b5jm99960ajdloZ ~]# 

2. JVM内存分配太多
   原因：服务器只有2G内存，默认使用1G，两个就不够了
   解决：调小JVM（修改环境变量KAFKA_HEAP_OPTS）
   报错：

# docker start kafka0 -i
日志....................
[2022-06-25 11:18:02,204] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.OutOfMemoryError: Java heap space
        at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:61)
        at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:348)
        at kafka.log.SkimpyOffsetMap.<init>(OffsetMap.scala:45)
        at kafka.log.LogCleaner$CleanerThread.<init>(LogCleaner.scala:301)
        at kafka.log.LogCleaner.$anonfun$startup$2(LogCleaner.scala:156)
        at kafka.log.LogCleaner$$Lambda$382/0x00000001003e4840.apply$mcVI$sp(Unknown Source)
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
        at kafka.log.LogCleaner.startup(LogCleaner.scala:155)
        at kafka.log.LogManager.startupWithConfigOverrides(LogManager.scala:491)
        at kafka.log.LogManager.startup(LogManager.scala:418)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:269)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)
日志....................
[2022-06-25 11:18:02,467] INFO Broker and topic stats closed (kafka.server.BrokerTopicStats)
[2022-06-25 11:18:02,489] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2022-06-25 11:18:02,490] INFO [KafkaServer id=0] shut down completed (kafka.server.KafkaServer)
[2022-06-25 11:18:02,490] ERROR Exiting Kafka. (kafka.Kafka$)
[2022-06-25 11:18:02,518] INFO [KafkaServer id=0] shutting down (kafka.server.KafkaServer)
日志....................

3. JVM内存分配太少
   原因：上面的问题发生后，我改成了100m。内存太小，加载不了，JVM OOM了
   解决：调小JVM（修改环境变量KAFKA_HEAP_OPTS）
   报错：同上