docker-compose快速部署Kafka，以及常见问题解决

一、概述

• docker镜像
  https://hub.docker.com/r/bitnami/kafka
• github地址
  https://github.com/bitnami/bitnami-docker-kafka

二、服务

不修改端口版

# 包含了zk，采用默认端口2181，9092
$ curl -sSL https://raw.githubusercontent.com/bitnami/bitnami-docker-kafka/master/docker-compose.yml > docker-compose.yml

修改端口版

version: "2"

services:
  zookeeper:
    image: docker.io/bitnami/zookeeper:3.7
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper_data:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: docker.io/bitnami/kafka:3
    ports:
      - "9092:9092"    # CONTAINER_PORT： service之间使用
      - "20222:20222"  # HOST_PORT： docker外使用HOST_PORT访问服务
    volumes:
      - "kafka_data:/bitnami"
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
      - KAFKA_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://:20222
      - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,EXTERNAL://192.168.2.134:20222
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
    depends_on:
      - zookeeper

volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

有Kafka-manager版

Kafka-manager使用docker-compose配置，遇到comsumers没有数据问题，生产不用docker，故不深究。

version: "2"

services:
  zookeeper:
    image: docker.io/bitnami/zookeeper:3.7
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper_data:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: docker.io/bitnami/kafka:3
    ports:
      - "9092:9092"    # CONTAINER_PORT： service之间使用
      - "20222:20222"  # HOST_PORT： docker外使用HOST_PORT访问服务
      - "9999:9999"    # JMX端口
    volumes:
      - "kafka_data:/bitnami"
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
      - KAFKA_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://:20923
      - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,EXTERNAL://192.168.2.134:20923
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
      - JMX_PORT=9999
      - KAFKA_JMX_OPTS=-Djava.rmi.server.hostname=192.168.2.134 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
    depends_on:
      - zookeeper
  kafka-manager:
    image: sheepkiller/kafka-manager
    container_name: kafka-manager
    ports:
      - 9000:9000
    environment:
      ZK_HOSTS: zookeeper

volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

常用命令

# 启动服务
docker-compose up -d
# 查看服务
docker-compose ps
# 停止服务
docker-compose down

# 进入容器
docker exec -it bitnami_kafka_1 /bin/bash
# 默认安装目录
cd /opt/bitnami/kafka
# 查看topic列表
bin/kafka-topics.sh --list --bootstrap-server 127.0.0.1:9092
# 查看消费者列表
bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --list

三、常见错误

docker未启动报错:

[root@insp-server docker]# docker-compose up -d
ERROR: Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running?

If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.

解决：

service docker start

pull不下来

[root@insp-server docker]# docker-compose up -d
Pulling zookeeper (docker.io/bitnami/zookeeper:3.7)...
Trying to pull repository docker.io/bitnami/zookeeper ...
ERROR: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

解决：

# 把默认仓库地址修改为国内地址
vim /etc/docker/daemon.json
{ "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn"] }
# 重启docker
systemctl daemon-reload
systemctl restart docker

启动容器报错

Error response from daemon: oci runtime error: container_linux.go:235: starting container process caused “process_linux.go:258: applying cgroup configuration for process caused “Cannot set property TasksAccounting, or unknown property.””.

解决：
更新一下centos系统版本兼容性问题

yum update

java 端不识别容器名

java.net.UnknownHostException: 2e513323d9bd: nodename nor servname provided, or not known

解决：

vim /etc/hosts
127.0.0.1       localhost
127.0.0.1       2e513323d9bd

java 端获取topic超时

2022-04-07 17:07:38.164 ERROR 42771 --- [           main] o.springframework.kafka.core.KafkaAdmin  : Could not configure topics

org.springframework.kafka.KafkaException: Timed out waiting to get existing topics; nested exception is java.util.concurrent.TimeoutException
# ------------------------
Caused by: java.lang.IllegalStateException: Topic(s) [test-topic] is/are not present and missingTopicsFatal is true

解决：
监听不到topic，默认会报错，该报错大多是由于topic不存在导致。

屏蔽报错：

spring:
  kafka:
    listener:
      missing-topics-fatal: false