docker快速搭建kafka集群测试环境

本地开发过程中需要用到kafka，虚拟机环境搭建太麻烦并且太耗费资源。利用docker结合docker-compose可以快速搭建一套完整的kafka集群环境，非常方便

资源准备：

• CentOS7
• Docker version 19.03.4
• Docker Compose 1.29.2
• [Docker IMAGE] wurstmeister/kafka:latest
• [Docker IMAGE] wurstmeister/zookeeper:latest

Docker Compose 安装

curl -L https://github.com/docker/compose/releases/download/1.29.2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

搭建集群环境

创建一个kafka目录，添加docker-compose文件

version: '3.5'

services:
  zookeeper:  #kafka的信息注册节点
    image: wurstmeister/zookeeper
    ports:
      - 2181:2181
    container_name: zookeeper
  kafka-manager: #kafka的管理页面
    image: sheepkiller/kafka-manager
    depends_on: #依赖于zookeeper服务
      - zookeeper
    environment:
      ZK_HOSTS: zookeeper:2181 #zookeeper指向内部zookeeper的网络ip
    ports:
      - 9000:9000
  kafka0:
    image: wurstmeister/kafka
    depends_on:
      - zookeeper
    container_name: kafka0
    ports:
      - 9092:9092
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka0:9092 #指定网络端口
      KAFKA_LISTENERS: PLAINTEXT://kafka0:9092 #指定网络端口
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 0
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  kafka1:
    image: wurstmeister/kafka
    depends_on:
      - zookeeper
    container_name: kafka1
    ports:
      - 9093:9093
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9093
      KAFKA_LISTENERS: PLAINTEXT://kafka1:9093
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  kafka2:
    image: wurstmeister/kafka
    depends_on:
      - zookeeper
    container_name: kafka2
    ports:
      - 9094:9094
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9094
      KAFKA_LISTENERS: PLAINTEXT://kafka2:9094
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 2
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

注意：

kafka内部会对客户端发送信息的主机和端口做校验，所以宿主机得在host文件中添加kafka0、kafka1、kafka2指向docker，或者内部通过内部代理统一对外暴露ip：port

执行docker-compose up -d创建一组容器

执行docker-compose scale kafka=3将kafka实例扩展到3台

kafka创建topic

在kafka_1节点创建 topic 3个分区 2

docker exec kafka_docker_kafka_1 \
kafka-topics.sh --create \
--topic topic001 \
--partitions 3 \
--zookeeper zookeeper:2181 \
--replication-factor 2

kafka相关参数
查看创建topic的信息

docker exec kafka-docker_kafka_3 \
kafka-topics.sh \
--describe \
--topic topic001 \
--zookeeper zookeeper:2181

查看topic创建信息

消费消息

执行如下命令，即可进入等待topic为topic001消息的状态：

docker exec kafka_kafka_2 \
kafka-console-consumer.sh \
--topic topic001 \
--bootstrap-server \ kafka_kafka_1:9092,kafka_kafka_2:9092,kafka_kafka_3:9092

生产消息

打开一个新的窗口，执行如下命令，进入生产消息的命令行模式，注意不要漏掉参数"-it"，我之前就是因为漏掉了参数"-it"，导致生产的消息时虽然不提示异常，但是始终无法消费到消息：

docker exec -it kafka_kafka_1 \
kafka-console-producer.sh \
--topic topic001 \
--broker-list kafka_kafka_1:9092,kafka_kafka_2:9092,kafka_kafka_3:9092

生产端发送消息：消费端接收到消息：
致此，docker搭建kafka集群完成