docker 下 zookeeper集群、kafka集群 安装

目录

docker网络环境设置

​规划网络和各个服务

安装zookeeper集群

kafka集群搭建

​测试集群是否正常

---

docker网络环境设置

关于docker下相关的网络操作不是本章的重点，所以这边只会将对应的版本和对应的子网配置列出来方便各位了解到对应的环境信息

docker 19.03.7

docker-compose 1.24.0

这边将应用到 default-network 子网络对应的配置可以见下下面的信息:

规划网络和各个服务

IP	端口映射	docker image	hostname	说明
192.168.10.66	41666:9090	registry.cn-shenzhen.aliyuncs.com/wunaozai/zkui:latest	zookeeper-cluster-ui	zookeeper 管理UI界面
192.168.10.67	41667:2181,41668:22,41669:2888,41670:3888	zookeeper:latest	zookeeper-cluster-1	zookeeper 服务器1
192.168.10.68	41671:2181,41672:22,41673:2888,41674:3888	zookeeper:latest	zookeeper-cluster-2	zookeeper 服务器2
192.168.10.69	41675:2181,41676:22,41677:2888,41678:3888	zookeeper:latest	zookeeper-cluster-3	zookeeper 服务器3
192.168.10.71	41681:9092	wurstmeister/kafka:latest	kafka-cluster-1	kafka 服务器1
192.168.10.72	41682:9092	wurstmeister/kafka:latest	kafka-cluster-2	kafka 服务器2
192.168.10.73	41683:9092	wurstmeister/kafka:latest	kafka-cluster-3	kafka 服务器3

 

安装zookeeper集群

docker-compose.yml 相关配置如下:

version: '3.4'

services:
  # zookeeper集群相关三台服务器的配置
  zookeeper-cluster-1:
    image: zookeeper:latest
    restart: "no"
    # restart: always
    hostname: zookeeper-cluster-1
    container_name: zookeeper-cluster-1
    # 按照前面网络规划的设置好对应的映射关系
    ports:
      - "41667:2181"
      - "41668:22"
      - "41669:2888"
      - "41670:3888"
    # 集群相关的环境变量设置，官网有标注（下面会贴出对应的官网标注）
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: "server.1=0.0.0.0:2888:3888;2181 server.2=zookeeper-cluster-2:2888:3888;2181 server.3=zookeeper-cluster-3:2888:3888;2181"
      # 这里应用到zkui权限不足的问题
      ZOO_4LW_COMMANDS_WHITELIST: "*"
    # 设置为固定IPV4
    networks:
      default-network:
        ipv4_address: 192.168.10.67
        
  zookeeper-cluster-2:
    image: zookeeper:latest
    restart: "no"
    # restart: always
    hostname: zookeeper-cluster-2
    container_name: zookeeper-cluster-2
    ports:
      - "41671:2181"
      - "41672:22"
      - "41673:2888"
      - "41674:3888"
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: "server.1=zookeeper-cluster-1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zookeeper-cluster-3:2888:3888;2181"
      ZOO_4LW_COMMANDS_WHITELIST: "*"
    networks:
      default-network:
        ipv4_address: 192.168.10.68
        
  zookeeper-cluster-3:
    image: zookeeper:latest
    restart: "no"
    # restart: always
    hostname: zookeeper-cluster-3
    container_name: zookeeper-cluster-3
    ports:
      - "41675:2181"
      - "41676:22"
      - "41677:2888"
      - "41678:3888"
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: "server.1=zookeeper-cluster-1:2888:3888;2181 server.2=zookeeper-cluster-2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181"
      ZOO_4LW_COMMANDS_WHITELIST: "*"
    networks:
      default-network:
        ipv4_address: 192.168.10.69

  zookeeper-cluster-ui:
    image: registry.cn-shenzhen.aliyuncs.com/wunaozai/zkui:latest
    restart: "no"
    hostname: zookeeper-cluster-ui
    container_name: zookeeper-cluster-ui
    ports:
      - "41666:9090"
    environment:
      ZK_SERVER: "zookeeper-cluster-1:2181,zookeeper-cluster-2:2181,zookeeper-cluster-3:2181"
    external_links:
      - "zookeeper-cluster-1"
      - "zookeeper-cluster-2"
      - "zookeeper-cluster-3"
    networks:
      default-network:
        ipv4_address: 192.168.10.66

# 外部载入设置的子网络 "default-network"
networks:
  default-network:
    external: true

启动和运行容器:

docker-compose up -d

查看已经启动的容器:

docker ps

到此处你的zookeeper集群就已经安装好了.我们打开下zookeeper-ui的可视化管理界面看一下：

 http://127.0.0.1:41666/login

输入红色方框的其中任意一组账号密码后你就可以进入到对应的管理界面了。

kafka集群搭建

version: '3.4'


services:
  kafka-cluster-1:
    image: wurstmeister/kafka:latest
    restart: "no"
    # restart: always
    hostname: kafka-cluster-1
    container_name: kafka-cluster-1
    ports:
      - "41681:9092"
    environment:
      # zookeeper的连接地址
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper-cluster-1:2181,zookeeper-cluster-2:2181,zookeeper-cluster-3:2181"
      # 当前主机的主机名
      KAFKA_ADVERTISED_HOST_NAME: "192.168.10.71"
      # broker id
      KAFKA_BROKER_ID: 1
    networks:
      default-network:
        ipv4_address: 192.168.10.71
        
  kafka-cluster-2:
    image: wurstmeister/kafka:latest
    restart: "no"
    # restart: always
    hostname: kafka-cluster-2
    container_name: kafka-cluster-2
    ports:
      - "41682:9092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper-cluster-1:2181,zookeeper-cluster-2:2181,zookeeper-cluster-3:2181"
      KAFKA_ADVERTISED_HOST_NAME: "192.168.10.72"
      KAFKA_BROKER_ID: 2
    networks:
      default-network:
        ipv4_address: 192.168.10.72
        
  kafka-cluster-3:
    image: wurstmeister/kafka:latest
    restart: "no"
    # restart: always
    hostname: kafka-cluster-3
    container_name: kafka-cluster-3
    ports:
      - "41683:9092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper-cluster-1:2181,zookeeper-cluster-2:2181,zookeeper-cluster-3:2181"
      KAFKA_ADVERTISED_HOST_NAME: "192.168.10.73"
      KAFKA_BROKER_ID: 3
    networks:
      default-network:
        ipv4_address: 192.168.10.73

        
networks:
  default-network:
    external: true

同样的启动运行：

docker-compose up -d

查看容器状态：

docker ps

打开zkui可以看到当前的控制节点id(可能要等一下下):

安装完成后查看当前节点下的zookeeper数据就会有对应的kafka broker信息：

测试集群是否正常

首先查看下kafka的安装位置：

docker inspect kafka-cluster-1 |grep KAFKA_HOME

连接进入容器内：

docker exec -it kafka-cluster-1 /bin/bash

在/opt/kafka/bin 底下就是一堆的kafka使用工具了：

1. 创建一个主题复制系数3、分区数量4； 然后查看下当前的信息

kafka-topics.sh --zookeeper zookeeper-cluster-1:2181 --create --topic test_topics --replication-factor 3 --partitions 4
kafka-topics.sh --zookeeper zookeeper-cluster-1:2181 --describe

2. 启用一个消费者(当前是有一个默认消费者组、默认好像是不提交偏移量的；忽略下面的sssdddfff)

kafka-console-consumer.sh --bootstrap-server kafka-cluster-1:9092 --topic test_topics

3. 查看消费者分组信息（同一分组下一个分区最多只能一个消费者）

kafka-consumer-groups.sh --bootstrap-server kafka-cluster-1:9092 --list
kafka-consumer-groups.sh --bootstrap-server kafka-cluster-1:9092 --describe --group console-consumer-21570

4. 启用一个生产者测试发生消息（任一台kafka broker都可以的）

kafka-console-producer.sh --bootstrap-server 192.168.10.73:9092 --topic test_topics

5. 接下来回过来看消费者端，消费了刚刚产生的数据

参考资料：

1.  zookeeper docker 配置信息资料 【https://hub.docker.com/_/zookeeper/】
2. 关于zkui权限不足的相关资料 ZOO_4LW_COMMANDS_WHITELIST
   1. 产生原因 【https://blog.csdn.net/x763795151/article/details/80599498】
   2. 官方文档对参数说明【https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_clusterOptions】
   3. docker 环境变量配置参数 【https://hub.docker.com/_/zookeeper/】
3. kafka镜像相关配置手册 【https://github.com/wurstmeister/kafka-docker】