docker-compose部署kafka服务时如何同时允许内外网访问？

背景

最近在学习kafka相关知识，需要搭建自己的kafka环境。综合考虑后决定使用docker-compose来管理维护这个环境。

docker-compose.yml

Bitnami的yml文件就很不错，这里直接拿来用了。

version: "2"

services:
  zookeeper:
    image: docker.io/bitnami/zookeeper:3.8
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper_data:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: docker.io/bitnami/kafka:3.4
    ports:
      - "9092:9092"
    volumes:
      - "kafka_data:/bitnami"
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

docker-compose初体验

下载或创建上述yml文件后，启动：

$ ls
docker-compose.yml

$ docker-compose up -d
Creating network "kafka_default" with the default driver
Pulling kafka (docker.io/bitnami/kafka:3.4)...
3.4: Pulling from bitnami/kafka
55154658374f: Pull complete
Digest: sha256:659549c08f8a1cfce344d31b608ec2d039a66a9b610423c4bc390c486a8cebbd
Status: Downloaded newer image for bitnami/kafka:3.4
Creating kafka_zookeeper_1 ... done
Creating kafka_kafka_1     ... done

$ docker ps -a       
CONTAINER ID   IMAGE                   COMMAND                  CREATED         STATUS         PORTS                                                  NAMES
4fe1da00ef96   bitnami/kafka:3.4       "/opt/bitnami/script…"   2 minutes ago   Up 2 minutes   0.0.0.0:9092->9092/tcp                                 kafka_kafka_1
bc018108f95e   bitnami/zookeeper:3.8   "/opt/bitnami/script…"   2 minutes ago   Up 2 minutes   2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, 8080/tcp   kafka_zookeeper_1

服务启动后，可以进入容器访问kafka服务

$ docker exec -it 4fe1 /bin/bash

允许外部访问

使用我们的java客户端连接上述docker容器中的kafka服务，结果却一直报错

Error connecting to node 4fe1da00ef96:9092 (id: 1001 rack: null)
java.net.UnknownHostException: 4fe1da00ef96: nodename nor servname provided, or not known

这是因为默认情况下，kafka服务只允许本机连接。

更新docker-compose.yml，开放外部访问。

• 添加环境变量

    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
+     - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
+     - KAFKA_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://:9093
+     - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,EXTERNAL://localhost:9093
+     - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT

• 暴露端口

    ports:
-     - '9092:9092'
+     - '9093:9093'

停止和删除容器，并重新启动新容器

$ docker-compose down
Stopping kafka_kafka_1     ... done
Stopping kafka_zookeeper_1 ... done
Removing kafka_kafka_1     ... done
Removing kafka_zookeeper_1 ... done
Removing network kafka_default

$ docker-compose up -d
Creating network "kafka_default" with the default driver
Creating kafka_zookeeper_1 ... done
Creating kafka_kafka_1     ... done

$ docker ps -a
CONTAINER ID   IMAGE                   COMMAND                  CREATED          STATUS          PORTS                                                  NAMES
13ba5ed71886   bitnami/kafka:3.4       "/opt/bitnami/script…"   24 seconds ago   Up 22 seconds   0.0.0.0:9092-9093->9092-9093/tcp                       kafka_kafka_1
e4ca7940e7ab   bitnami/zookeeper:3.8   "/opt/bitnami/script…"   25 seconds ago   Up 24 seconds   2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, 8080/tcp   kafka_zookeeper_1

更新hosts

查看宿主机IP信息，将docker-compose管理的kafka服务名称映射到当前主机IP

$ ifconfig | grep 192 
	inet 192.168.10.241 netmask 0xffffff00 broadcast 192.168.10.255

$ cat /etc/hosts
# docker
192.168.10.241 kafka

此时，我们可以在java代码中连接这个kafka服务了。

properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9093");
properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.10.241:9092");
properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.10.241:9093");

测试发现这几种方式均可连接。这是为什么呢？欢迎留言分享。

另一种参数配置方式：

• - KAFKA_CFG_LISTENERS=PLAINTEXT://0.0.0.0:9092
  

• - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://192.168.77.159:9092