docker创建kafka集群

今天我们来创建试着创建一下kafka集群,本次采用但节点的zookeeper注册中心搭建kafka集群,并且未配置kafka-manager.

1 、创建docker-compose的yml文件

创建kafka文件夹,并在目录下编写docker-compose的docker-compose-zoosingle-kafka.yml文件,文件名随便自己取

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"

  kafka1:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.1.202
      KAFKA_CREATE_TOPICS: TestComposeTopic:4:3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
    container_name: kafka01
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  kafka2:
    image: wurstmeister/kafka
    ports:
      - "9093:9093"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.1.202
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 2
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9093
    container_name: kafka02
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  kafka3:
    image: wurstmeister/kafka
    ports:
      - "9094:9094"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.1.202
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 3
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9094
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9094
    container_name: kafka03
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

记得将localhost换成自己宿主机ip

2、启动集群

docker-compose 会根据yml文件,一键启动集群

# docker-compose -f docker-compose-zoosingle-kafka.yml up -d

如图证明启动完毕

[root@VM_0_29_centos kafka]# docker-compose -f docker-compose-zoosingle-kafka.yml up -d
Creating network "kafka_default" with the default driver
Creating kafka01 ... 
Creating kafka03 ... 
Creating kafka02 ... 
Creating zookeeper ... 
Creating kafka03
Creating kafka01
Creating zookeeper
Creating kafka02 ... done

查看容器状态

# docker ps -a

CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                NAMES
929dc994ab44        wurstmeister/kafka       "start-kafka.sh"         14 seconds ago      Up 12 seconds       0.0.0.0:9093->9093/tcp                               kafka02
4abbde5e6a81        wurstmeister/zookeeper   "/bin/sh -c '/usr/sb…"   14 seconds ago      Up 12 seconds       22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   zookeeper
49bf018b39f8        wurstmeister/kafka       "start-kafka.sh"         14 seconds ago      Up 12 seconds       0.0.0.0:9094->9094/tcp                               kafka03
ef68c56ef19a        wurstmeister/kafka       "start-kafka.sh"         14 seconds ago      Up 12 seconds       0.0.0.0:9092->9092/tcp                               kafka01

3、查看kafka容器中查看topic信息

进入kafka01命令执行页面

# cd kafka/broker1
# docker exec -it kafka01 /bin/bash
# cd opt/kafka_2.12-2.4.1/

查看topic信息

# kafka-topics.sh --describe --zookeeper localhost:2181 --topic TestComposeTopic

如下:

Topic: TestComposeTopic	PartitionCount: 4	ReplicationFactor: 3	Configs: 
	Topic: TestComposeTopic	Partition: 0	Leader: 3	Replicas: 3,2,1	Isr: 3,2,1
	Topic: TestComposeTopic	Partition: 1	Leader: 1	Replicas: 1,3,2	Isr: 1,3,2
	Topic: TestComposeTopic	Partition: 2	Leader: 2	Replicas: 2,1,3	Isr: 2,1,3
	Topic: TestComposeTopic	Partition: 3	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2

同理查看kafka03,kafka03 的topic信息

4、开启的生产者模式

开启kafka01的生产者,发送消息

# kafka-console-producer.sh --broker-list localhost:9092 --topic TestComposeTopic

>1234567890
>2
>4
>5
>5
>6
>6
>456
>788

5、开启的消费者模式

开启kafka02,kafka03的消费者,订阅TestComposeTopic的主题,便可看到消费的kafka01的消息了
kafka02端

# kafka-console-consumer.sh --bootstrap-server 106.54.1.220:9093 --topic TestComposeTopic --from-beginning

kafka03端

# kafka-console-consumer.sh --bootstrap-server 106.54.1.220:9094 --topic TestComposeTopic --from-beginning

如下:

# kafka-console-consumer.sh --bootstrap-server 106.54.1.220:9094 --topic TestComposeTopic --from-beginning
2
1234567890
4
5
5
6
6
456
788