Kafka docker 集群搭建

下载zookeeper和 kafka 镜像:

docker pull wurstmeister/zookeeper
docker pull wurstmeister/kafka

创建一个docker文件夹

mkdir docker

cd docker

目录下创建一个docker-compose.yml文件

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 本地IP
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

启动docker-compose

docker-compose up -d

启动多个kafka 节点，比如3个 自动扩容

docker-compose scale kafka=3

 

 

或者启动的时候一次直接启动三个kafka

docker-compose up --scale kafka=3 -d

进入Kafka容器: 

创建一个topic

bash-4.4# env | grep KAFKA_HOME
KAFKA_HOME=/opt/kafka
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --create --topic test \
> --zookeeper docker_zookeeper_1:2181 --replication-factor 1 --partitions 1
Created topic "test".

查看新创建的topic:

bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --zookeeper docker_zookeeper_1:2181 --describe --topic test
Topic:test  PartitionCount:1    ReplicationFactor:1 Configs:
    Topic: test Partition: 0    Leader: 1003    Replicas: 1003  Isr: 1003

发布消息:

往broker Kafka 2 发送消息，到 kafka 1 检查能不能收到

bash-4.4# $KAFKA_HOME/bin/kafka-console-producer.sh --topic=test \
> --broker-list docker_kafka_2:9092
> send message from kafka 2
>line 2
>^Cbash-4.4#

往 Kafka 3 发送消息

bash-4.4# $KAFKA_HOME/bin/kafka-console-producer.sh --topic=test \
> --broker-list docker_kafka_3:9092
>send from kafka 2 to 3
>check if I can receive from kafka 1
>^Cbash-4.4#

接收消息: (按^C 退出接收)

 到 kafka 1 接收，也可以去其它kafka 接收。

docker exec -it docker_kafka_1  bash

bash-4.4# $KAFKA_HOME/bin/kafka-console-consumer.sh \
> --bootstrap-server root_kafka_1:9092 \
> --from-beginning --topic test
send message from kafka 2
line 2
send from kafka 2 to 3
check if I can receive from kafka 1
^CProcessed a total of 4 messages
bash-4.4#