前言
最近在研究kafka与rabbitMQ对比,而因为自己电脑性能不是那么好,要搭建一个kafka集群和zookeeper集群最少也要开三台虚拟机,而且也挺麻烦,所以毋庸置疑选择了docker-compose来搭建一个伪分布式的集群。参考了很多资料,很多乱七八糟都是转发的,真正去研究的很少,所以在这里记录一下自己的搭建过程
软件版本
- docker:
Docker version 1.13.1, build b2f74b2/1.13.1
- docke-compose:
docker-compose version 1.23.2, build 1110ad0
- docker_ zookeeper_image :
zookeeper:3.5.7
- docker_ kafka_iamge:
wurstmeister/kafka:latest
- docker_ kafka-manager_image:
sheepkiller/kafka-manager:latest
Compose file format compatibility matrix
Compose file format | Docker Engine |
---|---|
1 | 1.9.0+ |
2.0 | 1.10.0+ |
2.1 | 1.12.0+ |
2.2, 3.0, 3.1, 3.2 | 1.13.0+ |
2.3, 3.3, 3.4, 3.5 | 17.06.0+ |
2.4 | 17.12.0+ |
3.6 | 18.02.0+ |
3.7 | 18.06.0+ |
怎么安装的在这里就不多赘述了,相信必应、百度都有很多
准备
如果你对防火墙不感冒,那么在测试的时候建议你关闭防火墙;反之请开放如下端口
宿主机防火墙开放:9000/tcp 、2181/tcp、2182/tcp、2183/tcp、9092/tcp、9093/tcp、9094/tcp端口
Kafka是依赖zookeeper来存放一些数据的,所以kafka集群的启动必须依赖zookeeper集群
docker-compose.yml
version: '2'
services:
zookeeper1:
image: zookeeper:3.5.7
hostname: zookeeper1
container_name: zookeeper1
privileged: true
ports:
- "2181:2181"
volumes:
- ./zookeeper1/conf:/conf
- ./zookeeper1/data:/data
- ./zookeeper1/datalog:/datalog
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=zookeeper2:2888:3888;2181 server.3=zookeeper3:2888:3888;2181
networks:
net_zoo_kafka:
ipv4_address: 172.23.0.11
zookeeper2:
image: zookeeper:3.5.7
hostname: zookeeper2
container_name: zookeeper2
privileged: true
ports:
- "2182:2181"
volumes:
- ./zookeeper2/conf:/conf
- ./zookeeper2/data:/data
- ./zookeeper2/datalog:/datalog
environment:
ZOO_MY_ID: 2
ZOO_SERVERS: server.1=zookeeper1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zookeeper3:2888:3888;2181
networks:
net_zoo_kafka:
ipv4_address: 172.23.0.12
zookeeper3:
image: zookeeper:3.5.7
hostname: zookeeper3
container_name: zookeeper3
privileged: true
ports:
- "2183:2181"
volumes:
- ./zookeeper3/conf:/conf
- ./zookeeper3/data:/data
- ./zookeeper3/datalog:/datalog
environment:
ZOO_MY_ID: 3
ZOO_SERVERS: server.1=zookeeper1:2888:3888;2181 server.2=zookeeper2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181
networks:
net_zoo_kafka:
ipv4_address: 172.23.0.13
kafka1:
image: docker.io/wurstmeister/kafka
hostname: kafka1
container_name: kafka1
privileged: true
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.94.133:9092
KAFKA_ADVERTISED_HOST_NAME: 192.168.94.133
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181/kafka,zookeeper2:2182/kafka,zookeeper3:2183/kafka
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./kafka1:/kafka
links:
- zookeeper1
- zookeeper2
- zookeeper3
networks:
net_zoo_kafka:
ipv4_address: 172.23.0.14
kafka2:
image: docker.io/wurstmeister/kafka
hostname: kafka2
container_name: kafka2
privileged: true
ports:
- "9093:9092"
environment:
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.94.133:9093
KAFKA_ADVERTISED_HOST_NAME: 192.168.94.133
KAFKA_ADVERTISED_PORT: 9093
KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181/kafka,zookeeper2:2182/kafka,zookeeper3:2183/kafka
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./kafka2:/kafka
links:
- zookeeper1
- zookeeper2
- zookeeper3
networks:
net_zoo_kafka:
ipv4_address: 172.23.0.15
kafka3:
image: docker.io/wurstmeister/kafka
hostname: kafka3
container_name: kafka3
privileged: true
ports:
- "9094:9092"
environment:
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.94.133:9094
KAFKA_ADVERTISED_HOST_NAME: 192.168.94.133
KAFKA_ADVERTISED_PORT: 9094
KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181/kafka,zookeeper2:2182/kafka,zookeeper3:2183/kafka
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./kafka3:/kafka
links:
- zookeeper1
- zookeeper2
- zookeeper3
networks:
net_zoo_kafka:
ipv4_address: 172.23.0.16
kafka-manager:
image: docker.io/sheepkiller/kafka-manager
container_name: kafka-manager
hostname: kafka-manager
privileged: true
ports:
- "9000:9000"
links: # 连接本compose文件创建的container
- zookeeper1
- zookeeper2
- zookeeper3
- kafka1
- kafka2
- kafka3
# external_links: # 连接compose以外的container
environment:
ZK_HOSTS: zookeeper1:2181/kafka,zookeeper2:2182/kafka,zookeeper3:2183/kafka
KAFKA_BROKERS: kafka1:9092,kafka2:9093,kafka3:9094
APPLICATION_SECRET: letmein
KM_ARGS: -Djava.net.preferIPv4Stack=true
networks:
net_zoo_kafka:
ipv4_address: 172.23.0.17
networks:
net_zoo_kafka:
driver: bridge
ipam:
config:
- subnet: 172.23.0.0/25
gateway: 172.23.0.1
详解
- version 选择 2 版本,因为3版本不支持gateway, 我这里为了方便,将网络的创建放在了compose文件中,你也可以在外部用
docker network create --driver bridge --subnet 172.23.0.0/25 --gateway 172.23.0.1 zookeeper_network
的方式创建一个bridge的网桥,compose文件中用external
指定外部网络 - 所有容器都在创建的网桥的子网内,避免访问不到的问题
- zookeeper集群只需要暴露每个容器的2181端口出去,而2888/3888端口,因为容器之间在一个子网,可以互通,所以不需要暴露
kafka 集群的 KAFKA_ADVERTISED_LISTENERS / KAFKA_ADVERTISED_HOST_NAME 配置成你宿主机的IP ! 记得修改
验证zookeeper集群
启动后用 docker ps
命令查看容器启动没有问题,应该成功启动了3个zookeeper容器、3个kafka容器和1个kafka-manager容器
进入zookeeper集群中任意一个容器
输入 zkServer.sh status
命令查看zookeeper集群状态,此时你看到该节点 Node: follower
或者 Node: leader
,表示zookeeper集群启动成功
验证kafka集群
Ctrl + p + q 退出容器
进入任意一台kafka容器 docker exec -it kafka1 /bin/bash
,进入 /opt/kafka/bin
目录
键入命令创建一个topic ./kafka-topics.sh --zookeeper zookeeper1:2181/kafka,zookeeper2:2182/kafka,zookeeper3:2183/kafka --create --replication-factor 2 --partitions 1 --topic topic1
键入命令查看所有topic ./kafka-topics.sh --zookeeper zookeeper1:2181/kafka,zookeeper2:2182/kafka,zookeeper3:2183/kafka --list
键入命令删除topic ./kafka-topics.sh --zookeeper zookeeper1:2181/kafka,zookeeper2:2182/kafka,zookeeper3:2183/kafka --delete --topic topic1
注:kafka的topic信息都是保存在zookeeper中的,所以要连接zookeeper集群,并且在上面的compose文件中我们明确的指定了kafka在zookeeper中的namespace, 即
/kafka
, 所以在操作topic时,务必指定namespace, 否则,会报错提示broker 为0
验证使用 Kafka-manager
再次确认宿主机 9000/tcp 端口是否开放,或者防火墙是否关闭
浏览器访问 宿主机IP:9000
添加一个Cluster
标红的地方注意一下,其他的地方根据你的需求修改;输入完成点击save
OK, 至此,整个zookeeper集群、kafka集群、kafka-manger都部署成功
结语
坑总是一个又一个,踩坑了,睡一觉,但别忘了爬出来。
如果大家有什么疑问,欢迎给我留言