这篇文章分享如何在docker环境下搭建kafka集群。
bin/jdk:8u221镜像的构建,请参考 -- docker基础环境搭建
bin/zookeeper:3.5.7镜像的构建,请参考 -- docker环境,搭建zookeeper集群
kafka的集群信息由zookeeper管理,先创建kafka-net docker网络并启动zookeeper
sudo docker network create kafka-net
sudo docker run -d --name zk --network kafka-net --network-alias zk bin/zookeeper:3.5.7
可以启动zookeeper集群,这里为了简单,只使用了单机的zookeeper。
下载kafka_2.12-2.3.1.tgz,并构建bin/kafka:2.3.1,Dockerfile如下
FROM bin/jdk:8u221
WORKDIR /usr/lib
COPY kafka_2.12-2.3.1.tgz .
COPY docker-entrypoint.sh /usr/local/bin
RUN tar -xzf kafka_2.12-2.3.1.tgz && rm kafka_2.12-2.3.1.tgz
&& groupadd -r kafka && useradd -r -g kafka kafka
&& mkdir -p /usr/local/kafka/data/
&& chown -R kafka:kafka /usr/local/kafka/data/
&& chmod 777 /usr/local/bin/docker-entrypoint.sh
VOLUME /usr/local/kafka/data/
WORKDIR /usr/lib/kafka_2.12-2.3.1
ENTRYPOINT ["docker-entrypoint.sh"]
docker-entrypoint.sh负责修改配置文件,启动kafka进程
#!/bin/bash
sed -i "/zookeeper.connect=/c zookeeper.connect=${ZOOKEEPER_CONNECT}"
config/server.properties
sed -i "/log.dirs=/c log.dirs=/usr/local/kafka/data/"
config/server.properties
sed -i "/broker.id=/c broker.id=${BROKER_ID}"
config/server.properties
echo >> config/server.properties
if [ -n "$LISTENERS" ]; then
echo "listeners=${LISTENERS}" >>
config/server.properties
fi
if [ -n "$ADVERTISED_LISTENERS" ]; then
echo "advertised.listeners=${ADVERTISED_LISTENERS}" >>
config/server.properties
fi
exec gosu kafka bin/kafka-server-start.sh config/server.properties
使用sed的c命令对配置文件进行整行替换,共修改了zookeeper.connect/log.dirs/broker.id三个配置。
listeners:启动kafka服务监听的ip/hostname和端口,不配置则使用java.net.InetAddress.getCanonicalHostName()获取的值和9092端口。
advertised.listeners:生产者和消费者连接的地址,kafka会把该地址注册到zookeeper中,不配置则使用listeners配置。
启动kafka容器
for i in `seq 1 3`; do
sudo docker run -d --name kafka-$i
--network kafka-net --network-alias kafka-$i
-e KAFKA_HEAP_OPTS="-Xmx512M -Xms512M"
-e LISTENERS="PLAINTEXT://kafka-${i}:9092"
-e ZOOKEEPER_CONNECT=zk:2181
-e BROKER_ID=$i bin/kafka:2.3.1
done
由于宿主机内存较少,这里使用KAFKA_HEAP_OPTS配置将kafka进程使用内存修改为-Xmx512M -Xms512M
。
创建topic
sudo docker exec kafka-1 bin/kafka-topics.sh --create
--bootstrap-server kafka-1:9092,kafka-2:9092,kafka-3:9092
--replication-factor 1 --partitions 1 --topic helloKafka
发送消息
$ sudo docker exec -it kafka-1 bin/kafka-console-producer.sh
--broker-list kafka-1:9092,kafka-2:9092,kafka-3:9092 --topic helloKafka
>This is a message
>This is another message
消费消息
$ sudo docker exec -it kafka-1 bin/kafka-console-consumer.sh
--bootstrap-server kafka-1:9092,kafka-2:9092,kafka-3:9092
--topic helloKafka --from-beginning
This is a message
This is another message
通过zookeeper查看broker
$ sudo docker exec -it zk bin/zkCli.sh
[zk] get /brokers/ids/1
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://kafka-1:9092"],"jmx_port":-1,"host":"kafka-1","timestamp":"1583589384712","port":9092,"version":4}
如果您觉得本文不错,欢迎关注我的微信公众号,您的关注是我坚持的动力!