基于docker搭建zookeeper集群、kafka集群---二（多台真机之间的集群）

最新推荐文章于 2022-11-03 17:58:29 发布

deye1993

最新推荐文章于 2022-11-03 17:58:29 发布

阅读量255

点赞数

文章标签：大数据运维 awk

原文链接：http://www.cnblogs.com/fanever/p/10979036.html

版权

三台真机，容器采用host

192.168.0.128 192.168.0.141 192.168.0.142

zookeeper集群搭建

在每台机上执行脚本，

#!/bin/bash
#Get zookeeper image
zkimage=`docker images | grep zookeeper | awk {'print $1'}`
if [ -n "$zkimage" ]
then
    echo 'The zookeeper image is already existed.'
else
    echo 'Pull the latest zookeeper image.'
    docker pull zookeeper
fi

#Create network for zookeeper containers
zknet=`docker network ls | grep yapi_net | awk {'print $2'}`
if [ -n "$zknet" ]
then
    echo 'The zknetwork is already existed.'
else
    echo 'Create zknetwork.'
    docker network create --subnet 172.30.0.0/16 yapi_net
fi

#Start zookeeper cluster
echo 'Start 3 zookeeper servers.'
rm -rf   /opt/zookeeper_1/data  /opt/zookeeper_1/datalog  /var/log/zookeeper_1/log
rm -rf   /opt/zookeeper_2/data  /opt/zookeeper_2/datalog  /var/log/zookeeper_2/log
rm -rf   /opt/zookeeper_3/data  /opt/zookeeper_3/datalog  /var/log/zookeeper_3/log

mkdir -p  /opt/zookeeper_1/data  /opt/zookeeper_1/datalog  /var/log/zookeeper_1/log
mkdir -p  /opt/zookeeper_2/data  /opt/zookeeper_2/datalog  /var/log/zookeeper_2/log
mkdir -p  /opt/zookeeper_3/data  /opt/zookeeper_3/datalog  /var/log/zookeeper_3/log


ZOO_SERVERS="server.1=192.168.0.128:2888:3888 server.2=192.168.0.142:2888:3888 server.3=192.168.0.141:2888:3888"
#新版 ZOO_SERVERS="clientPort=2181  server.1=192.168.0.146:2888:3888 server.2=192.168.0.145:2888:3888"

docker run --network host -d --restart always -v /opt/zookeeper_1/data:/data -v /opt/zookeeper_1/datalog:/datalog -v /var/log/zookeeper_1/log:/logs  -e ZOO_SERVERS="$ZOO_SERVERS" -e ZOO_MY_ID=1 --name zookeeper_1 -p 2182:2181 -p 2888:2888 -p 3888:3888 docker.io/zookeeper

注意每台机的id号不同

官方的镜像有可能配置的环境变量不同，需要修改脚本，参考Entrypoint文件，和配置文件，配置文件如下：

clientPort=2181
dataDir=/data
dataLogDir=/datalog
tickTime=2000
initLimit=5
syncLimit=2
autopurge.snapRetainCount=3
autopurge.purgeInterval=0
maxClientCnxns=60
server.1=192.168.0.128:2888:3888
server.2=192.168.0.142:2888:3888
server.3=192.168.0.141:2888:3888

单机（非集群）的配置文件，telnet ip 2181 = 》stat ==》Mode: standalone

telnet 新版的不能验证stat，访问http://192.168.0.145:8080/commands/stat，可验证，参考docker hub的zookeeper镜像

clientPort=2181
dataDir=/data
dataLogDir=/datalog
tickTime=2000
initLimit=5
syncLimit=2
autopurge.snapRetainCount=3
autopurge.purgeInterval=0
maxClientCnxns=60

kafka集群

在每台机上执行脚本，

#!/bin/bash
#Get zookeeper image
kfkimage=`docker images | grep 'docker.io/wurstmeister/kafka' | awk {'print $1'}`
if [ -n "$kfkimage" ]
then
    echo 'The docker.io/wurstmeister/kafka is already existed.'
else
    echo 'Pull the image.'
    docker pull docker.io/wurstmeister/kafka
fi

#Create network for zookeeper containers
kfknet=`docker network ls | grep yapi_net | awk {'print $2'}`
if [ -n "$kfknet" ]
then
    echo 'The kfknetwork is already existed.'
else
    echo 'Create kfknetwork.'
    docker network create --subnet 172.30.0.0/16 yapi_net
fi

#Start 3  zookeeper cluster
echo 'Start 3 kafka servers.'

rm -rf  /opt/kafka_1/logdata
rm -rf  /opt/kafka_2/logdata
rm -rf  /opt/kafka_3/logdata

mkdir -p  /opt/kafka_1/logdata
mkdir -p  /opt/kafka_2/logdata
mkdir -p  /opt/kafka_3/logdata

#kafka ip
kfk_1_ip='172.30.0.41'
kfk_2_ip='172.30.0.42'
kfk_3_ip='172.30.0.43'
#zk_jiqun_ip='172.30.0.31:2181'
zk_jiqun_ip='192.168.0.128:2181,192.168.0.142:2181,192.168.0.141:2181'
docker run --restart always -d --name kafka_1 --network host -e KAFKA_ZOOKEEPER_CONNECT=${zk_jiqun_ip} -e KAFKA_LISTENERS='PLAINTEXT://0.0.0.0:9092' -e KAFKA_BROKER_ID='141' -e KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://192.168.0.141:9092' -v /opt/kafka_1/logdata:/kafka -p 9092:9092  docker.io/wurstmeister/kafka

注意id号和PLAINTEXT://192.168.0.141:9092，（创建容器后，再进入容器修改，restart后会重置，所以必须在创建容器时，就要指定好环境变量。因为启动脚本会重新修改配置文件，所以要想修改环境变量，只需要在启动脚本里加一个即可）

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

补充：单机docker run -d --restart always -v /opt/zookeeper/data:/data -v /opt/zookeeper/datalog:/datalog -v /var/log/zookeeper/log:/logs --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 -p 8081:8080 docker.io/zookeeper

docker run --restart always -d --name kafka -e KAFKA_ZOOKEEPER_CONNECT=${zk_jiqun_ip} -e KAFKA_LISTENERS='PLAINTEXT://0.0.0.0:9092' -e KAFKA_BROKER_ID='163' -e KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://172.18.72.163:9092' -v /opt/kafka/logdata:/kafka -p 9092:9092 docker.io/wurstmeister/kafka

---------------------------------------------------------------------------------------------------------

补充2：云上部署，不同主机节点，zookeeper采用host网络：因为采用容器网络，zookeeper不同主机节点之间通信，找不到容器网络，

不同主机节点，kafka采用host网络，单参数KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://172.18.72.163:9092' ，发消息时，会通过配置文件PLAINTEXT://172.18.72.163:9092，找kafka集群的节点，线上配置私网即可，因为，服务之间可以通过私网访问，如果本地需要用来访问kafka时，就需要要配置公网ip：因为远程，如果是私网，就会连接超时，