centos7 安装zookeeper集群和kafka集群

centos7 安装zookeeper集群和kafka集群

 

zookeeper集群和kafka集群由下面3台服务器完成

172.20.19.49 zookeeper-1

172.20.19.98 zookeeper-2

172.20.19.32 zookeeper-3

 

安装zookeeper集群

zookeeper集群至少需要3台服务器.否则会报如下错误

​

 

下载zookeeper-3.4.14 并解压

tar zxvf zookeeper-3.4.14.tar.gz

rm -rf  zookeeper-3.4.14.tar.gz

mv zookeeper-3.4.14  /usr/local/zookeeper

复制conf文件夹下zoo_sample.cfg 并重命名为zoo.cfg

cp /usr/local/zookeeper-3.4.14/conf/zoo_sample.cfg  /usr/local/zookeeper-3.4.14/conf/zoo.cfg

修改zoo.cfg的配置

cat > /usr/local/zookeeper-3.4.14/conf/zoo.cfg <<EOF

tickTime=2000

dataDir=/usr/local/zookeeper-3.4.14/data

dataLogDir=/usr/local/zookeeper-3.4.14/logs

clientPort=2181

initLimit=5

syncLimit=2

quorumListenOnAllIPs=true

server.1=172.20.19.49:2888:3888

server.2=172.20.19.98:2888:3888

server.3=172.20.19.32:2888:3888

EOF

 

 

防火墙放行2181和3888端口

firewall-cmd --zone=public --add-port=2181/tcp --permanent 

firewall-cmd --zone=public --add-port=3888/tcp --permanent 

firewall-cmd --zone=public --add-port=2888/tcp --permanent

firewall-cmd --reload

 

 

mkdir -p /usr/local/zookeeper-3.4.14/logs

mkdir -p /usr/local/zookeeper-3.4.14/data

 

修改各台机子的hostname  需要重启生效

172.20.19.49上执行

hostnamectl set-hostname zookeeper-1

cat > /usr/local/zookeeper-3.4.14/data/myid <<EOF

1

EOF

172.20.19.98上执行

hostnamectl set-hostname zookeeper-2

cat > /usr/local/zookeeper-3.4.14/data/myid <<EOF

2

EOF

172.20.19.32上执行

hostnamectl set-hostname zookeeper-3

cat > /usr/local/zookeeper-3.4.14/data/myid <<EOF

3

EOF

 

不修改hostname或者防火墙可能会报路由或者拒绝连接

 

 

 

 

 

启动zookeeper

cd /usr/local/zookeeper-3.4.14/data

/usr/local/zookeeper-3.4.14/bin/zkServer.sh  start

 

查看集群状态

/usr/local/zookeeper-3.4.14/bin/zkServer.sh  status

 

 

 

安装kafka集群

 

Centos7安装kafka

需要提前安装zookeeper

 

在每台机子上执行

cat >>/etc/hosts <<EOF

172.20.19.49 zookeeper-1

172.20.19.98 zookeeper-2

172.20.19.32 zookeeper-3

EOF

cat /etc/hosts

 

 

 

cd /tmp

wget http://archive.apache.org/dist/kafka/0.8.2.1/kafka_2.12-2.2.0.tgz

tar zxf  kafka_2.12-2.2.0.tgz -C /usr/local

 

 

172.20.19.49 zookeeper-1上执行

cat >/usr/local/kafka_2.12-2.2.0/config/server.properties <<EOF

log.dirs=/usr/local/kafka_2.12-2.2.0/log

broker.id=1

port=9092

host.name= zookeeper-1

zookeeper.connect=172.20.19.49:2181, 172.20.19.98:2181, 172.20.19.32:2181

EOF

 

172.20.19.98 zookeeper-2上执行

cat >/usr/local/kafka_2.12-2.2.0/config/server.properties <<EOF

log.dirs=/usr/local/kafka_2.12-2.2.0/log

broker.id=2

port=9092

host.name= zookeeper-2

zookeeper.connect=172.20.19.49:2181, 172.20.19.98:2181, 172.20.19.32:2181

EOF

 

172.20.19.32 zookeeper-3上执行

 

cat >/usr/local/kafka_2.12-2.2.0/config/server.properties <<EOF

log.dirs=/usr/local/kafka_2.12-2.2.0/log

broker.id=3

port=9092

host.name= zookeeper-3

zookeeper.connect=172.20.19.49:2181, 172.20.19.98:2181, 172.20.19.32:2181

EOF

 

 

 

 

 

 

 所有机子上执行

 

 

 

开启9092端口 firewall-cmd --zone=public --add-port=9092/tcp --permanent firewall-cmd --reload   cat >/usr/local/kafka_2.12-2.2.0/config/zookeeper.properties <<EOF clientPort=2181 maxClientCnxns=100 tickTime=2000 initLimit=10 syncLimit=5 dataDir=/usr/local/zookeeper-3.4.14/data dataLogDir=/usr/local/zookeeper-3.4.14/logs   EOF     chmod +x  /usr/local/kafka_2.12-2.2.0/bin/*.sh cd /usr/local/kafka_2.12-2.2.0/bin ./kafka-server-start.sh -daemon ../config/server.properties     echo "@reboot /usr/local/kafka_2.12-2.2.0/bin/kafka-server-start.sh -daemon ../config/server.properties" >> /var/spool/cron/root

 

检查kafka集群

依次执行

cd /usr/local/zookeeper-3.4.14/bin/

./zkCli.sh -server 172.20.19.49:2181

ls /

ls /brokers

ls /brokers/ids

 

 

 

　说明：zookeeper集群建好之后，通过“ls /”出来的只有zookeeper，连接kafka使用后，/ 下面多了不少东西，其中通过查看/brokers/ids可以发现已经检查到了已经安装的三台kafka的broker.id[1,2,3]。