title: Kafka集群部署
categories: Linux
tags:
- Kafka
- ZooKeeper
- Cluster
timezone: Asia/Shanghai
date: 2019-01-18
简介
Kafka强依赖ZK,如果想要使用Kafka,就必须安装ZK,Kafka中的消费偏置信息、kafka集群、topic信息会被存储在ZK中。有人可能会说我在使用Kafka的时候就没有安装ZK,那是因为Kafka内置了一个ZK,一般我们不使用它。
服务器规划:准备4台服务器,并将信息写入本地hosts
10.0.1.1 redhat610-ch-01
10.0.1.2 redhat610-ch-02
10.0.1.3 redhat610-ch-03
10.0.1.4 redhat610-ch-04
第一步:关闭默认防火墙
setenforce 0
sed -i -r "/^SELINUX=/c SELINUX=disabled" /etc/selinux/config
which systemctl && systemctl stop firewalld
which systemctl && systemctl disable firewalld
which systemctl && systemctl stop iptables || service iptables stop
which systemctl && systemctl disable iptables || chkconfig iptables off
第二步:部署kafka集群
0.上传并解压,保持4台服务器路径一致,方便以后管理维护
tar vxf kafka_2.11-0.10.0.0.tgz
tar vxf zookeeper-3.4.10.tar.gz
mv kafka_2.11-0.10.0.0 kafka
mv zookeeper-3.4.10 zookeeper
mv zookeeper kafka
2.zookeeper配置
1.在节点1上做以下设置并复制配置文件到节点2、3、4,不需要再修改东西
cp kafka/zookeeper/conf/zoo_sample.cfg kafka/zookeeper/conf/zoo.cfg
vim zoo.cfg
clientPort=20007
dataDir=/home/kafka/tmp
server.1=redhat610-ch-01:20008:20009
server.2=redhat610-ch-02:20008:20009
server.3=redhat610-ch-03:20008:20009
server.3=redhat610-ch-03:20008:20009
2.分别在4个节点上分别写入myid文件并验证
mkdir -p /home/monitor/bin/kafka/zookeeper/tmp
echo 1 > /home/monitor/bin/kafka/zookeeper/tmp/myid
echo 2 > /home/monitor/bin/kafka/zookeeper/tmp/myid
echo 3 > /home/monitor/bin/kafka/zookeeper/tmp/myid
echo 4 > /home/monitor/bin/kafka/zookeeper/tmp/myid
cat /home/monitor/bin/kafka/zookeeper/tmp/myid
3.分别在4个节点上建立启动脚本
vim /home/monitor/sh/1_zookeeper_start.sh
#!/bin/bash
/home/monitor/bin/kafka/zookeeper/bin/zkServer.sh start &
4.分别在4个节点上启动脚本并验证
chmod +x /home/monitor/sh/1_zookeeper_start.sh
/home/monitor/sh/1_zookeeper_start.sh
/home/monitor/bin/kafka/zookeeper/bin/zkServer.sh status
3.kafka部署
1.在节点1修改配置文件
cp /home/monitor/bin/kafka/config/server.properties /home/monitor/bin/kafka/config/server.properties.bak
vim /home/monitor/bin/kafka/config/server.properties
broker.id=0
log.dirs=/home/monitor/bin/kafka/logs
listeners = PLAINTEXT://gfzcptyy01:20010
zookeeper.connect=gfzcptyy01:20007,gfzcptyy02:20007,gfzcptwfw01:20007,gfzcptwfw02:20007
2.将配置文件复制到节点2、3、4
复制以后需要分别修改:broker.id=0、1、2、3
监听地址和端口
另外几台服务器的主机名加端口
vim /home/kafka/config/server.properties
3.分别在4个节点创建日志文件夹
mkdir -p /home/monitor/bin/kafka/logs
4.分别在4个节点建立启动脚本并验证
vim /home/monitor/sh/2_kafka_start.sh
#!/bin/bash
nohup /home/monitor/bin/kafka/bin/kafka-server-start.sh /home/monitor/bin/kafka/config/server.properties>/home/monitor/bin/kafka/logs/kafka.log &
chmod +x /home/monitor/sh/2_kafka_start.sh
/home/monitor/sh/2_kafka_start.sh
tail -f /home/monitor/bin/kafka/logs/kafka.log
netstat -antp
/home/monitor/bin/kafka/bin/kafka-server-stop.sh