Kafka集群部署

---

title: Kafka集群部署
categories: Linux
tags:
- Kafka
- ZooKeeper
- Cluster
timezone: Asia/Shanghai
date: 2019-01-18

简介

Kafka强依赖ZK，如果想要使用Kafka，就必须安装ZK，Kafka中的消费偏置信息、kafka集群、topic信息会被存储在ZK中。有人可能会说我在使用Kafka的时候就没有安装ZK，那是因为Kafka内置了一个ZK，一般我们不使用它。

服务器规划：准备4台服务器，并将信息写入本地hosts

10.0.1.1    redhat610-ch-01
10.0.1.2    redhat610-ch-02
10.0.1.3    redhat610-ch-03
10.0.1.4    redhat610-ch-04

第一步：关闭默认防火墙

setenforce 0
sed -i -r "/^SELINUX=/c SELINUX=disabled" /etc/selinux/config
which systemctl && systemctl stop firewalld
which systemctl && systemctl disable firewalld
which systemctl && systemctl stop iptables || service iptables stop
which systemctl && systemctl disable iptables || chkconfig iptables off

第二步：部署kafka集群

0.上传并解压，保持4台服务器路径一致，方便以后管理维护

tar vxf kafka_2.11-0.10.0.0.tgz
tar vxf zookeeper-3.4.10.tar.gz
mv kafka_2.11-0.10.0.0 kafka
mv zookeeper-3.4.10 zookeeper
mv zookeeper kafka

2.zookeeper配置

1.在节点1上做以下设置并复制配置文件到节点2、3、4，不需要再修改东西

cp kafka/zookeeper/conf/zoo_sample.cfg kafka/zookeeper/conf/zoo.cfg

vim zoo.cfg

clientPort=20007
dataDir=/home/kafka/tmp
server.1=redhat610-ch-01:20008:20009
server.2=redhat610-ch-02:20008:20009
server.3=redhat610-ch-03:20008:20009
server.3=redhat610-ch-03:20008:20009

2.分别在4个节点上分别写入myid文件并验证

mkdir -p /home/monitor/bin/kafka/zookeeper/tmp

echo 1 > /home/monitor/bin/kafka/zookeeper/tmp/myid
echo 2 > /home/monitor/bin/kafka/zookeeper/tmp/myid
echo 3 > /home/monitor/bin/kafka/zookeeper/tmp/myid
echo 4 > /home/monitor/bin/kafka/zookeeper/tmp/myid

cat /home/monitor/bin/kafka/zookeeper/tmp/myid

3.分别在4个节点上建立启动脚本

vim /home/monitor/sh/1_zookeeper_start.sh

#!/bin/bash
/home/monitor/bin/kafka/zookeeper/bin/zkServer.sh start &

4.分别在4个节点上启动脚本并验证

chmod +x /home/monitor/sh/1_zookeeper_start.sh

/home/monitor/sh/1_zookeeper_start.sh
/home/monitor/bin/kafka/zookeeper/bin/zkServer.sh status

3.kafka部署

1.在节点1修改配置文件

cp /home/monitor/bin/kafka/config/server.properties /home/monitor/bin/kafka/config/server.properties.bak
vim /home/monitor/bin/kafka/config/server.properties

broker.id=0
log.dirs=/home/monitor/bin/kafka/logs
listeners = PLAINTEXT://gfzcptyy01:20010
zookeeper.connect=gfzcptyy01:20007,gfzcptyy02:20007,gfzcptwfw01:20007,gfzcptwfw02:20007

2.将配置文件复制到节点2、3、4

​ 复制以后需要分别修改：broker.id=0、1、2、3

​ 监听地址和端口

​ 另外几台服务器的主机名加端口

vim /home/kafka/config/server.properties

3.分别在4个节点创建日志文件夹

mkdir -p /home/monitor/bin/kafka/logs

4.分别在4个节点建立启动脚本并验证

vim /home/monitor/sh/2_kafka_start.sh
#!/bin/bash
nohup /home/monitor/bin/kafka/bin/kafka-server-start.sh /home/monitor/bin/kafka/config/server.properties>/home/monitor/bin/kafka/logs/kafka.log &

chmod +x /home/monitor/sh/2_kafka_start.sh

/home/monitor/sh/2_kafka_start.sh
tail -f /home/monitor/bin/kafka/logs/kafka.log
netstat -antp

/home/monitor/bin/kafka/bin/kafka-server-stop.sh

附录：

Kafka官网：http://kafka.apache.org/downloads

本文用到离线包下载：https://pan.baidu.com/s/1zGyAdp979xI0dFDrhLoknw