Kafka 集群扩容

#目录
1、 前言
2、集群扩容简介
3、Zookeeper 集群扩容
4、Kafka 集群扩容

#详细过程

###1、前言
本文档是在前一篇《kafka环境搭建说明》的姊妹篇,专门将kafka的集群扩容
###2、集群扩容简介
前一篇中kafka的搭建使用的是kafka自带的zookeeper,所以本次Kafka的扩容包括两部分内容:zookeeper集群扩容和kafka集群扩容

Kafka集群扩容准备工作:

* 现有的3太机器,IP地址为:10.25.74.234、10.27.22.6、10.27.22.178
* 准备2台机器,IP地址分别为:10.28.9.32、10.28.8.123;

部署环境参照前一篇文章。

###3、Zookeeper 集群扩容
#####3.1、修改配置文件

(1)配置 /kafka_2.11-0.9.0.1/config/zookeeper.properties,配置内容如下所示:

tickTime=2000
dataDir=/srv/zookeeper/
clientPort=2181
initLimit=10
syncLimit=5
    server.1=10.25.74.234:2888:3888
    server.2=10.27.22.6:2888:3888
    server.3=10.27.22.178:2888:3888
    server.4=10.28.8.123:2888:3888      ##新增
server.5=10.28.9.32:2888:3888       ##新增

然后,使用命令创建 Zookeeper 主机标识:

echo “4” > /srv/zooieeper/myid         ##新增
echo  “5”> /srv/zooieeper/myid           ##新增

(2)启动新的4和5服务器的Zookeeper:
注意一定要用srvadmin用户

/srv/kafka_2.11-0.10.0.1/bin/zookeeper-server-start.sh /srv/kafka_2.11-0.10.0.1/config/zookeeper.properties  &

#####3.2、查看集群情况

在4和5服务器上分别查看Zookeeper运行情况,如果出现follower即可:

echo stat | nc  10.28.8.123 2181

命令返回:

Zookeeper version: 3.4.6-1569965, built on 02/20/2014 09:09 GMT
Clients:
/192.168.1.184:35620[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/4/117
Received: 37394
Sent: 37399
Connections: 1
Outstanding: 0
Zxid: 0x10000e33f
Mode: follower
Node count: 30
echo stat | nc 10.28.9.32 2181

命令返回:

Zookeeper version: 3.4.6-1569965, built on 02/20/2014 09:09 GMT
Clients:
 /192.168.1.184:60265[0](queued=0,recved=1,sent=0)
 /192.168.1.184:54585[1](queued=0,recved=32586,sent=32587)
 /192.168.1.203:1842[1](queued=0,recved=32563,sent=32565)
Latency min/avg/max: 0/1/297
Received: 95473
Sent: 95474
Connections: 3
Outstanding: 0
Zxid: 0x10000e4bf
Mode: follower
Node count: 30

如无返回说明该Zookeeper主机存在问题。如Zookeeper集群中的节点均无问题代表集群搭建成功。

参考文档:

http://blog.csdn.net/hackerwin7/article/details/43559991

如果没有问题,再一次重启其它3台老的zookeeper服务,并检查zookeeper运行情况,必须有一台角色是leader,如下:

Zookeeper version: 3.4.6-1569965, built on 02/20/2014 09:09 GMT
Clients:
/10.28.9.32:33035[0](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/0/3
Received: 1485
Sent: 1484
Connections: 1
Outstanding: 0
Zxid: 0x400000351
Mode: leader         (必须有一个是leader)
Node count: 381

###4、Kafka 集群扩容
#####4.1、修改配置文件

(1)第4台服务器的配置 /srv/kafka/config/server.properties,配置内容如下所示:

broker.id=204  (与老的3太服务器区别开)
port=9092
listeners=PLAINTEXT://IP:9092
advertised.host.name=IP
log.dirs=/data/log/kafka-logs
zookeeper.contact=10.25.74.234:2181,10.27.22.6:2181,10.27.22.178:2181,10.28.8.123:2181,10.28.9.32:2181

(2)第5台服务器的配置 /srv/kafka/config/server.properties,配置内容如下所示:

broker.id=205  (与老的4太服务器区别开)
port=9092
listeners=PLAINTEXT://IP:9092
advertised.host.name=IP
log.dirs=/data/log/kafka-logs
zookeeper.contact=10.25.74.234:2181,10.27.22.6:2181,10.27.22.178:2181,10.28.8.123:2181,10.28.9.32:2181

(3)Kafka集群启动

首先启动新增的2台kafka,检查正常后再重启老的3台kafka
注意一定要用srvadmin用户

启动命令:

/srv/kafka_2.11-0.10.0.1/bin/kafka-server-start.sh /srv/kafka_2.11-0.10.0.1/config/server.properties &

使用命令查看kafka的信息:

srv/kafka_2.11-0.10.0.1/bin/kafka-topics.sh --describe --zookeeper 10.28.9.32:2181

结果如下:204和205的节点已经加入新的Topic


Topic:test-new-kafka-node-5	PartitionCount:20	ReplicationFactor:2	Configs:
	Topic: test-new-kafka-node-5	Partition: 0	Leader: 205	Replicas: 205,201	Isr: 205,201
	Topic: test-new-kafka-node-5	Partition: 1	Leader: 201	Replicas: 201,202	Isr: 201,202
	Topic: test-new-kafka-node-5	Partition: 2	Leader: 202	Replicas: 202,203	Isr: 202,203
	Topic: test-new-kafka-node-5	Partition: 3	Leader: 203	Replicas: 203,204	Isr: 203,204
	Topic: test-new-kafka-node-5	Partition: 4	Leader: 204	Replicas: 204,205	Isr: 204,205
	Topic: test-new-kafka-node-5	Partition: 5	Leader: 205	Replicas: 205,202	Isr: 205,202
	Topic: test-new-kafka-node-5	Partition: 6	Leader: 201	Replicas: 201,203	Isr: 201,203
	Topic: test-new-kafka-node-5	Partition: 7	Leader: 202	Replicas: 202,204	Isr: 202,204
	Topic: test-new-kafka-node-5	Partition: 8	Leader: 203	Replicas: 203,205	Isr: 203,205
	Topic: test-new-kafka-node-5	Partition: 9	Leader: 204	Replicas: 204,201	Isr: 204,201
	Topic: test-new-kafka-node-5	Partition: 10	Leader: 205	Replicas: 205,203	Isr: 205,203
	Topic: test-new-kafka-node-5	Partition: 11	Leader: 201	Replicas: 201,204	Isr: 201,204
	Topic: test-new-kafka-node-5	Partition: 12	Leader: 202	Replicas: 202,205	Isr: 202,205
	Topic: test-new-kafka-node-5	Partition: 13	Leader: 203	Replicas: 203,201	Isr: 203,201
	Topic: test-new-kafka-node-5	Partition: 14	Leader: 204	Replicas: 204,202	Isr: 204,202
	Topic: test-new-kafka-node-5	Partition: 15	Leader: 205	Replicas: 205,204	Isr: 205,204
	Topic: test-new-kafka-node-5	Partition: 16	Leader: 201	Replicas: 201,205	Isr: 201,205
	Topic: test-new-kafka-node-5	Partition: 17	Leader: 202	Replicas: 202,201	Isr: 202,201
	Topic: test-new-kafka-node-5	Partition: 18	Leader: 203	Replicas: 203,202	Isr: 203,202
	Topic: test-new-kafka-node-5	Partition: 19	Leader: 204	Replicas: 204,203	Isr: 204,203
发布了18 篇原创文章 · 获赞 17 · 访问量 6万+
展开阅读全文

没有更多推荐了,返回首页

©️2019 CSDN 皮肤主题: 大白 设计师: CSDN官方博客

分享到微信朋友圈

×

扫一扫,手机浏览