Kafka入门（二）：集群搭建

最新推荐文章于 2024-11-03 12:44:11 发布

君子一笑

最新推荐文章于 2024-11-03 12:44:11 发布

阅读量100

点赞数

分类专栏： MQ消息中间件文章标签： kafka

本文链接：https://blog.csdn.net/linux_one_piece/article/details/99625936

版权

MQ消息中间件专栏收录该内容

8 篇文章 1 订阅

订阅专栏

1. Kafka集群安装

官网下载Kafka或者点击这里下载，并解压到/usr/local/kafka/

tar -xzf kafka_2.12-2.3.0.tgz

Kafka使用ZooKeeper，可以使用Kafka下载包中的脚本启动一个单节点zookeeper。

bin/zookeeper-server-start.sh config/zookeeper.properties

搭建本地集群，共三个节点：节点1: 192.168.0.214:9092、节点2: 192.168.0.214:9093、节点3: 192.168.0.214:9094，配置节点1并启动如下：
拷贝Kafka配置文件：

cp config/server.properties config/server-9092.properties

修改以下参数：

broker.id=server-9092 
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs-9092

启动节点1：

bin/kafka-server-start.sh config/server-9092.properties

同理配置节点2、节点3并启动：

bin/kafka-server-start.sh config/server-9093.properties
bin/kafka-server-start.sh config/server-9094.properties

扩容和重新分区

假设现有3个集群集群：192.168.0.214:9092、192.168.0.214:9093、192.168.0.214:9094，新增一个节点192.168.0.214:9095，分配一个唯一的代理ID，启动服务：

bin/kafka-server-start.sh config/server-9095.properties

原有的topic并不会在新增的服务器分配任何数据，因此需要对topic重新分配分区，官网提供了分区重新分配工具用于手动重新分区：
查看需要重新分区的topic分区情况：

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic topic-default

Topic:topic-default	PartitionCount:6	ReplicationFactor:1	Configs:
	Topic: topic-default	Partition: 0	Leader: 2	Replicas: 2	Isr: 2
	Topic: topic-default	Partition: 1	Leader: 0	Replicas: 0	Isr: 0
	Topic: topic-default	Partition: 2	Leader: 1	Replicas: 1	Isr: 1
	Topic: topic-default	Partition: 3	Leader: 2	Replicas: 2	Isr: 2
	Topic: topic-default	Partition: 4	Leader: 0	Replicas: 0	Isr: 0
	Topic: topic-default	Partition: 5	Leader: 1	Replicas: 1	Isr: 1

当前topic-default的topic有6个分区，但是只分配了3个分区[0,1,2]，因此需要重新分区
1.确定要重新分区的topic：

> cat topics-to-move.json
{
  "topics": [{"topic": "topic-default"}],
  "version":1
}

2.生成候选分配

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2,3" --generate

Current partition replica assignment
{"version":1,
"partitions":[{"topic":"topic-default","partition":0,"replicas":[2],"log_dirs":["any"]},
				   {"topic":"topic-default","partition":2,"replicas":[1],"log_dirs":["any"]},
				   {"topic":"topic-default","partition":5,"replicas":[1],"log_dirs":["any"]},
				   {"topic":"topic-default","partition":1,"replicas":[0],"log_dirs":["any"]},
				   {"topic":"topic-default","partition":4,"replicas":[0],"log_dirs":["any"]},
				   {"topic":"topic-default","partition":3,"replicas":[2],"log_dirs":["any"]}]
}
Proposed partition reassignment configuration
{"version":1,
"partitions":[{"topic":"topic-default","partition":5,"replicas":[0],"log_dirs":["any"]},
				   {"topic":"topic-default","partition":2,"replicas":[1],"log_dirs":["any"]},
				   {"topic":"topic-default","partition":4,"replicas":[3],"log_dirs":["any"]},
				   {"topic":"topic-default","partition":1,"replicas":[0],"log_dirs":["any"]},
				   {"topic":"topic-default","partition":0,"replicas":[3],"log_dirs":["any"]},
				   {"topic":"topic-default","partition":3,"replicas":[2],"log_dirs":["any"]}]
}

将Proposed partition reassignment configuration以下的json数据保存在json文件中（例如expand-cluster-reassignment.json）
3.重新分区：

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --execute

4.验证结果

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --verify

Status of partition reassignment: 
Reassignment of partition topic-default-0 completed successfully
Reassignment of partition topic-default-2 completed successfully
Reassignment of partition topic-default-5 completed successfully
Reassignment of partition topic-default-1 completed successfully
Reassignment of partition topic-default-4 completed successfully
Reassignment of partition topic-default-3 completed successfully

修改topic分区数

topic-default现有分区数

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic topic-default

Topic:topic-default	PartitionCount:6	ReplicationFactor:1	Configs:
	Topic: topic-default	Partition: 0	Leader: 3	Replicas: 3	Isr: 3
	Topic: topic-default	Partition: 1	Leader: 0	Replicas: 0	Isr: 0
	Topic: topic-default	Partition: 2	Leader: 1	Replicas: 1	Isr: 1
	Topic: topic-default	Partition: 3	Leader: 2	Replicas: 2	Isr: 2
	Topic: topic-default	Partition: 4	Leader: 3	Replicas: 3	Isr: 3
	Topic: topic-default	Partition: 5	Leader: 0	Replicas: 0	Isr: 0

修改topic分片数，注意：只可以增加分区数，不可以减少分区数

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic topic-default --partitions 7

Topic:topic-default	PartitionCount:7	ReplicationFactor:1	Configs:
	Topic: topic-default	Partition: 0	Leader: 3	Replicas: 3	Isr: 3
	Topic: topic-default	Partition: 1	Leader: 0	Replicas: 0	Isr: 0
	Topic: topic-default	Partition: 2	Leader: 1	Replicas: 1	Isr: 1
	Topic: topic-default	Partition: 3	Leader: 2	Replicas: 2	Isr: 2
	Topic: topic-default	Partition: 4	Leader: 3	Replicas: 3	Isr: 3
	Topic: topic-default	Partition: 5	Leader: 0	Replicas: 0	Isr: 0
	Topic: topic-default	Partition: 6	Leader: 1	Replicas: 1	Isr: 1