kafka分区，副本，倾斜及日常操作

最新推荐文章于 2022-08-27 13:23:11 发布

wangtingting_100

最新推荐文章于 2022-08-27 13:23:11 发布

阅读量4.2k

点赞数

分类专栏： kafka

本文链接：https://blog.csdn.net/wangtingting_100/article/details/89854867

版权

kafka 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

Topic Summary：

Replication     3
Number of Partitions    15
Sum of partition offsets    0
Total number of Brokers   13
Number of Brokers for Topic   10
Preferred Replicas %     100
Brokers Skewed %   30
Brokers Spread %   76
Under-replicated %   0

Partitions by Broker：

Broker   # of Partitions   Partitions   Skewed?
11   5   (4,5,6,13,14)   false
12   4   (5,6,7,14)   false
13   3   (6,7,8)   false
14   3   (7,8,9)   false
15   3   (0,8,9)   false
16   4   (0,1,9,10)   false
17   5   (0,1,2,10,11)   false
18   6   (1,2,3,10,11,12)   true
19   6   (2,3,4,11,12,13)   true
20   6   (3,4,5,12,13,14)   true

上表中，broker 18，19，20发生了倾斜。倾斜的计算方式参照 https://github.com/yahoo/kafka-manager/blob/5edd5e96ac4a8a3701b8e01922e256c052ce3f29/app/kafka/manager/model/ActorModel.scala#L416-L420：

    val brokersSkewPercentage : Int = {
      if(topicBrokers > 0)
        (100 * partitionsByBroker.count(_.isSkewed)) / topicBrokers
      else 0
    }

这里可能有人会困惑，为什么倾斜的是18，19，20三台？
Number of Partitions * Replication / Number of Brokers for Topic = 15 * 3 / 10 = 4.5
这意味着至少有一个broker上partitions个数为5，那么5不算倾斜，6的才算倾斜，所以18，19，20才发生了倾斜，3/10=30%，所倾斜比例是30%。

Partition Information：
Partition   Latest Offset   Leader   Replicas   In Sync Replicas   Preferred Leader?   Under Replicated?
0           1,881,213       15   (15,16,17)                       true               false
1           1,881,213       16   (16,17,18)                       true               false
2           1,881,213       17   (19,17,18)                       true               false
3           1,881,213       18   (20,19,18)                       true               false
4           1,881,213       11   (11,19,20)                       true               false
5           1,881,213       12   (11,12,20)                       true               false
6           1,881,213       13   (11,12,13)                       true               false
7           1,881,213       14   (12,13,14)                       true               false
8           1,881,213       x   (13,14,15) true false
9           1,881,213       16   (16,15,14)                       true               false
10           1,881,213       16   (16,17,18)                       true               false
11           1,881,213       19   (17,18,19)                       true               false
12           1,881,213       20   (18,19,20)                       true               false
13           1,881,213       20   (11,19,20)                       true               false
14           1,881,213       20   (11,12,20)                       true               false

#查看topics
$ sudo /opt/kafka/bin/kafka-topics.sh --list --zookeeper zk-daily.s.xxx.com:PORT/kafka|head -10
xxx_xxx
bigdata.xxx.xxx.xxx
bigdata.xxx.xxx.
log.xxx.xxx

#创建topic test1，该topic分区为1，副本为2（需注意的是，副本数不能多于Broker数）
$ sudo /opt/kafka/bin/kafka-topics.sh --create --topic test1 --zookeeper zk-daily.s.xxx.com:PORT/kafka --partitions 1 --replication-factor 2
Created topic "test1".

#生产消息
$ ./kafka-console-producer.sh --broker-list kafka-daily.s.xxx.com:PORT --topic test1
[2019-05-14 14:27:43,428] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
1
2
3

#查看 topic test1 消息分布
$ ./kafka-topics.sh --describe --zookeeper zk-daily.s.xxx.com:PORT/kafka --topic test1 Topic:test1 PartitionCount:3 ReplicationFactor:1 Configs:
Topic:test1 PartitionCount:1 ReplicationFactor:2 Configs:
Topic: test1 Partition: 0 Leader: 1 Replicas: 1,0 Isr: 1,0

#消费消息，从头开始消费
$ ./kafka-console-consumer.sh --zookeeper zk-daily.s.xxx.com:PORT/kafka --topic test1 --from-beginning

hi
la
hello world
hi
hello
how are you
1
2
3
^CConsumed 11 messages