kafka 如何确定分区数及不同分区测试效果

 

Partitions设计目的kafka是基于文件存储，每个partition在存储层面试append log文件，任何发布到此partition的消息都会被直接追加到log文件的尾部。
通过分区，可以将日志内容分散到多个server上，来避免文件尺寸达到单机磁盘的上限，每个partition都会被当前server（kafka实例）保存，可以将一个topic切分到任意多个partitions。越多的partitions意味着可以容纳更多的consumer，有效提升并发消费的能力。
那么如何确定分区的数量呢？
分区数 = Tt / max(Tp, Tc)
Tp表示producer的吞吐量。测试producer通常是很容易的，因为它的逻辑非常简单，就是直接发送消息到Kafka就好了。Tc表示consumer的吞吐量。测试Tc通常与应用的关系更大， 因为Tc的值取决于你拿到消息之后执行什么操作，因此Tc的测试通常也要麻烦一些。总的目标吞吐量是Tt

创建的分区数不同，测试效果不一样

默认0个分区   分区数只能增加 不能减少  

[root@apache01 kafka]# bin/kafka-topics.sh --zookeeper apache02:2181,apache01:2181,apache03:2181 --describe --topic test
Topic:test    PartitionCount:1    ReplicationFactor:1    Configs:
    Topic: test    Partition: 0    Leader: 2    Replicas: 2    Isr: 2

[root@apache01 kafka]# bin/kafka-producer-perf-test.sh  --topic test --record-size 100 --num-records 100000 --throughput -1 --producer-props bootstrap.servers=apache01:9092,apache02:9092,apache03:9092
100000 records sent, 23764.258555 records/sec (2.27 MB/sec), 1647.63 ms avg latency, 2280.00 ms max latency, 1606 ms 50th, 2248 ms 95th, 2262 ms 99th, 2280 ms 99.9th.
 

修改分区数量为3
[root@apache01 kafka]#  bin/kafka-topics.sh --zookeeper apache02:2181  --alter --topic test --partitions 3
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
[root@apache01 kafka]# bin/kafka-topics.sh --zookeeper apache02:2181,apache01:2181,apache03:2181 --describe --topic test
Topic:test    PartitionCount:3    ReplicationFactor:1    Configs:
    Topic: test    Partition: 0    Leader: 2    Replicas: 2    Isr: 2
    Topic: test    Partition: 1    Leader: 0    Replicas: 0    Isr: 0
    Topic: test    Partition: 2    Leader: 1    Replicas: 1    Isr: 1
[root@apache01 kafka]# bin/kafka-producer-perf-test.sh  --topic test --record-size 100 --num-records 100000 --throughput -1 --producer-props bootstrap.servers=apache01:9092,apache02:9092,apache03:9092
100000 records sent, 109289.617486 records/sec (10.42 MB/sec), 201.79 ms avg latency, 313.00 ms max latency, 237 ms 50th, 293 ms 95th, 306 ms 99th, 312 ms 99.9th.

修改分区数量为6
[root@apache01 kafka]#  bin/kafka-topics.sh --zookeeper apache02:2181  --alter --topic test --partitions 6
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
[root@apache01 kafka]# bin/kafka-topics.sh --zookeeper apache02:2181,apache01:2181,apache03:2181 --describe --topic test
Topic:test    PartitionCount:6    ReplicationFactor:1    Configs:
    Topic: test    Partition: 0    Leader: 2    Replicas: 2    Isr: 2
    Topic: test    Partition: 1    Leader: 0    Replicas: 0    Isr: 0
    Topic: test    Partition: 2    Leader: 1    Replicas: 1    Isr: 1
    Topic: test    Partition: 3    Leader: 2    Replicas: 2    Isr: 2
    Topic: test    Partition: 4    Leader: 0    Replicas: 0    Isr: 0
    Topic: test    Partition: 5    Leader: 1    Replicas: 1    Isr: 1
[root@apache01 kafka]# 
[root@apache01 kafka]# 
[root@apache01 kafka]# bin/kafka-producer-perf-test.sh  --topic test --record-size 100 --num-records 100000 --throughput -1 --producer-props bootstrap.servers=apache01:9092,apache02:9092,apache03:9092
100000 records sent, 158982.511924 records/sec (15.16 MB/sec), 50.68 ms avg latency, 160.00 ms max latency, 50 ms 50th, 90 ms 95th, 101 ms 99th, 121 ms 99.9th.

修改分区数量为9
[root@apache01 kafka]#  bin/kafka-topics.sh --zookeeper apache02:2181  --alter --topic test --partitions 9
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
[root@apache01 kafka]# bin/kafka-producer-perf-test.sh  --topic test --record-size 100 --num-records 100000 --throughput -1 --producer-props bootstrap.servers=apache01:9092,apache02:9092,apache03:9092
100000 records sent, 167785.234899 records/sec (16.00 MB/sec), 47.10 ms avg latency, 141.00 ms max latency, 42 ms 50th, 98 ms 95th, 110 ms 99th, 129 ms 99.9th.

修改分区数量为15
[root@apache01 kafka]#  bin/kafka-topics.sh --zookeeper apache02:2181  --alter --topic test --partitions 15
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
[root@apache01 kafka]# bin/kafka-producer-perf-test.sh  --topic test --record-size 100 --num-records 100000 --throughput -1 --producer-props bootstrap.servers=apache01:9092,apache02:9092,apache03:9092
100000 records sent, 159489.633174 records/sec (15.21 MB/sec), 74.48 ms avg latency, 240.00 ms max latency, 42 ms 50th, 207 ms 95th, 222 ms 99th, 237 ms 99.9th.