kafka是如何决定把某条信息发送到哪个partition的呢?
源码kafka的版本:
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>2.0.1</version>
</dependency>
分三种情形:
- key 为null
- key 不为null
- 指定了partitionid
看下源码:
先看这段注释:
/**
* The default partitioning strategy:
* <ul>
* <li>If a partition is specified in the record, use it
* <li>If no partition is specified but a key is present choose a partition based on a hash of the key
* <li>If no partition or key is present choose a partition in a round-robin fashion
*/
说的很清楚:
- 如果指定了分区,那么就把record放入到此分区里,这个有对应的API可以用
- 如果没有指定分区,且key存在,那么就根据hash取模来决定放入哪个分区
- 如果没有指定分区或者key不存在,那么就采用round-robin策略
kafka里提供了一个默认分区器:DefaultPartitioner
我们看下里面的分区方法:
public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
List<PartitionInfo> partitions = cluster.partitionsForTopic(topic);
int numPartitions = partitions.size();
if (keyBytes == null) {
int nextValue = nextValue(topic);
List<PartitionInfo> availablePartitions = cluster.availablePartitionsForTopic(topic);
if (availablePartitions.size() > 0) {
int part = Utils.toPositive(nextValue) % availablePartitions.size();
return availablePartitions.get(part).partition();
} else {
// no partitions are available, give a non-available partition
return Utils.toPositive(nextValue) % numPartitions;
}
} else {
// hash the keyBytes to choose a partition
return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
}
}
- 首先判断看看key是否为null ,如果是null并且有可用的partition那么就取模来决定partition id,这里不是对key做hash取模
- 如果没有可用的分区,则放入到不可用的分区里
- 如果说key存在那就是取hash然后取模