Kafka选择分区发送数据
Java向Kafka发送数据的时候,Kafka Client会根据partitioner.class来实例化一个对象来选择分区,默认的实现是org.apache.kafka.clients.producer.internals.DefaultPartitioner,要指定自己的分区,只需要实现Partitioner接口并且配置partitioner.class就可以了
一、pom依赖
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.9.0.1</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.9.0.1</version>
</dependency>
二、实现Partitioner接口
import java.util.Map;
import org.apache.kafka.clients.producer.Partitioner;
import org.apache.kafka.common.Cluster;
public class MyPartitioner implements Partitioner {
@Override
public void configure(Map<String, ?> configs) {
}
@Override
public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
int paritionCount = cluster.partitionCountForTopic(topic);
//主题分区数量
return key.hashCode() % paritionCount;
//自定义发送的分区
}
@Override
public void close() {
}
}
三、Producer配置
Properties kafkaProps = new Properties();
kafkaProps.put("partitioner.class", "com.master.kafka.MyPartitioner");
KafkaProducer<String, String> producer = new KafkaProducer<String, String>(kafkaProps);