3.5 Kafka Producer API之自定义分区负载均衡

1.代码示例

public class PartitionSample implements Partitioner {
    @Override
    public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
        //key-1,key-2
        String keyStr = key + "";
        String keyInt = keyStr.substring(4);
        System.out.println("keyStr : " + keyStr + "keyInt : " + keyInt);
        int i = Integer.parseInt(keyInt);
        return i % 2;
    }

    @Override
    public void close() {

    }

    @Override
    public void configure(Map<String, ?> configs) {

    }
}

public class ProducerSample {
    private static final String topicName = "steven";

    /**
     * Producer异步发送示例
     */
    public static void producerSend() {
        //Producer的配置
        Properties properties = new Properties();
        properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
        properties.put(ProducerConfig.ACKS_CONFIG, "all");
        properties.put(ProducerConfig.RETRIES_CONFIG, "0");
        properties.put(ProducerConfig.BATCH_SIZE_CONFIG, "16384");
        properties.put(ProducerConfig.LINGER_MS_CONFIG, "1");
        properties.put(ProducerConfig.BUFFER_MEMORY_CONFIG, "33554432");
        //序列化
        properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
        properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
        //自定义partition，配置自定义partition的全路径名
        properties.put(ProducerConfig.PARTITIONER_CLASS_CONFIG,"com.example.study.kafka.PartitionSample");
        //构建Producer对象
        Producer<String, String> producer = new KafkaProducer<>(properties);

        //消息对象ProducerRecord
        for (int i = 0; i < 10; i++) {
            String key = "key-" + i;
            ProducerRecord<String, String> record = new ProducerRecord<>(topicName, "key-" + i, "value-" + i);

            producer.send(record, new Callback() {
                @Override
                public void onCompletion(RecordMetadata recordMetadata, Exception e) {
                    System.out.println(key + ",partition:" + recordMetadata.partition() + ",offset:" + recordMetadata.offset());
                }
            });
        }

        //所有的通道打开都需要关闭
        producer.close();
    }

    public static void main(String[] args) {
        producerSend();
    }
}

2.代码运行结果
(1).控制台输出
可以看到消息均衡地分布在partition0和partition1的分区上。

key-0,partition:0,offset:0
key-1,partition:1,offset:1
key-2,partition:0,offset:2
key-3,partition:1,offset:3
key-4,partition:0,offset:4
key-5,partition:1,offset:5
key-6,partition:0,offset:6
key-7,partition:1,offset:7
key-8,partition:0,offset:8
key-9,partition:1,offset:9

(2).终端输出
打开一个cmd终端，在E:\Kafka\kafka_2.12-1.1.0\bin\windows目录下执行kafka-console-consumer.bat --zookeeper localhost:2181 --topic steven，可以看到生产的消息。