Kafka生产者发送消息的方式解析

kafka生产者发送消息的方式可以分为：发送并忘记、同步发送、异步发送。

1、发送并忘记

生产者只要调用send方法就可以了，不需要处理send的返回值，大多数情况下消息都可以正常发送出去，但可能会丢失消息。

public class KafkaProducer {

    public static void main(String[] args) {
        KafkaProducer<String, String> producer = new KafkaProducer(producerConfigs());
        ProducerRecord<String, String> record = new ProducerRecord("test_topic", "hello kafka");
        //调用send方法即完成发送
        producer.send(record);
    }

    public static Map<String, Object> producerConfigs() {
        Map<String, Object> props = new HashMap<>();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.93.132:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        return props;
    }
}

但是当我们获取元数据时出现问题，send的方法依旧会被阻塞。

try {
	//这边会阻塞
    clusterAndWaitTime = waitOnMetadata(record.topic(), record.partition(), maxBlockTimeMs);
} catch (KafkaException e) {
    if (metadata.isClosed())
        throw new KafkaException("Producer closed while send in progress", e);
    throw e;
}

比如指定一个不存在的分区发送消息，就会因为元数据获取问题，导致阻塞，默认是等待60s。

Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Topic test_topic not present in metadata after 1000 ms.
	at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1269)
	at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:933)
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:856)
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:743)
	at cn.enjoyedu.sendtype.KafkaFutureProducer.main(KafkaFutureProducer.java:26)
Caused by: org.apache.kafka.common.errors.TimeoutException: Topic test_topic not present in metadata after 1000 ms.

2、同步发送

接收send方法的返回值Future对象，并调用get方法。

public static void main(String[] args) throws ExecutionException, InterruptedException {
        KafkaProducer<String, String> producer = new KafkaProducer(producerConfigs());
        ProducerRecord<String, String> record = new ProducerRecord("test_topic", "hello kafka");
        Future<RecordMetadata> send = producer.send(record);
        //get方法会阻塞
        RecordMetadata recordMetadata = send.get();
        System.out.println("topic: " + recordMetadata.topic() + ", offset: " + recordMetadata.offset() + ", partition: " + recordMetadata.partition());
    }

可以获取发送的一些信息。

topic: test_topic, offset: 232, partition: 3

3、异步发送

使用带Callback的send方法，提供请求完成的异步处理，当发送到服务器的记录已被确认时，将调用此方法。

public Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback) {
    // intercept the record, which can be potentially modified; this method does not throw exceptions
    ProducerRecord<K, V> interceptedRecord = this.interceptors.onSend(record);
    return doSend(interceptedRecord, callback);
}

public static void main(String[] args) {
        KafkaProducer<String, String> producer = new KafkaProducer(producerConfigs());
        ProducerRecord<String, String> record = new ProducerRecord("test_topic", "hello_key", "hello kafka");
        //callback1
        producer.send(record,
                (metadata, e) -> {
                    if (e != null) {
                        e.printStackTrace();
                    } else {
                        System.out.println("The offset of the record we just sent is: " + metadata.offset());
                    }
                });
        //callback2
        producer.send(record,
                (metadata, e) -> {
                    if (e != null) {
                        e.printStackTrace();
                    } else {
                        System.out.println("The offset of the record we just sent is: " + metadata.offset());
                    }
                });
        producer.close();
    }

The offset of the record we just sent is: 98
The offset of the record we just sent is: 99

对于发送到同一分区的记录，保证按顺序执行回调，也就是说，在上面的示例中，保证在callback2之前执行callback1。