源码分析kafka是如何发送消息的（一）

最新推荐文章于 2024-09-12 18:54:58 发布

陈煦源

最新推荐文章于 2024-09-12 18:54:58 发布

阅读量457

点赞数 1

分类专栏： kafka

本文链接：https://blog.csdn.net/trntaken/article/details/105333422

版权

kafka 专栏收录该内容

3 篇文章 0 订阅

订阅专栏

介绍：

从这章开始讲kafka生产者。首先讲的是kafka的发送消息流程。kafka发送消息的流程和rocketmq有点类似，无非就是创建网络连接，获取kafka集群broker 的ip port地址，topic 的分区partition信息，选择一个分区partition然后将消息发送到该分区partition对应的broker上。事实上是不是这样的呢？

一条消息是如何发送到kafka的？

上面我们简单说了一下kafka发送消息的流程，现在让我们看一下一个发送消息的demo，如下图，创建一个kafkaProducer对象，指定kafka地址，创建一条消息message，指定mesage的topic，调send方法将消息发送出去。

  Properties properties=new Properties();
        properties.put("bootstrap.servers","localhost:9092");//kafkabroker 地址
        properties.put("key.serializer","org.apache.kafka.common.serialization.StringSerializer");
        properties.put("value.serializer","org.apache.kafka.common.serialization.StringSerializer");
        properties.put("acks","all");//ISR集合都写入才返回
//        properties.put("enable.idempotence",true);
        properties.put("batch.size","1");//
        KafkaProducer<String,String>  producer=new KafkaProducer(properties);
        ProducerRecord record = new ProducerRecord("topic-test-1", "test-2", "value");//消息
        Future<RecordMetadata> value = producer.send(record);//send 这里异步发送
        RecordMetadata recordMetadata = value.get();//获取结果
        producer.flush();
        System.out.println("ProducerDemo send result ,  topic="+recordMetadata.topic()+" ,partition="
                +recordMetadata.partition()+", offset="+recordMetadata.offset());

调producer.send方法之后到底发生了什么呢？ client是如何获取topic所有分区信息的呢？topic又会选择topic哪个分区发送消息呢？消息是同步发送到kafka还是异步发送到kafka呢？我们下面进入 send方法看一下：

  @Override
    public Future<RecordMetadata> send(ProducerRecord<K, V> record) {
        return send(record, null);
    }
 
public Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback) {
        // intercept the record, which can be potentially modified; this method does not throw exceptions
        ProducerRecord<K, V> interceptedRecord = this.interceptors == null ? record : this.interceptors.onSend(record);
        return doSend(interceptedRecord, callback);
    }

从上面可以看出调用过程，我们发现有个 ProducerInterceptors 拦截器，这个拦截器提供了几个方法，会在消息发送前，和发送后执行，如果我们需要在消息发送前或发送成功后做点事情，可以实现这个拦截器。

接着往下看，未完待续。。。。

 private Future<RecordMetadata> doSend(ProducerRecord<K, V> record, Callback callback) {
        TopicPartition tp = null;
        try {
            // first make sure the metadata for the topic is available
            long waitedOnMetadataMs = waitOnMetadata(record.topic(), this.maxBlockTimeMs);
            long remainingWaitMs = Math.max(0, this.maxBlockTimeMs - waitedOnMetadataMs);
            byte[] serializedKey;
            try {
                serializedKey = keySerializer.serialize(record.topic(), record.key());
            } catch (ClassCastException cce) {
                throw new SerializationException("Can't convert key of class " + record.key().getClass().getName() +
                        " to class " + producerConfig.getClass(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG).getName() +
                        " specified in key.serializer");
            }
            byte[] serializedValue;
            try {
                serializedValue = valueSerializer.serialize(record.topic(), record.value());
            } catch (ClassCastException cce) {
                throw new SerializationException("Can't convert value of class " + record.value().getClass().getName() +
                        " to class " + producerConfig.getClass(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG).getName() +
                        " specified in value.serializer");
            }
            int partition = partition(record, serializedKey, serializedValue, metadata.fetch());
            int serializedSize = Records.LOG_OVERHEAD + Record.recordSize(serializedKey, serializedValue);
            ensureValidRecordSize(serializedSize);
            tp = new TopicPartition(record.topic(), partition);
            long timestamp = record.timestamp() == null ? time.milliseconds() : record.timestamp();
            log.trace("Sending record {} with callback {} to topic {} partition {}", record, callback, record.topic(), partition);
            // producer callback will make sure to call both 'callback' and interceptor callback
            Callback interceptCallback = this.interceptors == null ? callback : new InterceptorCallback<>(callback, this.interceptors, tp);
            RecordAccumulator.RecordAppendResult result = accumulator.append(tp, timestamp, serializedKey, serializedValue, interceptCallback, remainingWaitMs);
            if (result.batchIsFull || result.newBatchCreated) {
                log.trace("Waking up the sender since topic {} partition {} is either full or getting a new batch", record.topic(), partition);
                this.sender.wakeup();
            }
            return result.future;
            // handling exceptions and record the errors;
            // for API exceptions return them in the future,
            // for other exceptions throw directly
        } catch (ApiException e) {