kafka源码解读系列：深入生产者核心

本文档基于kafka client 0.8.2.2 版本api进行代码详细说明，通过对该代码的深入剖析，可以加深对kafka生产者的理解，并且能够避免该版本坑点（说实话，该版本挺坑，建议升级到0.9以上）
通过阅读本文章，您将深刻理解kafka生产者内部逻辑，并且对0.8.2.2版本生产者存在的阻塞问题进行了代码级剖析，并根据代码调用逻辑整理了时序图以加深理解

1、版本：0.8.2.2

本文章对0.8.2.2版本kafka客户端进行源码剖析，其他版本大体上逻辑类似

2、结构图

首先，对kafka生产者内部结构进行剖析，如下图所示

上图表示kafka生产者生产消息，缓存消息，发送消息的内部流程与结构说明图，按官方的说法，send方法是异步执行的，从上图可以看到在KafkaProducer的send方法被调用后，数据将缓存进入RecordAccumulator对象实例（内部按分区存储），到这里客户端已经消息发送完成。最后由Sender对象开启线程实时读取RecordAccumulator内部数据，这是一个异步过程

3、KafkaProducer：生产者

摘取主要代码

//【重要】每次发送数据的大小，由max.request.size设置,默认1M（消息不是按照批次发送的）
private final int maxRequestSize;
//获取元数据超时阈值
private final long metadataFetchTimeoutMs;
//【重要】缓冲池总大小，每批次消息buffer由消息缓冲池中申请内存，
private final long totalMemorySize;
//kafka元数据，保存topic，partition，leader，isr等数据
private final Metadata metadata;
//【重要】消息累加器：顾名思义用于缓存消息
private final RecordAccumulator accumulator;
//【重要】消息发送者，读取消息累加器中的消息发送
private final Sender sender;

//发送消息，异步
public Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback) {
        try {
            //1、等待获取topic对应的元数据（如果还未获取，就等待一段时间，**可能导致堵塞**）
            this.waitOnMetadata(record.topic(), this.metadataFetchTimeoutMs);
            //2、序列化key和value
            byte[] serializedKey;
            try {
                serializedKey = this.keySerializer.serialize(record.topic(), record.key());
            } catch (ClassCastException var11) {
                throw new SerializationException("Can't convert key of class " + record.key().getClass().getName() + " to class " + this.producerConfig.getClass("key.serializer").getName() + " specified in key.serializer");
            }
            byte[] serializedValue;
            try {
                serializedValue = this.valueSerializer.serialize(record.topic(), record.value());
            } catch (ClassCastException var10) {
                throw new SerializationException("Can't convert value of class " + record.value().getClass().getName() + " to class " + this.producerConfig.getClass("value.serializer").getName() + " specified in value.serializer");
            }
            //3、构造消息对象
            ProducerRecord<byte[], byte[]> serializedRecord = new ProducerRecord(record.topic(), record.partition(), serializedKey, serializedValue);
            //4、计算消息对象归属的分区号
            int partition = this.partitioner.partition(serializedRecord, this.metadata.fetch());
            //5、获取序列化后的数据大小（包含key和value）
            int serializedSize = 12 + Record.recordSize(serializedKey, serializedValue);
            //6、验证单条记录大小，是否超过设置上限
            this.ensureValidRecordSize(serializedSize);
            //7、构造分区对象
            TopicPartition tp = new TopicPartition(record.topic(), partition);
            log.trace("Sending record {} with callback {} to topic {} partition {}", new Object[]{record, callback, record.topic(), partition});
            //8、消息加入累加器
            RecordAppendResult result = this.accumulator.append(tp, serializedKey, serializedValue, this.compressionType, callback);
            //9、如果累加器满了，那么强制唤醒发送者
            if (result.batchIsFull || result.newBatchCreated) {
                log.trace("Waking up the sender since topic {} partition {} is either full or getting a new batch", record.topic(), partition);
                this.sender.wakeup();
            }

            return result.future;
        } catch (ApiException var12) {
            log.debug("Exception occurred during message send:", var12);
            if (callback != null) {
                callback.onCompletion((RecordMetadata)null, var12);
            }

            this.errors.record();
            return new KafkaProducer.FutureFailure(var12);
        } catch (InterruptedException var13) {
            this.errors.record();
            throw new KafkaException(var13);
        } catch (KafkaException var14) {
            this.errors.record();
            throw var14;
        }
}


//【重要】获取元数据，最长执行时间根据入参maxWaitMs决定，maxWaitMs参数由metadata.fetch.timeout.ms设置，默认60s
private void waitOnMetadata(String topic, long maxWaitMs) {
        if (this.metadata.fetch().partitionsForTopic(topic) == null) {
            long begin = this.time.milliseconds();

            long elapsed;
            for(long remainingWaitMs = maxWaitMs; this.metadata.fetch().partitionsForTopic(topic) == null; remainingWaitMs = maxWaitMs - elapsed) {
                log.trace("Requesting metadata update for topic {}.", topic);
                int version = this.metadata.requestUpdate();
                this.metadata.add(topic);
                this.sender.wakeup();
                this.metadata.awaitUpdate(version, remainingWaitMs);
                elapsed = this.time.milliseconds() - begin;
                if (elapsed >= maxWaitMs) {
                    throw new TimeoutException("Failed to update metadata after " + maxWaitMs + " ms.");
                }
            }

        }
}

//确认单条记录大小是否超过阈值
private void ensureValidRecordSize(int size) {
    	//超过发送大小阈值
        if (size > this.maxRequestSize) {
            throw new RecordTooLargeException("The message is " + size + " bytes when serialized which is larger than the maximum request size you have configured with the " + "max.request.size" + " configuration.");
            //超过消息缓冲池总大小阈值
        } else if ((long)size > this.totalMemorySize) { 
            throw new RecordTooLargeException("The message is " + size + " bytes when serialized which is larger than the total memory buffer you have configured with the " + "buffer.memory" + " configuration.");
        }
}

总结：在该类中，核心方法是send方法，该方法内部代码逻辑主要分为3个步骤

第一步：获取kafka服务端元数据

第二步：构造消息对象

第三步：消息加入累加器

由此可见，该方法把消息加入累加器后就直接返回，因此该方法是异步的

4、RecordAccumulator：消息累加器

摘取主要代码
该类由KafkaProducer创建并持有，用于缓存用户消息

//【重要】消息批次大小，由batch.size设置，默认16KB
private final int batchSize;
//【重要】消息缓存区对象池，大小由buffer.memory设置
private final BufferPool free;
//【重要】批次数据，按分区存储
private final ConcurrentMap<TopicPartition, Deque<RecordBatch>> batches;
//关键方法：缓存消息对象
public RecordAccumulator.RecordAppendResult append(TopicPartition tp, byte[] key, byte[] value, CompressionType compression, Callback callback) throws InterruptedException {
        if (this.closed) {
            throw new IllegalStateException("Cannot send after the producer is closed.");
        } else {
            //根据分区对象，取出消息队列中的消息批次对象
            Deque<RecordBatch> dq = this.dequeFor(tp);
            synchronized(dq) {
                //取出队列中的批次对象
                RecordBatch last = (RecordBatch)dq.peekLast();
                //如果批次对象未满，直接把消息存入批次对象中后返回
                if (last != null) {
                    FutureRecordMetadata future = last.tryAppend(key, value, callback);
                    if (future != null) {
                        return new RecordAccumulator.RecordAppendResult(future, dq.size() > 1 || last.records.isFull(), false);
                    }
                }
            }
			//批次对象已经满的情况
            //【重要】计算新批次对象大小（取最大值：批次大小阈值或记录大小）
            int size = Math.max(this.batchSize, 12 + Record.recordSize(key, value));
            log.trace("Allocating a new {} byte message buffer for topic {} partition {}", new Object[]{size, tp.topic(), tp.partition()});
            //分配批次对象缓冲区（缓冲区无空间可分配后，将阻塞***）
            ByteBuffer buffer = this.free.allocate(size);
            //再判断一次批次对象是否满了（数据被发送完了，就会释放）==》需查看sender发送机制
            synchronized(dq) {  
                RecordBatch last = (RecordBatch)dq.peekLast();
                //如果批次对象未满，直接把消息存入批次对象中后，回收刚才分配的缓冲区
                if (last != null) {
                    FutureRecordMetadata future = last.tryAppend(key, value, callback);
                    if (future != null) {
                        this.free.deallocate(buffer);
                        return new RecordAccumulator.RecordAppendResult(future, dq.size() > 1 || last.records.isFull(), false);
                    }
                }
				//构造新的批次对象，加入消息队列中
                MemoryRecords records = MemoryRecords.emptyRecords(buffer, compression, this.batchSize);
                RecordBatch batch = new RecordBatch(tp, records, this.time.milliseconds());
                FutureRecordMetadata future = (FutureRecordMetadata)Utils.notNull(batch.tryAppend(key, value, callback));
                dq.addLast(batch);
                return new RecordAccumulator.RecordAppendResult(future, dq.size() > 1 || batch.records.isFull(), true);
            }
        }
    }


//Sender定时调用，生成已经做好发送数据准备的leader所在的节点对象集合
public RecordAccumulator.ReadyCheckResult ready(Cluster cluster, long nowMs) {
        Set<Node> readyNodes = new HashSet();
        long nextReadyCheckDelayMs = 9223372036854775807L;
        boolean unknownLeadersExist = false;
        boolean exhausted = this.free.queued() > 0;
        Iterator i$ = this.batches.entrySet().iterator();

        while(true) {
            while(i$.hasNext()) {
                Entry<TopicPartition, Deque<RecordBatch>> entry = (Entry)i$.next();
                TopicPartition part = (TopicPartition)entry.getKey();
                Deque<RecordBatch> deque = (Deque)entry.getValue();
                Node leader = cluster.leaderFor(part);
                if (leader == null) {
                    unknownLeadersExist = true;
                } else if (!readyNodes.contains(leader)) {
                    synchronized(deque) {
                        RecordBatch batch = (RecordBatch)deque.peekFirst();
                        if (batch != null) {
                            boolean backingOff = batch.attempts > 0 && batch.lastAttemptMs + this.retryBackoffMs > nowMs;
                            long waitedTimeMs = nowMs - batch.lastAttemptMs;
                            long timeToWaitMs = backingOff ? this.retryBackoffMs : this.lingerMs;
                            long timeLeftMs = Math.max(timeToWaitMs - waitedTimeMs, 0L);
                            boolean full = deque.size() > 1 || batch.records.isFull();
                            boolean expired = waitedTimeMs >= timeToWaitMs;
                            boolean sendable = full || expired || exhausted || this.closed;
                            if (sendable && !backingOff) {
                                readyNodes.add(leader);
                            } else {
                                nextReadyCheckDelayMs = Math.min(timeLeftMs, nextReadyCheckDelayMs);
                            }
                        }
                    }
                }
            }

            return new RecordAccumulator.ReadyCheckResult(readyNodes, nextReadyCheckDelayMs, unknownLeadersExist);
        }
}

//sender定时调用，获取本次发送的消息集合
public Map<Integer, List<RecordBatch>> drain(Cluster cluster, Set<Node> nodes, int maxSize, long now) {
        if (nodes.isEmpty()) {
            return Collections.emptyMap();
        } else {
            Map<Integer, List<RecordBatch>> batches = new HashMap();

            Node node;
            ArrayList ready;
            for(Iterator i$ = nodes.iterator(); i$.hasNext(); batches.put(node.id(), ready)) {
                node = (Node)i$.next();
                int size = 0;
                List<PartitionInfo> parts = cluster.partitionsForNode(node.id());
                ready = new ArrayList();
                int start = this.drainIndex %= parts.size();

                do {
                    PartitionInfo part = (PartitionInfo)parts.get(this.drainIndex);
                    Deque<RecordBatch> deque = this.dequeFor(new TopicPartition(part.topic(), part.partition()));
                    if (deque != null) {
                        synchronized(deque) {
                            RecordBatch first = (RecordBatch)deque.peekFirst();
                            if (first != null) {
                                if (size + first.records.sizeInBytes() > maxSize && !ready.isEmpty()) {
                                    break;
                                }

                                RecordBatch batch = (RecordBatch)deque.pollFirst();
                                batch.records.close();
                                size += batch.records.sizeInBytes();
                                ready.add(batch);
                                batch.drainedMs = now;
                            }
                        }
                    }

                    this.drainIndex = (this.drainIndex + 1) % parts.size();
                } while(start != this.drainIndex);
            }

            return batches;
        }
    }

//释放空间，由sender发送成功后调用
public void deallocate(RecordBatch batch) {
        this.free.deallocate(batch.records.buffer(), batch.records.capacity());
    }

总结：该类中有4个核心方法，分别是append，ready，drain，deallocate四个方法

append方法：由KafakProducer对象把消息加入累加器中时调用，代理逻辑主要有2步，第一步：若已有的消息批次还有空间存储，把当前消息存入该批次；第二步：若已有的消息批次空间无法容纳该消息，申请新的存储空间进行存储

ready方法：由发送者线程sender调用，返回可用节点对象集合

drain方法：由发送者线程sender调用，返回本批次发送的数据集合

deallocate方法：由发送者线程sender调用，回收之前分配的空间（该方法在消息发送成功后被调用）

5、RecordBatch：消息批次

摘取主要方法
该类由RecordAccumulator创建并持有，在消息累加器中进行消息批次存储

public int recordCount = 0;
public int maxRecordSize = 0;
public long drainedMs;
public final MemoryRecords records;
//尝试把消息加入该消息批次，如果空间不够，返回null
public FutureRecordMetadata tryAppend(byte[] key, byte[] value, Callback callback) {
    	
        if (!this.records.hasRoomFor(key, value)) {
            return null;
        } else {
            this.records.append(0L, key, value);
            //记录该批次中，单体消息最大的大小
            this.maxRecordSize = Math.max(this.maxRecordSize, Record.recordSize(key, value));
            FutureRecordMetadata future = new FutureRecordMetadata(this.produceFuture, (long)this.recordCount);
            if (callback != null) {
                this.thunks.add(new RecordBatch.Thunk(callback, future));
            }
			//累加本批次消息数量
            ++this.recordCount;
            return future;
        }
    }

//sender数据发送完成后回调
public void done(long baseOffset, RuntimeException exception) {
        this.produceFuture.done(this.topicPartition, baseOffset, exception);
        log.trace("Produced messages to topic-partition {} with base offset offset {} and error: {}.", new Object[]{this.topicPartition, baseOffset, exception});

        for(int i = 0; i < this.thunks.size(); ++i) {
            try {
                RecordBatch.Thunk thunk = (RecordBatch.Thunk)this.thunks.get(i);
                //回调回调函数
                if (exception == null) {
                    thunk.callback.onCompletion(thunk.future.get(), (Exception)null);
                } else {
                    thunk.callback.onCompletion((RecordMetadata)null, exception);
                }
            } catch (Exception var6) {
                log.error("Error executing user-provided callback on message for topic-partition {}:", this.topicPartition, var6);
            }
        }

    }

总结：该类中有2个核心方法，分别是tryAppend和done方法

tryAppend方法：由消息累加器RecordAccumulator调用，把消息加入该批次空间中

done方法：由发送者线程sender调用，在数据发送完成后回调，主要功能为回调用户传入的回调函数

6、MemoryRecords：消息批次数据

摘取主要方法
该类由RecordBatch创建并持有

//数据压缩机
private final Compressor compressor;
//批次实际大小，由buffer.capacity()回填
private final int capacity;
//批次配置大小，等于batch.size设置的值
private final int sizeLimit;
//批次数据缓存
private ByteBuffer buffer;
//是否可写
private boolean writable;

private MemoryRecords(ByteBuffer buffer, CompressionType type, boolean writable, int sizeLimit) {
        this.writable = writable;
        this.capacity = buffer.capacity();
        this.sizeLimit = sizeLimit;
    	//可写入状态的话
        if (this.writable) {
            this.buffer = null;
            this.compressor = new Compressor(buffer, type);
        } else {
            this.buffer = buffer;
            this.compressor = null;
        }

}
//生成空对象
public static MemoryRecords emptyRecords(ByteBuffer buffer, CompressionType type, int capacity) {
        return new MemoryRecords(buffer, type, true, capacity);
}

//判断是否可以容纳该条消息记录
public boolean hasRoomFor(byte[] key, byte[] value) {
        return this.writable && (long)this.capacity >= this.compressor.estimatedBytesWritten() + 12L + (long)Record.recordSize(key, value) && (long)this.sizeLimit >= this.compressor.estimatedBytesWritten();
}

//写入
public void append(long offset, byte[] key, byte[] value) {
        if (!this.writable) {
            throw new IllegalStateException("Memory records is not writable");
        } else {
            int size = Record.recordSize(key, value);
            this.compressor.putLong(offset);
            this.compressor.putInt(size);
            this.compressor.putRecord(key, value);
            this.compressor.recordWritten(size + 12);
        }
}

7、Compressor：消息数据压缩器

摘取主要方法，该类为消息最后写入该处，由MemoryRecords创建并持有

public Compressor(ByteBuffer buffer, CompressionType type, int blockSize) {
        this.type = type;
        this.initPos = buffer.position();
        this.numRecords = 0L;
        this.writtenUncompressed = 0L;
        if (type != CompressionType.NONE) {
            buffer.position(this.initPos + 12 + 14);
        }

        this.bufferStream = new ByteBufferOutputStream(buffer);
        this.appendStream = wrapForOutput(this.bufferStream, type, blockSize);
}

//估算可写入的空间大小
public long estimatedBytesWritten() {
        return this.type == CompressionType.NONE ? (long)this.bufferStream.buffer().position() : (long)((float)this.writtenUncompressed * typeToRate[this.type.id] * 1.05F);
}

8、BufferPool：消息缓冲池

摘取主要方法
该类使用buffer.memory参数设置大小，默认32M，由该缓冲池分配RecordBatch大小，被分配的总大小超过缓冲池总大小后将抛出异常
由RecordAccumulator创建并持有，该类的主要能力用于控制客户端消息缓存最大大小，作为消息池管理对象，管理整个消息空间的分配

//分配缓冲区，注意：该方法可能阻塞
public ByteBuffer allocate(int size) throws InterruptedException {
        if ((long)size > this.totalMemory) {
            throw new IllegalArgumentException("Attempt to allocate " + size + " bytes, but there is a hard limit of " + this.totalMemory + " on memory allocations.");
        } else {
            this.lock.lock();

            ByteBuffer var7;
            try {
                //有空间分配的情况，直接分配完返回
                //沿用之前分配好的
                if (size == this.poolableSize && !this.free.isEmpty()) {
                    ByteBuffer var14 = (ByteBuffer)this.free.pollFirst();
                    return var14;
                }
                //重新分配个新的
                int freeListSize = this.free.size() * this.poolableSize;
                if (this.availableMemory + (long)freeListSize >= (long)size) {
                    this.freeUp(size);
                    this.availableMemory -= (long)size;
                    this.lock.unlock();
                    ByteBuffer var15 = ByteBuffer.allocate(size);
                    return var15;
                }

                if (!this.blockOnExhaustion) {
                    throw new BufferExhaustedException("You have exhausted the " + this.totalMemory + " bytes of memory you configured for the client and the client is configured to error" + " rather than block when memory is exhausted.");
                }

                //没有空间分配的情况
                
                int accumulated = 0;
                ByteBuffer buffer = null;
                Condition moreMemory = this.lock.newCondition();
                this.waiters.addLast(moreMemory);

                while(accumulated < size) {
                    long startWait = this.time.nanoseconds();
                    //同步等待空间释放，阻塞（sender对象发送成功后，将调用signal唤醒阻塞）*****
                    moreMemory.await();
                    long endWait = this.time.nanoseconds();
                    this.waitTime.record((double)(endWait - startWait), this.time.milliseconds());
                    if (accumulated == 0 && size == this.poolableSize && !this.free.isEmpty()) {
                        buffer = (ByteBuffer)this.free.pollFirst();
                        accumulated = size;
                    } else {
                        this.freeUp(size - accumulated);
                        int got = (int)Math.min((long)(size - accumulated), this.availableMemory);
                        this.availableMemory -= (long)got;
                        accumulated += got;
                    }
                }

                Condition removed = (Condition)this.waiters.removeFirst();
                if (removed != moreMemory) {
                    throw new IllegalStateException("Wrong condition: this shouldn't happen.");
                }

                if ((this.availableMemory > 0L || !this.free.isEmpty()) && !this.waiters.isEmpty()) {
                    ((Condition)this.waiters.peekFirst()).signal();
                }

                this.lock.unlock();
                if (buffer == null) {
                    var7 = ByteBuffer.allocate(size);
                    return var7;
                }

                var7 = buffer;
            } finally {
                if (this.lock.isHeldByCurrentThread()) {
                    this.lock.unlock();
                }

            }

            return var7;
        }
    }

//释放空间，sender数据发送成功后调用
public void deallocate(ByteBuffer buffer, int size) {
        this.lock.lock();

        try {
            if (size == this.poolableSize && size == buffer.capacity()) {
                buffer.clear();
                this.free.add(buffer);
            } else {
                this.availableMemory += (long)size;
            }

            Condition moreMem = (Condition)this.waiters.peekFirst();
            if (moreMem != null) {
                //唤醒allocate中阻塞的线程 ******
                moreMem.signal();
            }
        } finally {
            this.lock.unlock();
        }

    }

9、Sender：消息发送者

摘取主要方法
该类通过线程启动，循环执行消息发送动作，由KafkaProducer创建并持有

public Sender(KafkaClient client, Metadata metadata, RecordAccumulator accumulator, int maxRequestSize, short acks, int retries, int requestTimeout, Metrics metrics, Time time, String clientId) {
        this.client = client;
        this.accumulator = accumulator;
        this.metadata = metadata;
        this.maxRequestSize = maxRequestSize;
        this.running = true;
        this.requestTimeout = requestTimeout;
        this.acks = acks;
        this.retries = retries;
        this.time = time;
        this.clientId = clientId;
        this.sensors = new Sender.SenderMetrics(metrics);
}

public void run() {
        log.debug("Starting Kafka producer I/O thread.");

        while(this.running) {
            try {
                this.run(this.time.milliseconds());
            } catch (Exception var3) {
                log.error("Uncaught error in kafka producer I/O thread: ", var3);
            }
        }

        log.debug("Beginning shutdown of Kafka producer I/O thread, sending remaining records.");

        while(this.accumulator.hasUnsent() || this.client.inFlightRequestCount() > 0) {
            try {
                this.run(this.time.milliseconds());
            } catch (Exception var2) {
                log.error("Uncaught error in kafka producer I/O thread: ", var2);
            }
        }

        this.client.close();
        log.debug("Shutdown of Kafka producer I/O thread has completed.");
}

public void run(long now) {
        Cluster cluster = this.metadata.fetch();
    	//查询已准备好的节点（leader不为空的）
        ReadyCheckResult result = this.accumulator.ready(cluster, now);
    	//重新设置获取元数据标识
        if (result.unknownLeadersExist) {
            this.metadata.requestUpdate();
        }

        Iterator<Node> iter = result.readyNodes.iterator();
        long notReadyTimeout = 9223372036854775807L;

        while(iter.hasNext()) {
            Node node = (Node)iter.next();
            if (!this.client.ready(node, now)) {
                iter.remove();
                notReadyTimeout = Math.min(notReadyTimeout, this.client.connectionDelay(node, now));
            }
        }
		//获取本次发送数据的集合
        Map<Integer, List<RecordBatch>> batches = this.accumulator.drain(cluster, result.readyNodes, this.maxRequestSize, now);
    	//构建请求
        List<ClientRequest> requests = this.createProduceRequests(batches, now);
        this.sensors.updateProduceRequestMetrics(requests);
        long pollTimeout = Math.min(result.nextReadyCheckDelayMs, notReadyTimeout);
        if (result.readyNodes.size() > 0) {
            log.trace("Nodes with data ready to send: {}", result.readyNodes);
            log.trace("Created {} produce requests: {}", requests.size(), requests);
            pollTimeout = 0L;
        }

        List<ClientResponse> responses = this.client.poll(requests, pollTimeout, now);
        Iterator i$ = responses.iterator();

        while(i$.hasNext()) {
            ClientResponse response = (ClientResponse)i$.next();
            if (response.wasDisconnected()) {
                this.handleDisconnect(response, now);
            } else {
                this.handleResponse(response, now);
            }
        }
}

private void completeBatch(RecordBatch batch, Errors error, long baseOffset, long correlationId, long now) {
        if (error != Errors.NONE && this.canRetry(batch, error)) {
            log.warn("Got error produce response with correlation id {} on topic-partition {}, retrying ({} attempts left). Error: {}", new Object[]{correlationId, batch.topicPartition, this.retries - batch.attempts - 1, error});
            this.accumulator.reenqueue(batch, now);
            this.sensors.recordRetries(batch.topicPartition.topic(), batch.recordCount);
        } else {
            //调用批次完成函数（）
            batch.done(baseOffset, error.exception());
            //归还缓冲池中已经分配的空间
            this.accumulator.deallocate(batch);
            if (error != Errors.NONE) {
                this.sensors.recordErrors(batch.topicPartition.topic(), batch.recordCount);
            }
        }

        if (error.exception() instanceof InvalidMetadataException) {
            this.metadata.requestUpdate();
        }

    }

10、时序图

根据上面的代码分析结果，我们可以总结消息生产与消息发送时序图如下

【消息生产时序图】

【消息发送时序图】

从以上代码可以看出，有2个方法可能对生产者的send方法导致阻塞或者异常

1、KafkaProducer.waitOnMetadata

在客户端程序启动时，如果此时kafka服务端处于挂死状态，
那么该方法将会执行一定的等待事件，耗时很长，最后抛出异常，启动失败

2、BufferPool.allocate

如果消息缓冲区空间被消耗殆尽（kafka服务端挂掉的情况下，导致消息积压，空间无法释放）的情况下，
再执行allocate后，将导致阻塞，直到服务端恢复后，数据被发送到服务端使得缓冲区空间被释放，该方法才会被执行

本次使用的kafka客户端版本是0.8.2.2版本，属于比较老旧的版本，0.9版本之后，对调用BufferPool.allocate设置了等待时间并且设置了批次存储的超时机制，批次数据存储超时后，将自动回收分配的空间并调用回调函数，因此不会阻塞

12、关键参数

这些关键参数将影响到生产者本地缓存及发送性能，在资源允许的情况下需按需设计

1）max.request.size

消息发送器每次发送消息的大小（发送消息不是按批次发的），默认1M

2）buffer.memory

消息缓冲池总大小，默认32M

3）batch.size

消息批次大小阈值，默认16K