本文详细介绍了如何从源码角度理解Kafka的异步发送过程和可能遇到的阻塞问题。从下载源码、搭建环境到分析生产者组件，特别是send方法中的拦截器和发送流程，包括消息增强、事务与幂等性、线程池的使用等关键点，揭示了Kafka异步发送的内部工作原理及其对性能的影响。
一、github下载kafka的源码
可以看到kafka的源码开源社区是非常活跃的。
二、搭建kafka环境
构建kafka环境，首先需要安装Scala和gradle，再安装的scala插件需要和你的idea需要匹配，同时不要忘了安装gradle时配置环境变量。
安装完之后，你就可以修改gradle.properties
group=org.apache.kafka
# NOTE: When you change this version number, you should also make sure to update
# the version numbers in
# - docs/js/templateData.js
# - tests/kafkatest/__init__.py
# - tests/kafkatest/version.py (variable DEV_VERSION)
# - kafka-merge-pr.py
version=1.1.2-SNAPSHOT
scalaVersion=2.11.12
task=build
org.gradle.jvmargs=-Xmx2g -Xss4m -XX:+UseParallelGC
修改完之后，就可以进行构建了。此时你需要输入gradle idea进行编译，这里由于我编译过了，所以时间较短，通常会较长时间。
重点关注example包：
三、生产者
生产者producer：
可以看到生产者里面有生产者、主题、是否是异步的相关变量，同时三个变量都是final，从而我们可以确切的直到它们都是在创建的时候就需要进行指定。构造函数中首先填充配置信息，将配置信息放入创建的kafka生产者中，将主题赋值给topic、生产者是否异步发送消息放入到isAsync。
重点在run方法中的send方法，发送方式分为同步和异步两种方式。
/**
* 生产者
*/
public class Producer extends Thread {
//生产者
private final KafkaProducer producer;//主题private final String topic;//是否是异步private final Boolean isAsync;//构造函数：配置信息服务器配置、客户端id、key序列化、value序列化、创建kafka生产者对象、主题、是否是异步public Producer(String topic, Boolean isAsync) {
Properties props = new Properties();
props.put("bootstrap.servers", KafkaProperties.KAFKA_SERVER_URL + ":" + KafkaProperties.KAFKA_SERVER_PORT);
props.put("client.id", "DemoProducer");
props.put("key.serializer", "org.apache.kafka.common.serialization.IntegerSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");//创建kafka生产者对象
producer = new KafkaProducer<>(props);this.topic = topic;this.isAsync = isAsync;
}//运行生产者public void run() {int messageNo = 1;while (true) {//发送的消息信息:Message_1
String messageStr = "Message_" + messageNo;//开始时间long startTime = System.currentTimeMillis();//是否异步if (isAsync) { // Send asynchronously//生产者发送消息 异步发送
producer.send(new ProducerRecord<>(topic,
messageNo,
messageStr), new DemoCallBack(startTime, messageNo, messageStr));
} else { // Send synchronouslytry {//同步发送
producer.send(new ProducerRecord<>(topic,
messageNo,
messageStr)).get();
System.out.println("Sent message: (" + messageNo + ", " + messageStr + ")");
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
}
++messageNo;
}
}
}//进行回调class DemoCallBack implements Callback {//开始时间private final long startTime;//keyprivate final int key;//消息private final String message;//构造函数public DemoCallBack(long startTime, int key, String message) {this.startTime = startTime;this.key = key;this.message = message;
}/**
* A callback method the user can implement to provide asynchronous handling of request completion. This method will
* be called when the record sent to the server has been acknowledged. Exactly one of the arguments will be
* non-null.
* 用户可以实现以提供对请求完成的异步处理的回调方法。 确认发送到服务器的消息后，将调用此方法。 确切地说，其中一个参数将为非null。
* @param metadata The metadata for the record that was sent (i.e. the partition and offset). Null if an error
* occurred.
* 发送的消息的元数据(即分区和偏移量)。 如果发生错误，则为Null。
* @param exception The exception thrown during processing of this record. Null if no error occurred.
* 处理此消息期间引发的异常。 如果没有发生错误，则为Null。
*/public void onCompletion(RecordMetadata metadata, Exception exception) {//经过的时间long elapsedTime = System.currentTimeMillis() - startTime;//如果元数据不为空，则返回null//打印消息发送到的分区和偏移量在经过的时间if (metadata != null) {
System.out.println("message(" + key + ", " + message + ") sent to partition(" + metadata.partition() +"), " +"offset(" + metadata.offset() + ") in " + elapsedTime + " ms");
} else {
exception.printStackTrace();
}
}
}send方法中做了两件事，一是对消息进行发送拦截，进行增强，同时进行消息发送。可以看到在send方法中有大段注释，而无疑这些注释是值得我们去读的。这里我查看的是异步的方法，通常异步的话，会进行回调。
这里大段的注释的大意：这里提到了消息发送和回调的方式，同时提到了存储中偏移量和分区，提到了事务、幂等，同时对事务进行了详细的介绍和在如果目标主题的消息格式未升级为0.11.0.0，则幂等和事务性生产请求将失败。并出现{@link org.apache.kafka.common.errors.UnsupportedForMessageFormatException}错误。如果在事务处理期间遇到此问题，则可以中止并继续。但是请注意，将来发送到同一主题的消息将继续收到相同的异常，直到升级该主题为止。同时对于阻塞和计算量大的方法需要自己实现线程池的并行。
下面是它的详细解释：
1.异步发送消息到主题，并在确认发送后调用提供的回调。
2.发送是异步的，并且一旦消息已存储在等待发送的消息缓冲区中，此方法将立即返回。这允许并行发送许多消息，而不会阻塞等待每条消息之后的响应。
3.发送的结果是{@link RecordMetadata}，指定消息发送到的分区，分配的偏移量和消息的时间戳。
如果主题使用{@link org.apache.kafka.common.record.TimestampType＃CREATE_TIME CreateTime}，则该时间戳记 将是用户提供的时间戳记，或者如果用户未为该消息指定时间戳记则是消息发送时间。
如果将{@link org.apache.kafka.common.record.TimestampType＃LOG_APPEND_TIME LogAppendTime}用作主题，则时间戳将是附加消息时的Kafka代理本地时间。
4.由于send调用是异步的，因此它将为将分配给该消息的{@link RecordMetadata}返回一个{@link java.util.concurrent.Future Future}。在此将来调用{@link java.util.concurrent.Future＃get()get()}将会阻塞，直到关联的请求完成，然后返回消息的元数据或引发在发送消息时发生的任何异常。
5.如果要模拟一个简单的阻塞调用，则可以立即调用get()方法
6.完全无阻塞的用法可以利用{@link Callback}参数来提供将在请求完成后调用的回调
7.当用作事务的一部分时，不必为了检测 send code>中的错误而定义回调或检查将来的结果。
如果任何发送调用失败并出现不可恢复的错误，则最后一个{@link #commitTransaction()}调用将失败，并从上次失败的发送中引发异常。
发生这种情况时，您的应用程序应调用{@link #abortTransaction()}以重置状态并继续发送数据。
8.某些事务发送错误无法通过调用{@link #abortTransaction()}来解决。特别是，如果事务发送以{@link ProducerFencedException}，{@ link org.apache.kafka.common.errors.OutOfOrderSequenceException}，
{@ link org.apache.kafka.common.errors.UnsupportedVersionException}结尾， 或{@link org.apache.kafka.common.errors.AuthorizationException}，那么剩下的唯一选择就是调用{@link #close()}。
致命错误导致生产者进入已失效状态，在这种状态下，将来的API调用将继续引发包裹在新{@link KafkaException}中的相同的下标错误。
9.这与启用幂等性但未配置 transactional.id code>时相似。在这种情况下，{@ link org.apache.kafka.common.errors.UnsupportedVersionException}和{@link org.apache.kafka.common.errors.AuthorizationException}被视为致命错误。但是，不需要处理{@link ProducerFencedException}。 此外，可以继续在收到{@link org.apache.kafka.common.errors.OutOfOrderSequenceException}之后发送消息，但这样做可能导致未决消息的发送顺序混乱。 为了确保正确的订阅，您应该关闭生产者并创建一个新实例。
10.如果目标主题的消息格式未升级为0.11.0.0，则幂等和事务性生产请求将失败，并出现{@link org.apache.kafka.common.errors.UnsupportedForMessageFormatException}错误。如果在事务处理期间遇到此问题，则可以中止并继续。 但是请注意，将来发送到同一主题的消息将继续收到相同的异常，直到升级该主题为止。
11. 注意，回调通常将在生产者的I/O线程中执行，因此应相当快，否则它们将延迟其他线程的消息发送。如果要执行阻塞或计算量大的回调，建议在回调主体中使用自己的{@link java.util.concurrent.Executor}来并行化处理。
@Override
public Future send(ProducerRecord record, Callback callback) {
// intercept the record, which can be potentially modified; this method does not throw exceptions
//拦截消息，进行增强
ProducerRecord interceptedRecord = this.interceptors.onSend(record);//发送消息return doSend(interceptedRecord, callback);
}拦截器拦截发送，进行自定义增强操作：
从注释里面我们可以看到其进行拦截的时候不会抛出异常，因此需要自己去try…catch
public ProducerRecord onSend(ProducerRecord record) {
ProducerRecord interceptRecord = record;for (ProducerInterceptor interceptor : this.interceptors) {try {//在发送中进行拦截
interceptRecord = interceptor.onSend(interceptRecord);
} catch (Exception e) {// do not propagate interceptor exception, log and continue calling other interceptors// be careful not to throw exception from hereif (record != null)
log.warn("Error executing interceptor onSend callback for topic: {}, partition: {}", record.topic(), record.partition(), e);else
log.warn("Error executing interceptor onSend callback", e);
}
}return interceptRecord;
}追加消息拦截器
//发送消息中拦截
@Override
public ProducerRecord onSend(ProducerRecord record) {
//计数器
onSendCount++;
//如果在发送中抛异常
if (throwExceptionOnSend)
throw new KafkaException("Injected exception in AppendProducerInterceptor.onSend");
//返回创建的生产者消息
return new ProducerRecord<>(
record.topic(), record.partition(), record.key(), record.value().concat(appendStr));
}
生产者消息：包含的信息主题、时间戳、分区、k-v、消息头
public ProducerRecord(String topic, Integer partition, Long timestamp, K key, V value, Iterable headers) {
//主题为空，抛异常
if (topic == null)
throw new IllegalArgumentException("Topic cannot be null.");
//时间戳不为空，或者时间戳小于0，则抛异常
if (timestamp != null && timestamp 0)
throw new IllegalArgumentException(
String.format("Invalid timestamp: %d. Timestamp should always be non-negative or null.", timestamp));
//分区不为空，或者分区小于0，则抛异常
if (partition != null && partition 0)
throw new IllegalArgumentException(
String.format("Invalid partition: %d. Partition number should always be non-negative or null.", partition));
//主题topic
this.topic = topic;
//分区
this.partition = partition;
//key
this.key = key;
//值
this.value = value;
//时间戳
this.timestamp = timestamp;
//消息头
this.headers = new RecordHeaders(headers);
}
doSend是我们需要关注的重点：
/**
* Implementation of asynchronously send a record to a topic.
* 实现的异步发送的记录到一个主题中
*/
private Future doSend(ProducerRecord record, Callback callback) {
TopicPartition tp = null;
try {
// first make sure the metadata for the topic is available
//首先确保元数据提供给topic是可用的 也即准备元数据阶段
ClusterAndWaitTime clusterAndWaitTime = waitOnMetadata(record.topic(), record.partition(), maxBlockTimeMs);
//记录等待时间
long remainingWaitMs = Math.max(0, maxBlockTimeMs - clusterAndWaitTime.waitedOnMetadataMs);
//集群
Cluster cluster = clusterAndWaitTime.cluster;
byte[] serializedKey;
try {
//序列化key
serializedKey = keySerializer.serialize(record.topic(), record.headers(), record.key());
} catch (ClassCastException cce) {
throw new SerializationException("Can't convert key of class " + record.key().getClass().getName() +
" to class " + producerConfig.getClass(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG).getName() +
" specified in key.serializer", cce);
}
byte[] serializedValue;
try {
//序列化value
serializedValue = valueSerializer.serialize(record.topic(), record.headers(), record.value());
} catch (ClassCastException cce) {
throw new SerializationException("Can't convert value of class " + record.value().getClass().getName() +
" to class " + producerConfig.getClass(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG).getName() +
" specified in value.serializer", cce);
}
//分区
int partition = partition(record, serializedKey, serializedValue, cluster);
tp = new TopicPartition(record.topic(), partition);
//将消息头转成数组
setReadOnly(record.headers());
Header[] headers = record.headers().toArray();
//序列化大小
int serializedSize = AbstractRecords.estimateSizeInBytesUpperBound(apiVersions.maxUsableProduceMagic(),
compressionType, serializedKey, serializedValue, headers);
//确保校验消息大小
ensureValidRecordSize(serializedSize);
long timestamp = record.timestamp() == null ? time.milliseconds() : record.timestamp();
log.trace("Sending record {} with callback {} to topic {} partition {}", record, callback, record.topic(), partition);
// producer callback will make sure to call both 'callback' and interceptor callback
//生产者回调将确保同时调用“回调”和拦截器回调
Callback interceptCallback = new InterceptorCallback<>(callback, this.interceptors, tp);
//如果事务管理不为空同时是有事务的，则添加事务
if (transactionManager != null && transactionManager.isTransactional())
transactionManager.maybeAddPartitionToTransaction(tp);
//在消息收集器中追加信息
RecordAccumulator.RecordAppendResult result = accumulator.append(tp, timestamp, serializedKey,
serializedValue, headers, interceptCallback, remainingWaitMs);
//如果结果为空，则sender唤醒 重点
if (result.batchIsFull || result.newBatchCreated) {
log.trace("Waking up the sender since topic {} partition {} is either full or getting a new batch", record.topic(), partition);
this.sender.wakeup();
}
return result.future;
// handling exceptions and record the errors;
// for API exceptions return them in the future,
// for other exceptions throw directly
} catch (ApiException e) {
//api接口异常
log.debug("Exception occurred during message send:", e);
if (callback != null)
callback.onCompletion(null, e);
this.errors.record();
this.interceptors.onSendError(record, tp, e);
return new FutureFailure(e);
} catch (InterruptedException e) {
//中断异常
this.errors.record();
this.interceptors.onSendError(record, tp, e);
throw new InterruptException(e);
} catch (BufferExhaustedException e) {
//缓冲区耗尽异常
this.errors.record();
this.metrics.sensor("buffer-exhausted-records").record();
this.interceptors.onSendError(record, tp, e);
throw e;
} catch (KafkaException e) {
//kafka异常
this.errors.record();
this.interceptors.onSendError(record, tp, e);
throw e;
} catch (Exception e) {
// we notify interceptor about all exceptions, since onSend is called before anything else in this method
this.interceptors.onSendError(record, tp, e);
throw e;
}
}
元数据准备阶段
* Wait for cluster metadata including partitions for the given topic to be available.
* 等待集群元数据包括给定主题的分区可用。
* @param topic The topic we want metadata for
* @param partition A specific partition expected to exist in metadata, or null if there's no preference
* @param maxWaitMs The maximum time in ms for waiting on the metadata
* @return The cluster containing topic metadata and the amount of time we waited in ms
*/
private ClusterAndWaitTime waitOnMetadata(String topic, Integer partition, long maxWaitMs) throws InterruptedException {
// add topic to metadata topic list if it is not there already and reset expiry
//添加主题
metadata.add(topic);
//获取当前的集群信息而不会阻塞
Cluster cluster = metadata.fetch();
//统计分区数
Integer partitionsCount = cluster.partitionCountForTopic(topic);
// Return cached metadata if we have it, and if the record's partition is either undefined
// or within the known partition range
//返回缓存的元数据(如果有)，并且记录的分区未定义或在已知分区范围内
if (partitionsCount != null && (partition == null || partition return new ClusterAndWaitTime(cluster, 0);
//开始时间
long begin = time.milliseconds();
//记录等待时间
long remainingWaitMs = maxWaitMs;
long elapsed;
// Issue metadata requests until we have metadata for the topic or maxWaitTimeMs is exceeded.
// In case we already have cached metadata for the topic, but the requested partition is greater
// than expected, issue an update request only once. This is necessary in case the metadata
// is stale and the number of partitions for this topic has increased in the meantime.
/**
*发出元数据请求，直到超过该主题的元数据或maxWaitTimeMs。如果我们已经为该主题缓存了元数据，
*但是请求的分区大于预期，则仅发出一次更新请求。
*如果元数据过时并且与此主题相关的分区数量同时增加，则这是必需的。
*/
do {
log.trace("Requesting metadata update for topic {}.", topic);
//添加主题
metadata.add(topic);
//拿到版本
int version = metadata.requestUpdate();
//唤醒sender
sender.wakeup();
try {
//等待更新
metadata.awaitUpdate(version, remainingWaitMs);
} catch (TimeoutException ex) {
// Rethrow with original maxWaitMs to prevent logging exception with remainingWaitMs
throw new TimeoutException("Failed to update metadata after " + maxWaitMs + " ms.");
}
//获取集群数据
cluster = metadata.fetch();
//计算时间
elapsed = time.milliseconds() - begin;
if (elapsed >= maxWaitMs)
throw new TimeoutException("Failed to update metadata after " + maxWaitMs + " ms.");
if (cluster.unauthorizedTopics().contains(topic))
throw new TopicAuthorizationException(topic);
remainingWaitMs = maxWaitMs - elapsed;
partitionsCount = cluster.partitionCountForTopic(topic);
} while (partitionsCount == null);
if (partition != null && partition >= partitionsCount) {
throw new KafkaException(
String.format("Invalid partition given with record: %d is not in the range [0...%d).", partition, partitionsCount));
}
return new ClusterAndWaitTime(cluster, elapsed);
}
集群和等待时间
//集群和等待时间
private static class ClusterAndWaitTime {
//集群
final Cluster cluster;
//等待在元数据的时间
final long waitedOnMetadataMs;
ClusterAndWaitTime(Cluster cluster, long waitedOnMetadataMs) {
this.cluster = cluster;
this.waitedOnMetadataMs = waitedOnMetadataMs;
}
}
这里值得我们关注的sender:
sender相关变量和构造函数
public class Sender implements Runnable {
private final Logger log;
/* the state of each nodes connection */
//kafka客户端 每个节点连接的状态
private final KafkaClient client;
/* the record accumulator that batches records */
//消息收集器 批量消息
private final RecordAccumulator accumulator;
/* the metadata for the client */
//元数据
private final Metadata metadata;
/* the flag indicating whether the producer should guarantee the message order on the broker or not. */
//生产者是否应保证broker上的消息顺序的标志
private final boolean guaranteeMessageOrder;
/* the maximum request size to attempt to send to the server */
//尝试发送到服务器的最大请求大小
private final int maxRequestSize;
/* the number of acknowledgements to request from the server */
//要从服务器请求的确认数
private final short acks;
/* the number of times to retry a failed request before giving up */
//放弃之前重试失败请求的次数
private final int retries;
/* the clock instance used for getting the time */
//用于获取时间的时钟实例
private final Time time;
/* true while the sender thread is still running */
//当发送方线程仍在运行时为true
private volatile boolean running;
/* true when the caller wants to ignore all unsent/inflight messages and force close. */
//当caller想忽略所有未发送/正在进行的消息并强制关闭时为true
private volatile boolean forceClose;
/* metrics */
//发送的度量信息 相关指标
private final SenderMetrics sensors;
/* the max time to wait for the server to respond to the request*/
//等待服务器响应请求的最长时间
private final int requestTimeout;
/* The max time to wait before retrying a request which has failed */
//重试失败的请求之前等待的最长时间
private final long retryBackoffMs;
/* current request API versions supported by the known brokers */
//已知broker支持的当前请求API版本
private final ApiVersions apiVersions;
/* all the state related to transactions, in particular the producer id, producer epoch, and sequence numbers */
//与事务相关的所有状态，特别是生产者ID，生产者时期和序列号
private final TransactionManager transactionManager;
//构造函数
public Sender(LogContext logContext,
KafkaClient client,
Metadata metadata,
RecordAccumulator accumulator,boolean guaranteeMessageOrder,int maxRequestSize,short acks,int retries,
SenderMetricsRegistry metricsRegistry,
Time time,int requestTimeout,long retryBackoffMs,
TransactionManager transactionManager,
ApiVersions apiVersions) {
//通过日志上下文拿到日志信息
this.log = logContext.logger(Sender.class);
//客户端
this.client = client;
//消息收集器
this.accumulator = accumulator;
//元数据
this.metadata = metadata;
//保证消息有序
this.guaranteeMessageOrder = guaranteeMessageOrder;
//最大请求大小
this.maxRequestSize = maxRequestSize;
//运行
this.running = true;
//acks
this.acks = acks;
//重试次数
this.retries = retries;
//时间
this.time = time;
//发送指标信息
this.sensors = new SenderMetrics(metricsRegistry);
//请求超时时间
this.requestTimeout = requestTimeout;
//重试间隔的最短时间
this.retryBackoffMs = retryBackoffMs;
//api版本信息
this.apiVersions = apiVersions;
//事务管理器
this.transactionManager = transactionManager;
}
}
run方法：
/**
* The main run loop for the sender thread
* 发送消息线程的主运行循环
*/
public void run() {
log.debug("Starting Kafka producer I/O thread.");
// main loop, runs until close is called
while (running) {
try {
//执行run方法 重点
run(time.milliseconds());
} catch (Exception e) {
log.error("Uncaught error in kafka producer I/O thread: ", e);
}
}
log.debug("Beginning shutdown of Kafka producer I/O thread, sending remaining records.");
// okay we stopped accepting requests but there may still be
// requests in the accumulator or waiting for acknowledgment,
// wait until these are completed.
while (!forceClose && (this.accumulator.hasUndrained() || this.client.inFlightRequestCount() > 0)) {
try {
//执行run方法 重点
run(time.milliseconds());
} catch (Exception e) {
log.error("Uncaught error in kafka producer I/O thread: ", e);
}
}
if (forceClose) {
// We need to fail all the incomplete batches and wake up the threads waiting on
// the futures.
log.debug("Aborting incomplete batches due to forced shutdown");
this.accumulator.abortIncompleteBatches();
}
try {
this.client.close();
} catch (Exception e) {
log.error("Failed to close network client", e);
}
log.debug("Shutdown of Kafka producer I/O thread has completed.");
}
执行发送数据
/**
* Run a single iteration of sending
* 运行一次发送
* @param now The current POSIX time in milliseconds
*/
void run(long now) {
//如果事务不为空，则放入事务信息
if (transactionManager != null) {
try {
if (transactionManager.shouldResetProducerStateAfterResolvingSequences())
// Check if the previous run expired batches which requires a reset of the producer state.
//重置生产者id
transactionManager.resetProducerId();
if (!transactionManager.isTransactional()) {
// this is an idempotent producer, so make sure we have a producer id
maybeWaitForProducerId();
} else if (transactionManager.hasUnresolvedSequences() && !transactionManager.hasFatalError()) {
transactionManager.transitionToFatalError(new KafkaException("The client hasn't received acknowledgment for " +
"some previously sent messages and can no longer retry them. It isn't safe to continue."));
} else if (transactionManager.hasInFlightTransactionalRequest() || maybeSendTransactionalRequest(now)) {
// as long as there are outstanding transactional requests, we simply wait for them to return
client.poll(retryBackoffMs, now);
return;
}
// do not continue sending if the transaction manager is in a failed state or if there
// is no producer id (for the idempotent case).
if (transactionManager.hasFatalError() || !transactionManager.hasProducerId()) {
RuntimeException lastError = transactionManager.lastError();
if (lastError != null)
maybeAbortBatches(lastError);
client.poll(retryBackoffMs, now);
return;
} else if (transactionManager.hasAbortableError()) {
accumulator.abortUndrainedBatches(transactionManager.lastError());
}
} catch (AuthenticationException e) {
// This is already logged as error, but propagated here to perform any clean ups.
log.trace("Authentication exception while processing transactional request: {}", e);
transactionManager.authenticationFailed(e);
}
}
//发送生产者数据 重点
long pollTimeout = sendProducerData(now);
//执行poll轮询操作 进行读取和写入操作
client.poll(pollTimeout, now);
}
此时进入重要方法sendProduerData()方法和poll()方法。
604
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
