文章目录
3.生产者源码初探
3.1 生产者基本原理
生产者发送消息基本过程如下图所示：
kakfa生产者客户端由两个线程协调运行，分别为main线程（主线程）和Sender线程（发送线程）。在主线程中由KafkaProducer创建消息，然后经过拦截器、序列化器和分区器之后缓存到消息累加器（RecordAccumulator）中。Sender 线程负责从RecordAccumulator中获取消息并将其发送到Kafka服务端。
其中数据结构会经过如下转变：
①Deque＜ProducerBatch＞
消息累加器（RecordAccumulator）在内部为每个分区维护了一个双端队列，队列中的内容就是ProducerBatch，即 Deque＜ProducerBatch＞，消息写入缓存时，追加到双端队列的尾部；Sender读取消息时，从双端队列的头部读取。其主要目的是用来缓存消息以便 Sender 线程可以批量发送，进而减少网络传输的资源消耗以提升性能。
②＜Node，List＜ ProducerBatch＞
Sender线程 从 RecordAccumulator 中获取缓存的消息之后，会进一步将原本＜分区，Deque＜ProducerBatch＞＞的保存形式转变成＜Node，List＜ ProducerBatch＞的形式，其中Node表示Kafka集群的broker节点。
③＜Node，Request＞
在转换成＜Node，List＜ProducerBatch＞＞的形式之后，Sender 线程还会进一步将其封装成＜Node，Request＞的形式，这样就可以将Request请求发往各个Node了。
④Map＜NodeId，Deque＜Request＞＞
Sender线程发往Kafka之前还会保存到InFlightRequests中，InFlightRequests保存对象的具体形式为Map＜NodeId，Deque＜Request＞＞，它的主要作用是缓存了已经发出去但还没有收到响应的请求，未确认的请求越多则认为负载越大。
设计的原因：Kafka生产者只关注向哪个分区中发送哪些消息，而进行网络传输时是向具体的 broker 节点发送消息，因此这里需要做一个应用逻辑层面到网络I/O层面的转换。
3.2 生产者源码
3.2.1 示例代码
public class KafkaProducerAnalysis {
//kafka broker的ip和端口
public static final String brokerList = "hadoop101:9092";
//发送消息的主题
public static final String topic = "topic-demo";
public static Properties initConfig() {
Properties props = new Properties();
props.put("bootstrap.servers", brokerList);
//设置序列化器
props.put("key.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
props.put("client.id", "producer.client.id.demo");
return props;
}
public static void main(String[] args) throws InterruptedException {
//1.配置生产者客户端参数
Properties props = initConfig();
//创建生产者实例
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
//2.构建待发送的消息ProducerRecord
ProducerRecord<String, String> record = new ProducerRecord<>(topic, "hello, Kafka!");
try {
//3.发送消息
producer.send(record);
} catch (Exception e) {
e.printStackTrace();
}finally {
//4.关闭生产者实例
producer.close();
}
}
}
其中生产者发送消息核心代码为创建生产者实例和发送消息两步，下面进行详细介绍。
3.2.2 KafkaProducer初始化
同所有需要配置的框架一样，KafkaProducer初始化核心原理就是读取我们配置的数据，根据配置的数据实例出KafkaProducer对象（包括其内部属性和其他需要的对象）。
大致流程图如下：
源码如下：
//KafkaProducer的构造方法
KafkaProducer(ProducerConfig config,
Serializer<K> keySerializer,
Serializer<V> valueSerializer,
Metadata metadata,
KafkaClient kafkaClient) {
try {
//1.获取用户自定义参数
Map<String, Object> userProvidedConfigs = config.originals();
this.producerConfig = config;
this.time = Time.SYSTEM;
//2.获取客户端id
String clientId = config.getString(ProducerConfig.CLIENT_ID_CONFIG);
if (clientId.length() <= 0)
clientId = "producer-" + PRODUCER_CLIENT_ID_SEQUENCE.getAndIncrement();
this.clientId = clientId;
//3.获取事务id
String transactionalId = userProvidedConfigs.containsKey(ProducerConfig.TRANSACTIONAL_ID_CONFIG) ?
(String) userProvidedConfigs.get(ProducerConfig.TRANSACTIONAL_ID_CONFIG) : null;
LogContext logContext;
if (transactionalId == null)
logContext = new LogContext(String.format("[Producer clientId=%s] ", clientId));
else
logContext = new LogContext(String.format("[Producer clientId=%s, transactionalId=%s] ", clientId, transactionalId));
log = logContext.logger(KafkaProducer.class);
log.trace("Starting the Kafka producer");
Map<String, String> metricTags = Collections.singletonMap("client-id", clientId);
MetricConfig metricConfig = new MetricConfig().samples(config.getInt(ProducerConfig.METRICS_NUM_SAMPLES_CONFIG))
.timeWindow(config.getLong(ProducerConfig.METRICS_SAMPLE_WINDOW_MS_CONFIG), TimeUnit.MILLISECONDS)
.recordLevel(Sensor.RecordingLevel.forName(config.getString(ProducerConfig.METRICS_RECORDING_LEVEL_CONFIG)))
.tags(metricTags);
List<MetricsReporter> reporters = config.getConfiguredInstances(ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG,
MetricsReporter.class);
//4.监控相关的设置
reporters.add(new JmxReporter(JMX_PREFIX));
this.metrics = new Metrics(metricConfig, reporters, time);
ProducerMetrics metricsRegistry = new ProducerMetrics(this.metrics);
//5.分区器相关设置
this.partitioner = config.getConfiguredInstance(ProducerConfig.PARTITIONER_CLASS_CONFIG, Partitioner.class);
//6.重试时间间隔参数设置，默认为100ms
long retryBackoffMs = config.getLong(ProducerConfig.RETRY_BACKOFF_MS_CONFIG);
//7.key和value序列化器设置
if (keySerializer == null) {
this.keySerializer = ensureExtended(config.getConfiguredInstance(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
Serializer.class));
this.keySerializer.configure(config.originals(), true);
} else {
config.ignore(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG);
this.keySerializer = ensureExtended(keySerializer);
}
if (valueSerializer == null) {
this.valueSerializer = ensureExtended(config.getConfiguredInstance(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
Serializer.class));
this.valueSerializer.configure(config.originals(), false);
} else {
config.ignore(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG);
this.valueSerializer = ensureExtended(valueSerializer);
}
// load interceptors and make sure they get clientId
userProvidedConfigs.put(ProducerConfig.CLIENT_ID_CONFIG, clientId);
//8.拦截器设置
List<ProducerInterceptor<K, V>> interceptorList = (List) (new ProducerConfig(userProvidedConfigs, false)).getConfiguredInstances(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG,
ProducerInterceptor.class);
this.interceptors = new ProducerInterceptors<>(interceptorList);
ClusterResourceListeners clusterResourceListeners = configureClusterResourceListeners(keySerializer, valueSerializer, interceptorList, reporters);
//9.生产者发往kafka集群单条信息的最大值，默认为1m
this.maxRequestSize = config.getInt(ProducerConfig.MAX_REQUEST_SIZE_CONFIG);
//10.缓存大小，默认为32m
this.totalMemorySize = config.getLong(ProducerConfig.BUFFER_MEMORY_CONFIG);
//11.压缩配置，默认为None
this.compressionType = CompressionType.forName(config.getString(ProducerConfig.COMPRESSION_TYPE_CONFIG));
this.maxBlockTimeMs = config.getLong(ProducerConfig.MAX_BLOCK_MS_CONFIG);
this.requestTimeoutMs = config.getInt(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG);
this.transactionManager = configureTransactionState(config, logContext, log);
int retries = configureRetries(config, transactionManager != null, log);
int maxInflightRequests = configureInflightRequests(config, transactionManager != null);
short acks = configureAcks(config, transactionManager != null, log);
this.apiVersions = new ApiVersions();
//12.消息累加器设置
//参数：上下文环境；批次大下，默认 16k；是否压缩，默认 none；linger.ms，默认值 0。
// 重试间隔时间，默认值 100ms；delivery.timeout.ms 默认值 2 分钟。；request.timeout.ms 默认值 30s。
this.accumulator = new RecordAccumulator(logContext,
config.getInt(ProducerConfig.BATCH_SIZE_CONFIG),
this.totalMemorySize,
this.compressionType,
config.getLong(ProducerConfig.LINGER_MS_CONFIG),
retryBackoffMs,
metrics,
time,
apiVersions,
transactionManager);
//13.kafka集群地址
List<InetSocketAddress> addresses = ClientUtils.parseAndValidateAddresses(config.getList(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG));
//14.从kafka集群获取元数据
if (metadata != null) {
this.metadata = metadata;
} else {
//metadata.max.age.ms 默认值 5 分钟。生产者每隔多久需要更新一下自己的元数据
this.metadata = new Metadata(retryBackoffMs, config.getLong(ProducerConfig.METADATA_MAX_AGE_CONFIG),
true, true, clusterResourceListeners);
this.metadata.update(Cluster.bootstrap(addresses), Collections.<String>emptySet(), time.milliseconds());
}
ChannelBuilder channelBuilder = ClientUtils.createChannelBuilder(config);
Sensor throttleTimeSensor = Sender.throttleTimeSensor(metricsRegistry.senderMetrics);
KafkaClient client = kafkaClient != null ? kafkaClient : new NetworkClient(
new Selector(config.getLong(ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG),
this.metrics, time, "producer", channelBuilder, logContext),
this.metadata,
clientId,
maxInflightRequests,
config.getLong(ProducerConfig.RECONNECT_BACKOFF_MS_CONFIG),
config.getLong(ProducerConfig.RECONNECT_BACKOFF_MAX_MS_CONFIG),
config.getInt(ProducerConfig.SEND_BUFFER_CONFIG),
config.getInt(ProducerConfig.RECEIVE_BUFFER_CONFIG),
this.requestTimeoutMs,
time,
true,
apiVersions,
throttleTimeSensor,
logContext);
//15.初始化sender线程
this.sender = new Sender(logContext,
client,
this.metadata,
this.accumulator,
maxInflightRequests == 1,
config.getInt(ProducerConfig.MAX_REQUEST_SIZE_CONFIG),
acks,
retries,
metricsRegistry.senderMetrics,
Time.SYSTEM,
this.requestTimeoutMs,
config.getLong(ProducerConfig.RETRY_BACKOFF_MS_CONFIG),
this.transactionManager,
apiVersions);
String ioThreadName = NETWORK_THREAD_PREFIX + " | " + clientId;
//启动发送线程
this.ioThread = new KafkaThread(ioThreadName, this.sender, true);
this.ioThread.start();
this.errors = this.metrics.sensor("errors");
config.logUnused();
AppInfoParser.registerAppInfo(JMX_PREFIX, clientId, metrics);
log.debug("Kafka producer started");
} catch (Throwable t) {
// call close methods if internal objects are already constructed this is to prevent resource leak. see KAFKA-2121
close(0, TimeUnit.MILLISECONDS, true);
// now propagate the exception
throw new KafkaException("Failed to construct kafka producer", t);
}
}
3.2.3 sender线程初始化
sender线程初始化，主要是为sender线程设置其需要的参数，例如网络通信客户端KafkaClient，ack应答机制等。
public Sender(LogContext logContext,
KafkaClient client,
Metadata metadata,
RecordAccumulator accumulator,
boolean guaranteeMessageOrder,
int maxRequestSize,
short acks,
int retries,
SenderMetricsRegistry metricsRegistry,
Time time,
int requestTimeoutMs,
long retryBackoffMs,
TransactionManager transactionManager,
ApiVersions apiVersions) {
this.log = logContext.logger(Sender.class);
//网络请求客户端NetworkClient
this.client = client;
this.accumulator = accumulator;
this.metadata = metadata;
this.guaranteeMessageOrder = guaranteeMessageOrder;
this.maxRequestSize = maxRequestSize;
//循环拉取数据标志位
this.running = true;
//ack应答
this.acks = acks;
this.retries = retries;
this.time = time;
this.sensors = new SenderMetrics(metricsRegistry);
this.requestTimeoutMs = requestTimeoutMs;
this.retryBackoffMs = retryBackoffMs;
this.apiVersions = apiVersions;
this.transactionManager = transactionManager;
}
Sender实现了runnable接口，其run()方法作用是不断循环从缓冲区中拉取数据（刚启动拉不到数据）。
while (running) {
try {
run(time.milliseconds());
} catch (Exception e) {
log.error("Uncaught error in kafka producer I/O thread: ", e);
}
}
void run(long now) {
if (transactionManager != null) {
......
}
//sender线程发送数据
long pollTimeout = sendProducerData(now);
//拉取服务端响应
client.poll(pollTimeout, now);
}
3.2.4 发送数据到缓冲区（main线程）
KafkaProducerd的send()方法大致流程如下图：
源码如下：
public Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback) {
//调用每个拦截器的onSend()方法
ProducerRecord<K, V> interceptedRecord = this.interceptors.onSend(record);
//发送消息逻辑
return doSend(interceptedRecord, callback);
}
private Future<RecordMetadata> doSend(ProducerRecord<K, V> record, Callback callback) {
TopicPartition tp = null;
try {
throwIfProducerClosed();
// first make sure the metadata for the topic is available
ClusterAndWaitTime clusterAndWaitTime;
try {
//从kafka拉取元数据。maxBlockTimeMs表示最多能等待多长时间
clusterAndWaitTime = waitOnMetadata(record.topic(), record.partition(), maxBlockTimeMs);
} catch (KafkaException e) {
if (metadata.isClosed())
throw new KafkaException("Producer closed while send in progress", e);
throw e;
}
//剩余时间 = 最多能等待时间 - 用了多长时间
long remainingWaitMs = Math.max(0, maxBlockTimeMs - clusterAndWaitTime.waitedOnMetadataMs);
// 更新集群元数据
Cluster cluster = clusterAndWaitTime.cluster;
// 序列化数据为byte数组
byte[] serializedKey;
try {
serializedKey = keySerializer.serialize(record.topic(), record.headers(), record.key());
} catch (ClassCastException cce) {
throw new SerializationException("Can't convert key of class " + record.key().getClass().getName() +
" to class " + producerConfig.getClass(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG).getName() +
" specified in key.serializer", cce);
}
byte[] serializedValue;
try {
serializedValue = valueSerializer.serialize(record.topic(), record.headers(), record.value());
} catch (ClassCastException cce) {
throw new SerializationException("Can't convert value of class " + record.value().getClass().getName() +
" to class " + producerConfig.getClass(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG).getName() +
" specified in value.serializer", cce);
}
//分区操作(计算消息发往的分区)
int partition = partition(record, serializedKey, serializedValue, cluster);
tp = new TopicPartition(record.topic(), partition);
setReadOnly(record.headers());
Header[] headers = record.headers().toArray();
int serializedSize = AbstractRecords.estimateSizeInBytesUpperBound(apiVersions.maxUsableProduceMagic(),
compressionType, serializedKey, serializedValue, headers);
//检验发送消息的大小是否超过最大值，默认为1m
ensureValidRecordSize(serializedSize);
long timestamp = record.timestamp() == null ? time.milliseconds() : record.timestamp();
log.trace("Sending record {} with callback {} to topic {} partition {}", record, callback, record.topic(), partition);
//消息发送的回调函数
Callback interceptCallback = new InterceptorCallback<>(callback, this.interceptors, tp);
if (transactionManager != null && transactionManager.isTransactional())
transactionManager.maybeAddPartitionToTransaction(tp);
//批次(内存)，默认32m，里面默认16k为一个批次
RecordAccumulator.RecordAppendResult result = accumulator.append(tp, timestamp, serializedKey,
serializedValue, headers, interceptCallback, remainingWaitMs);
//如果批次满了 或者 创建了一个新的批次，则唤醒sender发送线程
if (result.batchIsFull || result.newBatchCreated) {
log.trace("Waking up the sender since topic {} partition {} is either full or getting a new batch", record.topic(), partition);
this.sender.wakeup();
}
return result.future;
} catch (Exception e) {
...........
}
}
3.2.5 sender线程的run方法
sender的run方法一直循环运行，只做两件事：将准备好的数据发往服务端和等待服务端的响应。
其大致流程如图所示：
源码如下：
private long sendProducerData(long now) {
//获取元数据
Cluster cluster = metadata.fetch();
//检查32m缓存是否准备好
RecordAccumulator.ReadyCheckResult result = this.accumulator.ready(cluster, now);
//如果Leader信息未知，则不能发送数据
if (!result.unknownLeaderTopics.isEmpty()) {
for (String topic : result.unknownLeaderTopics)
this.metadata.add(topic);
log.debug("Requesting metadata update due to unknown leader topics from the batched records: {}", result.unknownLeaderTopics);
this.metadata.requestUpdate();
}
//删除没有准备好发送的数据
Iterator<Node> iter = result.readyNodes.iterator();
long notReadyTimeout = Long.MAX_VALUE;
while (iter.hasNext()) {
Node node = iter.next();
if (!this.client.ready(node, now)) {
iter.remove();
notReadyTimeout = Math.min(notReadyTimeout, this.client.pollDelayMs(node, now));
}
}
//发往同一个broker节点的数据，打包为一个批次
Map<Integer, List<ProducerBatch>> batches = this.accumulator.drain(cluster, result.readyNodes,
this.maxRequestSize, now);
if (guaranteeMessageOrder) {
for (List<ProducerBatch> batchList : batches.values()) {
for (ProducerBatch batch : batchList)
this.accumulator.mutePartition(batch.topicPartition);
}
}
List<ProducerBatch> expiredBatches = this.accumulator.expiredBatches(this.requestTimeoutMs, now);
if (!expiredBatches.isEmpty())
log.trace("Expired {} batches in accumulator", expiredBatches.size());
for (ProducerBatch expiredBatch : expiredBatches) {
failBatch(expiredBatch, -1, NO_TIMESTAMP, expiredBatch.timeoutException(), false);
if (transactionManager != null && expiredBatch.inRetry()) {
transactionManager.markSequenceUnresolved(expiredBatch.topicPartition);
}
}
sensors.updateProduceRequestMetrics(batches);
long pollTimeout = Math.min(result.nextReadyCheckDelayMs, notReadyTimeout);
if (!result.readyNodes.isEmpty()) {
log.trace("Nodes with data ready to send: {}", result.readyNodes);
pollTimeout = 0;
}
//以request方式发送数据
sendProduceRequests(batches, now);
return pollTimeout;
}
private void sendProduceRequest(long now, int destination, short acks, int timeout, List<ProducerBatch> batches) {
//省略部分代码.....
String nodeId = Integer.toString(destination);
//创建发送请求对象
ClientRequest clientRequest = client.newClientRequest(nodeId, requestBuilder, now, acks != 0,
requestTimeoutMs, callback);
//发送请求
client.send(clientRequest, now);
log.trace("Sent produce request to {}: {}", nodeId, requestBuilder);
}
private void doSend(ClientRequest clientRequest, boolean isInternalRequest, long now, AbstractRequest request) {
//省略部分代码......
Send send = request.toSend(destination, header);
InFlightRequest inFlightRequest = new InFlightRequest(
clientRequest,
header,
isInternalRequest,
request,
send,
now);
//添加请求到inFlightRequest
this.inFlightRequests.add(inFlightRequest);
//发送数据
selector.send(send);
}
3.3 生产者常用参数
- bootstrap.servers：生产者客户端连接Kafka集群所需的broker地址清单
- key.serializer 和 value.serializer：消息key和value的序列化器
- client.id：客户端id，默认为""，如果客户端不设置，则KafkaProducer会自动生成
- acks：用来指定分区中必须要有多少个副本收到这条消息，之后生产者才会认为这条消息是成功写入的
eg:
ack=1：只要分区的leader副本成功写入消息，便认为消息写入成功。是消息可靠性和吞吐量之间的折中方案
ack=0：无需等待任何服务端的响应。性能最高、可靠性最差。
acks=-1或acks=all：等待ISR中的所有副本都成功才认为消息写入成功。可靠性最强 - max.request.size：生产者客户端能发送的消息的最大值，默认为1M
- retries：消息发送失败时，生产者重试的次数。默认为0
- retry.backoff.ms：生产者两次重试之间的时间间隔
- compression.type：消息的压缩方式。默认值为“none”，还可以配置为“gzip”“snappy”和“lz4“
- connections.max.idle.ms：指定在多久之后关闭限制的连接，默认值为54000ms（即9分钟）
- batch.size：在分区累加器中，只有数据累计达到batch.size之后，sender线程才会发送数据，默认为16k
- linger.ms：在分区累加器中，如果数据迟迟未达到batch.size，sender线程会在等待linger.ms时间后发送数据，单位为ms。默认值为0。
增大这个参数的值会增加消息的延迟，但是同时能提升一定的吞吐量；设置为0可以避免消息饥饿。 - request.timeout.ms：Producer等待请求响应的最长时，请求超时之后选择进行重试。
朱忠华：深入理解kafka：核心设计与实现原理
尚硅谷kafka教学文档
博主微信，欢迎交流：jhy2496085873
扫一扫
54
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
