二，offset操作
2.1 提交offset
在消费者消费过程，以及rebalance操作之前，抖要提交一次offset记录consumer当前的消费位置。提交offset的功能也由ConsumerCoordinator实现
我们在第一节中讲过，Consumer..Client的SubscriptionState字段，使用TopicPartitionState记录每个TopicPartition的消费情况，它的position字段则记录了消费者下次要从服务器获取消息的offset。他们形成了ConsumerCoordinator.commitOffset*()方法的第一个参数。
private RequestFuture<Void> sendOffsetCommitRequest(final Map<TopicPartition, OffsetAndMetadata> offsets) {
if (offsets.isEmpty())
return RequestFuture.voidSuccess();
Node coordinator = coordinator();
if (coordinator == null)
return RequestFuture.coordinatorNotAvailable();
// create the offset commit request 创建request中的data
Map<TopicPartition, OffsetCommitRequest.PartitionData> offsetData = new HashMap<>(offsets.size());
for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
OffsetAndMetadata offsetAndMetadata = entry.getValue();
if (offsetAndMetadata.offset() < 0) {
return RequestFuture.failure(new IllegalArgumentException("Invalid offset: " + offsetAndMetadata.offset()));
}
offsetData.put(entry.getKey(), new OffsetCommitRequest.PartitionData(
offsetAndMetadata.offset(), offsetAndMetadata.metadata()));
}
//设置 年代
final Generation generation;
if (subscriptions.partitionsAutoAssigned())
generation = generation();
else
generation = Generation.NO_GENERATION;
if (generation == null)
return RequestFuture.failure(new CommitFailedException());
//request 创建
OffsetCommitRequest.Builder builder = new OffsetCommitRequest.Builder(this.groupId, offsetData).
setGenerationId(generation.generationId).
setMemberId(generation.memberId).
setRetentionTime(OffsetCommitRequest.DEFAULT_RETENTION_TIME);
log.trace("Sending OffsetCommit request with {} to coordinator {}", offsets, coordinator);
// 发送
return client.send(coordinator, builder)
.compose(new OffsetCommitResponseHandler(offsets)); //【入】 处理handler
}handler 处理响应
for (Map.Entry<TopicPartition, Errors> entry : commitResponse.responseData().entrySet()) {
TopicPartition tp = entry.getKey();
OffsetAndMetadata offsetAndMetadata = this.offsets.get(tp);
long offset = offsetAndMetadata.offset();
Errors error = entry.getValue();
if (error == Errors.NONE) {
log.debug("Committed offset {} for partition {}", offset, tp);
if (subscriptions.isAssigned(tp))
// update the local cache only if the partition is still assigned
subscriptions.committed(tp, offsetAndMetadata);//给SubscriptionState 重新赋值
}2.2 fetch offset
在rebalance操作结束之后，我们要开始消费消息，但是这个时候consumer不知道对应的分区已经被消费到哪个位置了，所以需要在消费之前，拉取存放于kafka内部的offsets topic中的offset。
所以我们要发送OffsetFetchRequest请求
public void refreshCommittedOffsetsIfNeeded() {
if (subscriptions.refreshCommitsNeeded()) {
Map<TopicPartition, OffsetAndMetadata> offsets = fetchCommittedOffsets(subscriptions.assignedPartitions()); //【入】
for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
TopicPartition tp = entry.getKey();
// verify assignment is still active
if (subscriptions.isAssigned(tp))
this.subscriptions.committed(tp, entry.getValue()); //修改offset值
}
this.subscriptions.commitsRefreshed();
}
}public Map<TopicPartition, OffsetAndMetadata> fetchCommittedOffsets(Set<TopicPartition> partitions) {
while (true) {
ensureCoordinatorReady();
// contact coordinator to fetch committed offsets
RequestFuture<Map<TopicPartition, OffsetAndMetadata>> future = sendOffsetFetchRequest(partitions); //【入】
client.poll(future);
if (future.succeeded())//返回从服务端 获取的offset
return future.value();
if (!future.isRetriable())
throw future.exception();
time.sleep(retryBackoffMs);
}
}创建request,参数：groupId,partition list
private RequestFuture<Map<TopicPartition, OffsetAndMetadata>> sendOffsetFetchRequest(Set<TopicPartition> partitions) {
Node coordinator = coordinator();
if (coordinator == null)
return RequestFuture.coordinatorNotAvailable();
// construct the request
OffsetFetchRequest.Builder requestBuilder = new OffsetFetchRequest.Builder(this.groupId,
new ArrayList<>(partitions));
// send the request with a callback
return client.send(coordinator, requestBuilder)
.compose(new OffsetFetchResponseHandler());
}handler处理response
Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>(response.responseData().size());//记录从服务端获取的offset集合
for (Map.Entry<TopicPartition, OffsetFetchResponse.PartitionData> entry : response.responseData().entrySet()) {
TopicPartition tp = entry.getKey();
OffsetFetchResponse.PartitionData data = entry.getValue();
if (data.hasError()) {
...
} else if (data.offset >= 0) { // 记录正常的offset
// record the position with the offset (-1 indicates no committed offset to fetch)
offsets.put(tp, new OffsetAndMetadata(data.offset, data.metadata));
} else {
log.debug("Found no committed offset for partition {}", tp);
}
}
//传播offsets 集合，最终通过fetchCommitedOffsets()方法返回
future.complete(offsets);三，fetcher
到了这里，我们一切准备就绪，消费者可以从server拉取消息进行消费了，consumer使用fetcher来拉取消息，通过FetchRequest请求！
字段详解：
private final ConsumerNetworkClient client; //负责网络通信
private final Time time;
private final int minBytes; //从服务端收到request之后，不是立即响应，而是当可返回的消息数据至少累积到minBytes字节的时候才响应，（server延时任务）
private final int maxBytes;
private final int maxWaitMs; //等待response最长时间，server根据这个决定何时响应
private final int fetchSize; //每次fetch操作的最大字节数
private final long retryBackoffMs;
private final int maxPollRecords; //每次获取record的最大数量
private final boolean checkCrcs;
private final Metadata metadata;
private final FetchManagerMetrics sensors;
private final SubscriptionState subscriptions; //记录每个partition的消费情况
private final ConcurrentLinkedQueue<CompletedFetch> completedFetches; //每个response首先转成fetch对象进入缓冲队列，等待解析
private final BufferSupplier decompressionBufferSupplier = BufferSupplier.create();
//两个 反序列化 器
private final ExtendedDeserializer<K> keyDeserializer;
private final ExtendedDeserializer<V> valueDeserializer;
private final IsolationLevel isolationLevel;
//保存了CompleteFetch解析后的结果集合，(内部类)
private PartitionRecords nextInLineRecords = null;Fetch类的核心方法分为三类：
1.fetch消息的，获取消息
2.更新offset，更新TopicPartitionState中的position字段
3.获取metadata信息的方法，用于指定topic的元信息
3.1 创建FetchRequest请求
1.按条件查找fetchable分区
2.查找fatchable分区的leader所在的node
3.找到node节点，如果Node已经在unsent集合中，则不发送FetchRequest请求。
4.通过SubscriptionState查找每个分区对应的position，
5.最后，按照Node分类，将发往同一个node节点的topicPartition封城成FetchRequest对象
我们先看createFetchRequest()的代码，如上面叙述的步骤一样，先遍历fetchable，然后封装成FetchRequest
private Map<Node, FetchRequest.Builder> createFetchRequests() {
// create the fetch info
Cluster cluster = metadata.fetch(); //获取kafka集群元数据
Map<Node, LinkedHashMap<TopicPartition, FetchRequest.PartitionData>> fetchable = new LinkedHashMap<>();
// 根据条件 筛选tp,放入fetchAble
for (TopicPartition partition : fetchablePartitions()) {
Node node = cluster.leaderFor(partition); //获得leader副本所在node
if (node == null) {
metadata.requestUpdate();
} else if (!this.client.hasPendingRequests(node)) { //是否还有pending请求
// if there is a leader and no in-flight requests, issue a new fetch
LinkedHashMap<TopicPartition, FetchRequest.PartitionData> fetch = fetchable.get(node);
if (fetch == null) { //fetch初始化
fetch = new LinkedHashMap<>();
fetchable.put(node, fetch);
}
//记录每个分区对应的position
long position = this.subscriptions.position(partition);
fetch.put(partition, new FetchRequest.PartitionData(position, FetchRequest.INVALID_LOG_START_OFFSET,
this.fetchSize));
log.debug("Added {} fetch request for partition {} at offset {} to node {}", isolationLevel,
partition, position, node);
} else {
log.trace("Skipping fetch for partition {} because there is an in-flight request to {}", partition, node);
}
}
// create the fetches
Map<Node, FetchRequest.Builder> requests = new HashMap<>();
for (Map.Entry<Node, LinkedHashMap<TopicPartition, FetchRequest.PartitionData>> entry : fetchable.entrySet()) {
Node node = entry.getKey();
FetchRequest.Builder fetch = FetchRequest.Builder.forConsumer(this.maxWaitMs, this.minBytes,
entry.getValue(), isolationLevel)
.setMaxBytes(this.maxBytes);
requests.put(node, fetch); //整合
}
return requests;
}之后我们再来看发送FetchRequest代码，还是老模式，放入unsent中缓冲起来，而与之前XXRequest发送不同的是，这里的send()它没有绑定handler对response进行处理，而是直接使用listener对response进行处理，应该是消费消息提高效率
,对于response的处理，就是把获取到的数据组装成为completedFetch放到列表中，等待解析。
public int sendFetches() {
Map<Node, FetchRequest.Builder> fetchRequestMap = createFetchRequests(); //【入】返回request
for (Map.Entry<Node, FetchRequest.Builder> fetchEntry : fetchRequestMap.entrySet()) {
final FetchRequest.Builder request = fetchEntry.getValue();
final Node fetchTarget = fetchEntry.getKey();
//发送+ 监听 将发往每个node的request都缓冲到unsent中。
client.send(fetchTarget, request)
.addListener(new RequestFutureListener<ClientResponse>() {
@Override //【★】
public void onSuccess(ClientResponse resp) {
//response强转为fetchResponse
FetchResponse response = (FetchResponse) resp.responseBody();
if (!matchesRequestedPartitions(request, response)) {
return;
}
Set<TopicPartition> partitions = new HashSet<>(response.responseData().keySet());
FetchResponseMetricAggregator metricAggregator = new FetchResponseMetricAggregator(sensors, partitions);
//遍历响应中的数据
for (Map.Entry<TopicPartition, FetchResponse.PartitionData> entry : response.responseData().entrySet()) {
TopicPartition partition = entry.getKey();
long fetchOffset = request.fetchData().get(partition).fetchOffset;
FetchResponse.PartitionData fetchData = entry.getValue();
//创建completeFetch，并缓冲到xxfetches队列中
completedFetches.add(new CompletedFetch(partition, fetchOffset, fetchData, metricAggregator,
resp.requestHeader().apiVersion()));
}
sensors.fetchLatency.record(resp.requestLatencyMs());
}
});
}
return fetchRequestMap.size();
}主要内容在处理response上面，对于创建request，我们在第一步中已经看到了，之后的重点，就转移到了如何解析CompleteFetches中的response封装了！在这里稍显一点复杂，请提高注意力！
我们的解析是放在fetchedRecords()方法中，同样位于Fetcher类中，同时还存在fetchRecords()方法，主要两者的区别，不要搞混淆！
public Map<TopicPartition, List<ConsumerRecord<K, V>>> fetchedRecords() {
Map<TopicPartition, List<ConsumerRecord<K, V>>> fetched = new HashMap<>();
int recordsRemaining = maxPollRecords; //一次最多取出x条消息
while (recordsRemaining > 0) { //如果解析结果为空，也就是第一次 消费消息
if (nextInLineRecords == null || nextInLineRecords.isFetched) {
CompletedFetch completedFetch = completedFetches.peek();
if (completedFetch == null) break;
nextInLineRecords = parseCompletedFetch(completedFetch); //解析并赋值
completedFetches.poll();
} else { //后续消费，要核对offset //【入】
List<ConsumerRecord<K, V>> records = fetchRecords(nextInLineRecords, recordsRemaining);
TopicPartition partition = nextInLineRecords.partition;
if (!records.isEmpty()) {
List<ConsumerRecord<K, V>> currentRecords = fetched.get(partition);
//将解析结果，放入容器中返回
if (currentRecords == null) {
fetched.put(partition, records);
} else {
List<ConsumerRecord<K, V>> newRecords = new ArrayList<>(records.size() + currentRecords.size());
newRecords.addAll(currentRecords);
newRecords.addAll(records);
fetched.put(partition, newRecords);
}
recordsRemaining -= records.size(); //这里才-1
}
}
}
return fetched;
}该方法里面什么参数都没有传入，而是获取了字段：maxPollRecords，后续的操作都与它有关，它表示的是每次获取record的最大数量，所以它也对应了一个response里面存放的最多record。这里主要是最多，而不是一定是这么多！
进入了循环：循环的条件就是maxPollRecords条件，该条件的减少出现在其内容体中的else{}段中，那么第一段if{}中出现的nextInlineRecords字段，它代表的是CompleteFetch解析出来的结果集合，类型是PartitionRecords，是Fetcher的内部类，我们看一下主要字段：
private class PartitionRecords {
private final TopicPartition partition; //对应的partion
private final CompletedFetch completedFetch; //对应的源数据
private final Iterator<? extends RecordBatch> batches;
private final Set<Long> abortedProducerIds;
private final PriorityQueue<FetchResponse.AbortedTransaction> abortedTransactions;
private int recordsRead;
private int bytesRead;
private RecordBatch currentBatch; //出现了batch,联想到producer端的batch
private Record lastRecord;
private CloseableIterator<Record> records; //消息集合
private long nextFetchOffset;
private boolean isFetched = false;
private Exception cachedRecordException = null;
private boolean corruptLastRecord = false;
...
}里面有一个CompletedFetch，它的字段很全面，主要的是CloseableIterator<Record> records。
可以判定，在第一次拉取消息，nextInLineRecords为null,那么就需要进入if{}段中，使用peek()拿到CompletedFetch，对其进行解析，我们看解析过程：parseCompletedFetch()
private PartitionRecords parseCompletedFetch(CompletedFetch completedFetch) {
//解析 fetch中的 信息，取出来
TopicPartition tp = completedFetch.partition;
FetchResponse.PartitionData partition = completedFetch.partitionData;
long fetchOffset = completedFetch.fetchedOffset;
//待返回值，先准备好
PartitionRecords partitionRecords = null;
Errors error = partition.error;
try {
if (!subscriptions.isFetchable(tp)) {
} else if (error == Errors.NONE) { //【★】 内容不多，就是一大堆检测
Long position = subscriptions.position(tp);
Iterator<? extends RecordBatch> batches = partition.records.batches().iterator();
partitionRecords = new PartitionRecords(tp, completedFetch, batches); //结果 组装
if (!batches.hasNext() && partition.records.sizeInBytes() > 0) {
if (completedFetch.responseVersion < 3) {
Map<TopicPartition, Long> recordTooLargePartitions = Collections.singletonMap(tp, fetchOffset);
} else {
}
}
....
return partitionRecords;
}注意这里的内容，并没有进行反序列化，
我们继续回到while()循环中，这一次，我们走到else{}代码块中：第一行就根据if()中获得的值+maxPollRecord调用了fetchRecords()方法
private List<ConsumerRecord<K, V>> fetchRecords(PartitionRecords partitionRecords, int maxRecords) {
if (!subscriptions.isAssigned(partitionRecords.partition)) {
//需要进行rebalance操作，则返回 空集合
} else {
long position = subscriptions.position(partitionRecords.partition);
if (!subscriptions.isFetchable(partitionRecords.partition)) { //核对partition
} else if (partitionRecords.nextFetchOffset == position) { //核对 offset
List<ConsumerRecord<K, V>> partRecords = partitionRecords.fetchRecords(maxRecords); //【入】
long nextOffset = partitionRecords.nextFetchOffset;
subscriptions.position(partitionRecords.partition, nextOffset);
Long partitionLag = subscriptions.partitionLag(partitionRecords.partition, isolationLevel);
if (partitionLag != null)
this.sensors.recordPartitionLag(partitionRecords.partition, partitionLag);
return partRecords;
}
}
partitionRecords.drain(); //【入】
return emptyList();
}可以看到，重点代码就两个，其他全部都是检测字段是否合法。我们进入partitionRecords.fetchRecords()内部类的方法中：
private List<ConsumerRecord<K, V>> fetchRecords(int maxRecords) {
List<ConsumerRecord<K, V>> records = new ArrayList<>();
try {
//遍历 从0开始，i拿来没用
for (int i = 0; i < maxRecords; i++) {
if (cachedRecordException == null) {
corruptLastRecord = true;
lastRecord = nextFetchedRecord(); //★
corruptLastRecord = false;
}
if (lastRecord == null)
break;
//【入】 解析
records.add(parseRecord(partition, currentBatch, lastRecord));
recordsRead++;
bytesRead += lastRecord.sizeInBytes();
nextFetchOffset = lastRecord.offset() + 1;
cachedRecordException = null;
}
}
return records;
}同样如此，重点方法出现在了add()的参数中，调用了parseRecord()方法，重点是传入的lastRecord值，我们在第一段if()里面生成PartitionRecord实例的时候，其字段records 是空的，它的生成就是在nextFetchedRecord()中！具体代码我们这里不查看了，使用的是迭代器，有点看不懂！
我们直接来到parseRecord()解析record!
private ConsumerRecord<K, V> parseRecord(TopicPartition partition,
RecordBatch batch,
Record record) {
try {
long offset = record.offset();
long timestamp = record.timestamp();
TimestampType timestampType = batch.timestampType();
Headers headers = new RecordHeaders(record.headers());
ByteBuffer keyBytes = record.key();
//反序列化
byte[] keyByteArray = keyBytes == null ? null : Utils.toArray(keyBytes);
K key = keyBytes == null ? null : this.keyDeserializer.deserialize(partition.topic(), headers, keyByteArray);
ByteBuffer valueBytes = record.value();
byte[] valueByteArray = valueBytes == null ? null : Utils.toArray(valueBytes);
V value = valueBytes == null ? null : this.valueDeserializer.deserialize(partition.topic(), headers, valueByteArray);
//返回值的 创建，看似比较复杂，有两个是三元运算
return new ConsumerRecord<>(partition.topic(), partition.partition(), offset,
timestamp, timestampType, record.checksumOrNull(),
keyByteArray == null ? ConsumerRecord.NULL_SIZE : keyByteArray.length,
valueByteArray == null ? ConsumerRecord.NULL_SIZE : valueByteArray.length,
key, value, headers);
}
}解析结果：ConsumerRecord<K,V>出场了！，。。。。。。。回到fetchedRecords()中，我们现在取到了值：
再更具topic,把结果放入容器中，返回，到这里，我们的消息就成功的解析出来了！！！
扫一扫
8970
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
