继续接着上一下消费者的知识点,本节我们将追踪消费者中最重要的rebalance代码和offset提交代码!
一,rebalance
首先,我们看看哪些情况会触发rebalance:
1.有新的consumer 加入
2.有consumer宕机,可能网络延迟,规定时间内没有进行heartbean
3.有消费者主动退出 consumer gorup
4.conmuser group订阅的topic出现 partition数量变化
5.消费者调用unsubscrible() 取消对topic的订阅
我们把rebalance整体分为三个阶段,分别寻找GroupCoordinator,当一个consumer上线的时候,不知道我们订阅的topic,在哪个node上面。第二阶段:进行join Group,发送JoinGroupRequest请求,处理response。第三阶段:进入Synchronizing Group State阶段,
1.1 第一阶段
找到GroupCoordinator,我们随便向集群中的任意node,发送一个GroupCoordinatorRequest请求,在该请求里面就只带了groupId这一个变量,集群收到之后,返会GroupCoordinatorResponse响应,包含了coordinator节点的id,host,port三个值,还有错误码(便于查找问题,short类型)
注意poll()是整个rebalance流程都会走的主方法,它位于ConsumerCoordinator类中
public void poll(long now, long remainingMs) {
invokeCompletedOffsetCommitCallbacks();
if (subscriptions.partitionsAutoAssigned()) {
if (coordinatorUnknown()) {
ensureCoordinatorReady();//第一阶段
now = time.milliseconds();
}
....
}
我们首先看一下if中的判断方法,如果coordinator不存在,那么我们就需要通过GroupCoodinatorRequest请求找到coordinator。
public boolean coordinatorUnknown() {
return coordinator() == null;
}
现在步入正题,发送request,处理response,这里的方法,只是发送request的入口,真正的send()还需要深入
它位于AbstractCoordinator类中,也就是ConsumerCoordinator的父类
----- request创建与发送 -----------
protected synchronized boolean ensureCoordinatorReady(long startTimeMs, long timeoutMs) {
long remainingMs = timeoutMs;
while (coordinatorUnknown()) {
RequestFuture<Void> future = lookupCoordinator(); //发送【入】
client.poll(future, remainingMs); //等待结果
if (future.failed()) {
if (future.isRetriable()) {
remainingMs = timeoutMs - (time.milliseconds() - startTimeMs);
if (remainingMs <= 0)
break;
client.awaitMetadataUpdate(remainingMs); //阻塞更新metadata中记录的集群元数据,重新执行
} else
throw future.exception();
} else if (coordinator != null && client.connectionFailed(coordinator)) {
//找到了,但是网络连接失败,清空unsent,重新发送
coordinatorDead();
time.sleep(retryBackoffMs);
}
remainingMs = timeoutMs - (time.milliseconds() - startTimeMs);
if (remainingMs <= 0)
break;
}
return !coordinatorUnknown();
}
我们要了解GroupDoordinatorRequest是如何产生的,所以我们进入lookupCoordinator()方法中:
protected synchronized RequestFuture<Void> lookupCoordinator() {
if (findCoordinatorFuture == null) {
// find a node to ask about the coordinator
Node node = this.client.leastLoadedNode(); //找一个负载最小的node 发送data
if (node == null) {
log.debug("No broker available to send GroupCoordinator request");
return RequestFuture.noBrokersAvailable();
} else
findCoordinatorFuture = sendGroupCoordinatorRequest(node);//【入】
}
return findCoordinatorFuture;
}
它的主要功能是,找一个负载最小的node节点,用来发送信息,然后再深入:
private RequestFuture<Void> sendGroupCoordinatorRequest(Node node) {
// initiate the group metadata request
log.debug("Sending GroupCoordinator request to broker {}", node);
FindCoordinatorRequest.Builder requestBuilder =
new FindCoordinatorRequest.Builder(FindCoordinatorRequest.CoordinatorType.GROUP, this.groupId);
return client.send(node, requestBuilder)
.compose(new GroupCoordinatorResponseHandler()); //发送
}
可以看到,我们找到了CoordinatorRequest的构建了,直接new出来的,参数也很简单,就是一个grooupId,这个Builder类是FindCoorDinatorRequest的内部类,这样设计的原因是什么?
public static class Builder extends AbstractRequest.Builder<FindCoordinatorRequest> {
private final String coordinatorKey; //groupId
private final CoordinatorType coordinatorType; //Group
private final short minVersion;
...
}
最后,使用client发送:注意这里的CompletionHandler,处理器,它也是内部类。
public RequestFuture<ClientResponse> send(Node node, AbstractRequest.Builder<?> requestBuilder) {
long now = time.milliseconds();
RequestFutureCompletionHandler completionHandler = new RequestFutureCompletionHandler();
ClientRequest clientRequest = client.newClientRequest(node.idString(), requestBuilder, now, true,
completionHandler);
unsent.put(node, clientRequest);
// wakeup the client in case it is blocking in poll so that we can send the queued request
client.wakeup();
return completionHandler.future;
}
--------- response 处理 ---------------
我们分析一下回调过程:
首先我们在send()里面返回的是一个future,这个future是什么?
是一个ClinetResponse,我们还有对应的ClientRequest。我们再看一下ClientResponse字段:
ClientResponse类
public class ClientResponse {
private final RequestHeader requestHeader;
private final RequestCompletionHandler callback; //★
private final String destination;
private final long receivedTimeMs;
private final long latencyMs;
private final boolean disconnected;
private final UnsupportedVersionException versionMismatch;
private final AbstractResponse responseBody;
...
}
注意★这个字段,我们回忆一下,调用send()方法的时候,后面还有一个尾巴:
这里就对应上了,所以我们查看这个responseHandler是如何处理的:
public void onSuccess(ClientResponse resp, RequestFuture<Void> future) {
...
if (error == Errors.NONE) { //无错误
synchronized (AbstractCoordinator.this) { //给coordinator赋值
AbstractCoordinator.this.coordinator = new Node(
Integer.MAX_VALUE - findCoordinatorResponse.node().id(),
findCoordinatorResponse.node().host(),
findCoordinatorResponse.node().port());
log.info("Discovered coordinator {}", coordinator);
client.tryConnect(coordinator);
heartbeat.resetTimeouts(time.milliseconds());
}
future.complete(null);
} else if (error == Errors.GROUP_AUTHORIZATION_FAILED) {
future.raise(new GroupAuthorizationException(groupId));
} else {
log.debug("Group coordinator lookup failed: {}", error.message());
future.raise(error);
}
}
好了,我们到这里就成功的把coordinator初始化了!
1.2 第二阶段
成功找到GroupCoordinator之后,进入第二阶段,消费者向GroupCoordinator发送JoinGroupRequest请求,并处理响应。
入口同样在ConsumerCoordinator类中的poll()方法中
public void poll(long now, long remainingMs) {
...
if (needRejoin()) {
if (subscriptions.hasPatternSubscription())
client.ensureFreshMetadata();
ensureActiveGroup(); //【入】第二阶段
now = time.milliseconds();
}
...
}
调用方法判断是否需要join()
public boolean needRejoin() {
if (!subscriptions.partitionsAutoAssigned()) //是否为自动分区分配订阅模式
return false;
// we need to rejoin if we performed the assignment and metadata has changed
if (assignmentSnapshot != null && !assignmentSnapshot.equals(metadataSnapshot))
return true;
// we need to join if our subscription has changed since the last join
if (joinedSubscription != null && !joinedSubscription.equals(subscriptions.subscription()))
return true;
return super.needRejoin();
}
好了,现在我们需要rejoin(),并且GroupCoordinator已经找到了,继续跟踪
来到AbstractCoordinator类中,ConsumerCoordinator的父类
public void ensureActiveGroup() {
ensureCoordinatorReady(); //保证 coordinator已经找到
startHeartbeatThreadIfNeeded(); //心跳检测 开启
joinGroupIfNeeded(); //是否需要join
}
看一下第二个方法,开启心跳检测,内容简单。
private synchronized void startHeartbeatThreadIfNeeded() {
if (heartbeatThread == null) {
heartbeatThread = new HeartbeatThread();
heartbeatThread.start();
}
}
现在进入正题:joinGroupIfNeeded()方法:
void joinGroupIfNeeded() {
while (needRejoin() || rejoinIncomplete()) {
ensureCoordinatorReady(); //确保 找到了groupCoordinator
if (needsJoinPrepare) {
onJoinPrepare(generation.generationId, generation.memberId); //1.预处理
needsJoinPrepare = false;
}
RequestFuture<ByteBuffer> future = initiateJoinGroup(); //2.【入】
client.poll(future); //阻塞等待完成请求
if (future.succeeded()) {
onJoinComplete(generation.generationId, generation.memberId, generation.protocol, future.value());
resetJoinGroupFuture();
needsJoinPrepare = true;
} else {
resetJoinGroupFuture();
RuntimeException exception = future.exception();
if (exception instanceof UnknownMemberIdException ||
exception instanceof RebalanceInProgressException ||
exception instanceof IllegalGenerationException)
continue;
else if (!future.isRetriable())
throw exception;
time.sleep(retryBackoffMs);
}
}
}
看第一个1.预处理 onJoinPrepare()方法,
protected void onJoinPrepare(int generation, String memberId) {
// commit offsets prior to rebalance if auto-commit enabled
maybeAutoCommitOffsetsSync(rebalanceTimeoutMs); //进行一次同步offset操作
// execute the user's callback before rebalance
ConsumerRebalanceListener listener = subscriptions.listener();
log.info("Revoking previously assigned partitions {}", subscriptions.assignedPartitions());
try {
Set<TopicPartition> revoked = new HashSet<>(subscriptions.assignedPartitions());
listener.onPartitionsRevoked(revoked);
} catch (WakeupException | InterruptException e) {
throw e;
} catch (Exception e) {
log.error("User provided listener {} failed on partition revocation", listener.getClass().getName(), e);
}
isLeader = false;
subscriptions.resetGroupSubscription();
}
第二个方法,----- request创建与发送 -----------
private synchronized RequestFuture<ByteBuffer> initiateJoinGroup() {
if (joinFuture == null) {
disableHeartbeatThread(); //中断 心跳检测
state = MemberState.REBALANCING;
joinFuture = sendJoinGroupRequest(); //【入】 发送
joinFuture.addListener(new RequestFutureListener<ByteBuffer>() {
@Override
public void onSuccess(ByteBuffer value) {
synchronized (AbstractCoordinator.this) {
log.info("Successfully joined group with generation {}", generation.generationId);
state = MemberState.STABLE;
rejoinNeeded = false;
if (heartbeatThread != null)
heartbeatThread.enable();
}
}
@Override
public void onFailure(RuntimeException e) {
synchronized (AbstractCoordinator.this) {
state = MemberState.UNJOINED;
}
}
});
}
return joinFuture;
}
继续深入,创建request
private RequestFuture<ByteBuffer> sendJoinGroupRequest() {
if (coordinatorUnknown())
return RequestFuture.coordinatorNotAvailable();
JoinGroupRequest.Builder requestBuilder = new JoinGroupRequest.Builder(
groupId,
this.sessionTimeoutMs,
this.generation.memberId,
protocolType(),
metadata()).setRebalanceTimeout(this.rebalanceTimeoutMs);
return client.send(coordinator, requestBuilder)
.compose(new JoinGroupResponseHandler());
}
到这里,套路就和基本的发送request讨论一样了,绑定一个回调方法,处理response。
request字段
private final String groupId;
private final int sessionTimeout; //超过该时间没有收到心跳,则任务consumer下线
private final int rebalanceTimeout;
private final String memberId; //GroupCoordinator分配给消费者的ID
private final String protocolType; //实现的协议,默认是“consumer”
private final List<ProtocolMetadata> groupProtocols; //针对不同的PartitionAssignor,序列化后的消费者的订阅信息
--------- response 处理 ---------------
我们这里直接跳转道response的处理,位于JoinGroupResponseHandler类中
if (error == Errors.NONE) {
log.debug("Received successful JoinGroup response: {}", joinResponse);
sensors.joinLatency.record(response.requestLatencyMs());
synchronized (AbstractCoordinator.this) {
if (state != MemberState.REBALANCING) {
// if the consumer was woken up before a rebalance completes, we may have already left
// the group. In this case, we do not want to continue with the sync group.
future.raise(new UnjoinedGroupException());
} else { //走这里
AbstractCoordinator.this.generation = new Generation(joinResponse.generationId(),
joinResponse.memberId(), joinResponse.groupProtocol());
if (joinResponse.isLeader()) { //如果是leader消费者
onJoinLeader(joinResponse).chain(future);
} else { //跟随者
onJoinFollower().chain(future);
}
}
}
}
response字段:
private final int throttleTimeMs;
private final Errors error;
private final int generationId; //GroupCoordinator分配的年代信息
private final String groupProtocol; //GroupCoordinator选择的Assignor
private final String memberId; // GroupCoordinator分配给消费者的Id
private final String leaderId; //leader的memberId
private final Map<String, ByteBuffer> members; //对应消费者的订阅信息
这里就重点了。返回结果中,如果consumer被定位为leader,则要执行分区分配,并且把分区分配的结果,返回给server,而follower则要再发送request,获得分配分区后的结果。
1.首先看一下follower(),这个简单一点
private RequestFuture<ByteBuffer> onJoinFollower() {
SyncGroupRequest.Builder requestBuilder =
new SyncGroupRequest.Builder(groupId, generation.generationId, generation.memberId,
Collections.<String, ByteBuffer>emptyMap());
return sendSyncGroupRequest(requestBuilder);
}
2. onJoinLeader()
它与follower()方法差不多,只不过多了一个分区分配的结果放到request中,而follower的request放入的是一个空集合。
private RequestFuture<ByteBuffer> onJoinLeader(JoinGroupResponse joinResponse) {
try {
Map<String, ByteBuffer> groupAssignment = performAssignment(joinResponse.leaderId(), joinResponse.groupProtocol(),
joinResponse.members());
SyncGroupRequest.Builder requestBuilder =
new SyncGroupRequest.Builder(groupId, generation.generationId, generation.memberId, groupAssignment);
return sendSyncGroupRequest(requestBuilder);
}
具体的分区分配,我们在消费者(1)中讲解了,主要就是调用AbstractPartitionAssignor的assign方法,然后调用对应策略的assign()方法,然后获得最后分配的结果。
这里我们按栈的方式慢慢的返回结果:
得到future之后,第一个 (这些之前都出场过的)
public void onSuccess(ByteBuffer value) {
synchronized (AbstractCoordinator.this) {
state = MemberState.STABLE;
rejoinNeeded = false;
if (heartbeatThread != null)
heartbeatThread.enable();
}
}
if (future.succeeded()) {
onJoinComplete(generation.generationId, generation.memberId, generation.protocol, future.value());
resetJoinGroupFuture();
needsJoinPrepare = true;
}
这个OnJoinComplete()方法,也是很重要的,进入第三阶段的时候,讲解
1.3 第三阶段
完成分区分配之后,进入Synchronizing Group state阶段,主要逻辑是向GroupCoordinator发送SyncGroupRequest请求并处理SyncGroupResponse响应。
我们知道,在分区分配完成之后,leader和follower都会发送这个请求,逻辑简单,我们这里主要是看它是如何处理response的,进入handler代码:
private class SyncGroupResponseHandler extends CoordinatorResponseHandler<SyncGroupResponse, ByteBuffer> {
@Override
public void handle(SyncGroupResponse syncResponse,
RequestFuture<ByteBuffer> future) {
Errors error = syncResponse.error();
if (error == Errors.NONE) {
sensors.syncLatency.record(response.requestLatencyMs());
//传播分区分配的结果
future.complete(syncResponse.memberAssignment());
}
其实,真正处理结果的地方,是在前面的listener的处理方案中的OnJoinComplete()中,位于ConsumerCoordinator
protected void onJoinComplete(int generation,
String memberId,
String assignmentStrategy,
ByteBuffer assignmentBuffer) {
if (!isLeader)
assignmentSnapshot = null;
//查找使用的分配策略
PartitionAssignor assignor = lookupAssignor(assignmentStrategy); //对比策略,本地是否存在
if (assignor == null)
throw new IllegalStateException("Coordinator selected invalid assignment protocol: " + assignmentStrategy);
//2.反序列化,更新assignment
Assignment assignment = ConsumerProtocol.deserializeAssignment(assignmentBuffer);
//将needsFetchCommittedOffsets设置为true,运行从服务端获取最近一次的提交的offset
subscriptions.needRefreshCommits();
//填充assignment集合
subscriptions.assignFromSubscribed(assignment.partitions());
Set<String> addedTopics = new HashSet<>();
for (TopicPartition tp : subscriptions.assignedPartitions()) {
if (!joinedSubscription.contains(tp.topic()))
addedTopics.add(tp.topic());
}
if (!addedTopics.isEmpty()) {
Set<String> newSubscription = new HashSet<>(subscriptions.subscription());
Set<String> newJoinedSubscription = new HashSet<>(joinedSubscription);
newSubscription.addAll(addedTopics);
newJoinedSubscription.addAll(addedTopics);
this.subscriptions.subscribeFromPattern(newSubscription);
this.joinedSubscription = newJoinedSubscription;
}
this.metadata.setTopics(subscriptions.groupSubscription());
client.ensureFreshMetadata();
//回调函数
assignor.onAssignment(assignment);
this.nextAutoCommitDeadline = time.milliseconds() + autoCommitIntervalMs;
ConsumerRebalanceListener listener = subscriptions.listener();
log.info("Setting newly assigned partitions {}", subscriptions.assignedPartitions());
try {
Set<TopicPartition> assigned = new HashSet<>(subscriptions.assignedPartitions());
listener.onPartitionsAssigned(assigned);
} catch (WakeupException | InterruptException e) {
throw e;
} catch (Exception e) {
log.error("User provided listener {} failed on partition assignment", listener.getClass().getName(), e);
}
}