kafka 笔记三 kafka消费者 心跳

消费者心跳机制

Kafka 的心跳是 Kafka Consumer 和 Broker 之间的健康检查，只有当 Broker Coordinator 正常时，Consumer 才会发送心跳。

Consumer 和 Rebalance 相关的 2 个配置参数：

broker 端，sessionTimeoutMs 参数

broker 处理心跳的逻辑在 GroupCoordinator 类中：如果心跳超期， broker coordinator 会把消费者从 group 中移除，并触发 rebalance 。

private def completeAndScheduleNextHeartbeatExpiration(group: GroupMetadata, member: MemberMetadata) { 
    // complete current heartbeat expectation 
    member.latestHeartbeat = time.milliseconds() 
    val memberKey = MemberKey(member.groupId, member.memberId)         
    heartbeatPurgatory.checkAndComplete(memberKey)                    
    // reschedule the next heartbeat expiration deadline 
    // 计算心跳截止时刻 
    val newHeartbeatDeadline = member.latestHeartbeat + member.sessionTimeoutMs 
    val delayedHeartbeat = new DelayedHeartbeat(this, group, member, newHeartbeatDeadline, member.sessionTimeoutMs) 
    heartbeatPurgatory.tryCompleteElseWatch(delayedHeartbeat, Seq(memberKey)) 
}                    

// 心跳过期
def onExpireHeartbeat(group: GroupMetadata, member: MemberMetadata, heartbeatDeadline: Long) { 
    group.inLock { 
        if (!shouldKeepMemberAlive(member, heartbeatDeadline)) { 
            info(s"Member ${member.memberId} in group ${group.groupId} has failed, removing it from the group") 
            removeMemberAndUpdateGroup(group, member) 
        } 
    } 
}

private def shouldKeepMemberAlive(member: MemberMetadata, heartbeatDeadline: Long) = 
member.awaitingJoinCallback != null || member.awaitingSyncCallback != null || 
member.latestHeartbeat + member.sessionTimeoutMs > heartbeatDeadline

consumer 端：sessionTimeoutMs，rebalanceTimeoutMs 参数

如果客户端发现心跳超期，客户端会标记 coordinator 为不可用，并阻塞心跳线程；如果poll 消息的间隔超过了 rebalanceTimeoutMs ，则 consumer 告知 broker 主动离开消费组，也会触发rebalance
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.HeartbeatThread

if (coordinatorUnknown()) {
    if (findCoordinatorFuture != null || lookupCoordinator().failed())
        // the immediate future check ensures that we backoff properly in the case that no
        // brokers are available to connect to.
        AbstractCoordinator.this.wait(retryBackoffMs);
} else if (heartbeat.sessionTimeoutExpired(now)) {
    // the session timeout has expired without seeing a successful heartbeat, so we should
    // probably make sure the coordinator is still healthy.
    markCoordinatorUnknown();
} else if (heartbeat.pollTimeoutExpired(now)) {
    // the poll timeout has expired, which means that the foreground thread has stalled
    // in between calls to poll(), so we explicitly leave the group.
    maybeLeaveGroup();
} else if (!heartbeat.shouldHeartbeat(now)) {
    // poll again after waiting for the retry backoff in case the heartbeat failed or the
    // coordinator disconnected
    AbstractCoordinator.this.wait(retryBackoffMs);
} else {
    heartbeat.sentHeartbeat(now);

    sendHeartbeatRequest().addListener(new RequestFutureListener<Void>() {
        @Override
        public void onSuccess(Void value) {
            synchronized (AbstractCoordinator.this) {
                heartbeat.receiveHeartbeat(time.milliseconds());
            }
        }

        @Override
        public void onFailure(RuntimeException e) {
            synchronized (AbstractCoordinator.this) {
                if (e instanceof RebalanceInProgressException) {
                    // it is valid to continue heartbeating while the group is rebalancing. This
                    // ensures that the coordinator keeps the member in the group for as long
                    // as the duration of the rebalance timeout. If we stop sending heartbeats,
                    // however, then the session timeout may expire before we can rejoin.
                    heartbeat.receiveHeartbeat(time.milliseconds());
                } else {
                    heartbeat.failHeartbeat();

                    // wake up the thread if it's sleeping to reschedule the heartbeat
                    AbstractCoordinator.this.notify();
                }
            }
        }
    });
}

手动提交offset  异步提交commitAsync出现问题不会自动重试

try {
    while(true) { 
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));     
        process(records); // 处理消息 
        commitAysnc(); // 使用异步提交规避阻塞 
    } 
} catch(Exception e) { 
    handle(e); // 处理异常 
} finally 
    { 
        try {
            consumer.commitSync(); // 最后一次提交使用同步阻塞式提交 
        } finally { 
            consumer.close(); 
        } 
}

避免重平衡
在分布式系统中，由于网络问题你不清楚没接收到心跳，是因为对方真正挂了还是只是因为负载过重没来得及发生心跳或是网络堵塞。所以一般会约定一个时间，超时即判定对方挂了。而在kafka消费者场景中，session.timout.ms参数就是规定这个超时时间是多少。
还有一个参数，heartbeat.interval.ms，这个参数控制发送心跳的频率，频率越高越不容易被误判，但也会消耗更多资源。
此外，还有最后一个参数，max.poll.interval.ms，消费者poll数据后，需要一些处理，再进行拉取。如果两次拉取时间间隔超过这个参数设置的值，那么消费者就会被踢出消费者组。也就是说，拉取，然后处理，这个处理的时间不能超过 max.poll.interval.ms 这个参数的值。这个参数的默认值是5分钟，而如果消费者接收到数据后会执行耗时的操作，则应该将其设置得大一些。
三个参数，
session.timout.ms控制心跳超时时间，
heartbeat.interval.ms控制心跳发送频率，
max.poll.interval.ms控制poll的间隔。

这里给出一个相对较为合理的配置，如下：
session.timout.ms：设置为6s
heartbeat.interval.ms：设置2s
max.poll.interval.ms：推荐为消费者处理消息最长耗时再加1分钟