Consumer的状态机
consumer,是一个单线程机制,包括和coordinator通讯,rebalance, heartbeat等,都是在单线程的poll函数里面。也因此,在consumer的代码中不需要任何锁。
启动过程
一般在代码中使用consumer是下面两句
consumer = new KafkaConsumer<>(props);//构造一个KafkaConsumer
consumer.subscribe(Arrays.asList(topicName));//提交之
ConsumerRecords<String, byte[]> records = consumer.poll(100);//接收之
首先看构造过程,配置自己的client.id和group.id。然后配置各种个样的参数,在这里不加以解释。
String clientId = config.getString("client.id");
if (clientId.isEmpty()) {//clientId如果找不到,则利用AtomicInteger自增
clientId = "consumer-" + CONSUMER_CLIENT_ID_SEQUENCE.getAndIncrement();
}
this.clientId = clientId;
String groupId = config.getString("group.id");
在KafkaConsumer对象中,维护者字段来记录该Consumer的订阅状态
private final SubscriptionState subscriptions;
在subscriptions对象中,通过一个Set来保存所订阅的topic
private final Set<String> groupSubscription;
提交commit
//KafkaConsumer.java
private static final long NO_CURRENT_THREAD = -1L;
private final AtomicLong currentThread = new AtomicLong(NO_CURRENT_THREAD);
//虽然如下代码里有获取锁和释放锁的逻辑,但是kafka consumer并不支持并发访问。实现锁的方式就是简单跑出不支持并发访问的异常
public void commitAsync(final Map<TopicPartition, OffsetAndMetadata> offsets, OffsetCommitCallback callback) {
acquireAndEnsureOpen();//获取锁
try {
log.debug("Committing offsets: {}", offsets);
//向协调者异步提交commit
coordinator.commitOffsetsAsync(new HashMap<>(offsets), callback);
} finally {
release();//释放锁
}
}
private void acquireAndEnsureOpen() {
acquire();
if (this.closed) {
release();
throw new IllegalStateException("This consumer has already been closed.");
}
}
private void acquire() {
long threadId = Thread.currentThread().getId();
if (threadId != currentThread.get() && !currentThread.compareAndSet(NO_CURRENT_THREAD, threadId))
throw new ConcurrentModificationException("KafkaConsumer is not safe for multi-threaded access");
refcount.incrementAndGet();
}
private void release() {
if (refcount.decrementAndGet() == 0)
currentThread.set(NO_CURRENT_THREAD);
}
再平衡过程
所谓rebalance,就是在某些条件下,partition要在consumer中重新分配。那哪些条件下,会触发rebalance呢?
- 有新的consumer加入
- 旧的consumer挂了
- coordinator挂了,集群选举出新的coordinator
- topic的partition新加
- consumer调用unsubscrible(),取消topic的订阅