Kafka源代码阅读（5）：consumer

Consumer的状态机

consumer，是一个单线程机制，包括和coordinator通讯，rebalance, heartbeat等，都是在单线程的poll函数里面。也因此，在consumer的代码中不需要任何锁。

启动过程

一般在代码中使用consumer是下面两句

        consumer = new KafkaConsumer<>(props);//构造一个KafkaConsumer
        consumer.subscribe(Arrays.asList(topicName));//提交之
        ConsumerRecords<String, byte[]> records = consumer.poll(100);//接收之

首先看构造过程，配置自己的client.id和group.id。然后配置各种个样的参数，在这里不加以解释。

            String clientId = config.getString("client.id");
            if (clientId.isEmpty()) {//clientId如果找不到，则利用AtomicInteger自增
                clientId = "consumer-" + CONSUMER_CLIENT_ID_SEQUENCE.getAndIncrement();
            }

            this.clientId = clientId;
            String groupId = config.getString("group.id");

在KafkaConsumer对象中，维护者字段来记录该Consumer的订阅状态

private final SubscriptionState subscriptions;

在subscriptions对象中，通过一个Set来保存所订阅的topic

private final Set<String> groupSubscription;

提交commit

    //KafkaConsumer.java
	 private static final long NO_CURRENT_THREAD = -1L;
	 private final AtomicLong currentThread = new AtomicLong(NO_CURRENT_THREAD);


    //虽然如下代码里有获取锁和释放锁的逻辑，但是kafka consumer并不支持并发访问。实现锁的方式就是简单跑出不支持并发访问的异常
    public void commitAsync(final Map<TopicPartition, OffsetAndMetadata> offsets, OffsetCommitCallback callback) {
        acquireAndEnsureOpen();//获取锁
        try {
            log.debug("Committing offsets: {}", offsets);
            //向协调者异步提交commit
            coordinator.commitOffsetsAsync(new HashMap<>(offsets), callback);
        } finally {
            release();//释放锁
        }
    }

    private void acquireAndEnsureOpen() {
        acquire();
        if (this.closed) {
            release();
            throw new IllegalStateException("This consumer has already been closed.");
        }
    }

    private void acquire() {
        long threadId = Thread.currentThread().getId();
        if (threadId != currentThread.get() && !currentThread.compareAndSet(NO_CURRENT_THREAD, threadId))
            throw new ConcurrentModificationException("KafkaConsumer is not safe for multi-threaded access");
        refcount.incrementAndGet();
    }

    private void release() {
        if (refcount.decrementAndGet() == 0)
            currentThread.set(NO_CURRENT_THREAD);
    }

再平衡过程

所谓rebalance，就是在某些条件下，partition要在consumer中重新分配。那哪些条件下，会触发rebalance呢？

• 有新的consumer加入
• 旧的consumer挂了
• coordinator挂了，集群选举出新的coordinator
• topic的partition新加
• consumer调用unsubscrible()，取消topic的订阅