kafka0.9

http://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

@InterfaceStability.Unstable
public class KafkaConsumer

offset和consumer position

kafka在每个partition内部用offset标示为每条记录。具有position5的消费者，已经消费了0到4的offset，下一次要读取offset5的记录。事实上有两个关于消费者使用者的notion。

• 消费者的position通知下一次要消费的record的offset，这个position比消费者在一个partition中看到的最高的offset大1，当消费者调用poll接收到数据后就会自动加1。
• 提交的postiion是已经安全保存的最后一个offset。如果进程失败或者重启，这时进程要恢复的那个offset。消费者或者可以周期性的提交offset，或者可以通过调用commitSync方法手动来完成，这个方法会阻塞直到offset被成功提交，或者在提交的过程中有错误也会停止阻塞。或者也可以使用commitAsync，这是一个非阻塞方法，如果成功提交或者提交失败会触发OffsetCommitCallback方法。

这种差异可以让consumer控制何时一个record被消费了。后边详细讨论。
This distinction gives the consumer control over when a record is considered consumed. It is discussed in further detail below.

消费者组和topic订阅

kafka使用消费者组的概念来让一组进程切分消费任务，这些进程可以运行在同一台机器上，但是在多数情况下，这些进程是分布在很多台机器上的。每个kafka消费者可以配置一个消费者组，并且可以动态设置要订阅的topic集合(通过方法subsribe(List, ConsumerRebalanceListener))，或者通过模式匹配来订阅所有相关的topic(subscribe(Pattern, ConsumerRebalanceListener))。

kafka会给每一个消费者组内的进程发送一次消息，这是通过在每一个组内对消费者进程的partition进行balance做到的。所以如果一个topic有4个partition，消费组哟两个进程，那么每个进程将消费两个partition。这种关系是动态维护的：如果一个进程挂了，它上边的partition将会被重新分配给同组内的其他进程，如果有新的进程加入到这个组，partition就从现有的消费者中被移动到新的进程。

所以如果两个进程通过不同的组名订阅了一个topic，每个进程都会获取到topic的全部消息。

Conceptually you can think of a consumer group as being a single logical subscriber that happens to be made up of multiple processes. As a multi-subscriber system, Kafka naturally supports having any number of consumer groups for a given topic without duplicating data (additional consumers are actually quite cheap).

This is a slight generalization of the functionality that is common in messaging systems. To get semantics similar to a queue in a traditional messaging system all processes would be part of a single consumer group and hence record delivery would be balanced over the group like with a queue. Unlike a traditional messaging system, though, you can have multiple such groups. To get semantics similar to pub-sub in a traditional messaging system each process would have its own consumer group, so each process would subscribe to all the records published to the topic.

In addition, when group reassignment happens automatically, consumers can be notified through ConsumerRebalanceListener, which allows them to finish necessary application-level logic such as state cleanup, manual offset commits (note that offsets are always committed for a given consumer group), etc. See Storing Offsets Outside Kafka for more details

It is also possible for the consumer to manually specify the partitions that are assigned to it through assign(List), which disables this dynamic partition assignment.

例子

The consumer APIs offer flexibility to cover a variety of consumption use cases. Here are some examples to demonstrate how to use them.
Automatic Offset Committing

This example demonstrates a simple usage of Kafka’s consumer api that relying on automatic offset committing.

     Properties props = new Properties();
     props.put("bootstrap.servers", "localhost:9092");
     props.put("group.id", "test");
     props.put("enable.auto.commit", "true");
     props.put("auto.commit.interval.ms", "1000");
     props.put("session.timeout.ms", "30000");
     props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
     props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
     KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
     consumer.subscribe(Arrays.asList("foo", "bar"));
     while (true) {
         ConsumerRecords<String, String> records = consumer.poll(100);
         for (ConsumerRecord<String, String> record : rec