kafka源码解析之十七消费者流程(客户端如何获取topic的数据)

Kafka消费数据的角色分为普通消费者和高级消费者，其介绍如下：

17.1 普通消费者

特点：1）一个消息读取多次

   2）在一个处理过程中只消费某个broker上的partition的部分消息

   3）必须在程序中跟踪offset值

   4）必须找出指定TopicPartition中的lead broker

   5）必须处理broker的变动

客户端编程必须按照以下步骤：

   1）从所有活跃的broker中找出哪个是指定TopicPartition中的leader broker

   2）构造请求

   3）发送请求查询数据

   4）处理leader broker变更

客户端代码如下：

public class KafkaSimpleConsumer {
    private List<String> m_replicaBrokers = new ArrayList<String>();
    public KafkaSimpleConsumer() {
        m_replicaBrokers = new ArrayList<String>();
    }
    public static void main(String args[]) {
        KafkaSimpleConsumer example = new KafkaSimpleConsumer();
        // 最大读取消息数量
        long maxReads = Long.parseLong("3");
        // 要订阅的topic
        String topic = "mytopic";
        // 要查找的分区
        int partition = Integer.parseInt("0");
        // broker节点的ip
        List<String> seeds = new ArrayList<String>();
        seeds.add("192.168.4.30");
        seeds.add("192.168.4.31");
        seeds.add("192.168.4.32");
        // 端口
        int port = Integer.parseInt("9092");
        try {
            example.run(maxReads, topic, partition, seeds, port);
        } catch (Exception e) {
            System.out.println("Oops:" + e);
            e.printStackTrace();
        }
    }
    public void run(long a_maxReads, String a_topic, int a_partition, List<String> a_seedBrokers, int a_port) throws Exception {
        // 获取指定Topic partition的元数据
        PartitionMetadata metadata = findLeader(a_seedBrokers, a_port, a_topic, a_partition);
        if (metadata == null) {
            System.out.println("Can't find metadata for Topic and Partition. Exiting");
            return;
        }
        if (metadata.leader() == null) {
            System.out.println("Can't find Leader for Topic and Partition. Exiting");
            return;
        }
        //找到leader broker
        String leadBroker = metadata.leader().host();
        String clientName = "Client_" + a_topic + "_" + a_partition;
//链接leader broker
        SimpleConsumer consumer = new SimpleConsumer(leadBroker, a_port, 100000, 64 * 1024, clientName);
//获取topic的最新偏移量
        long readOffset = getLastOffset(consumer, a_topic, a_partition, kafka.api.OffsetRequest.EarliestTime(), clientName);
        int numErrors = 0;
        while (a_maxReads > 0) {
            if (consumer == null) {
                consumer = new SimpleConsumer(leadBroker, a_port, 100000, 64 * 1024, clientName);
            }
//本质上就是发送FetchRequest请求
            FetchRequest req = new FetchRequestBuilder().cl