kafka 客户端, 生产者、消费者

生产者

生产者要发送消息，并不是直接发送给服务端，而是先在客户端把消息放入队列中，然后由一个消息发送线程从队列中拉取消息，以批量的方式发送给服务端。

maven

 <dependency>
     <groupId>org.apache.kafka</groupId>
     <artifactId>kafka_2.12</artifactId>
     <version>1.0.1</version>
 </dependency>

实现一个生产者

package com.example.demo;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.util.Properties;

public class Producer extends Thread {
    private final KafkaProducer<Integer, String> prod;
    private final String topic;
    private final Boolean isAsync;
    public Producer(String topic, Boolean isAsync) {
        Properties properties = new Properties();
        properties.setProperty("bootstrap.servers", "192.168.10.151:9092");
        properties.setProperty("key.serializer", "org.apache.kafka.common.serialization.IntegerSerializer");
        properties.setProperty("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        properties.setProperty("client.id", "demoProducer");
        prod = new KafkaProducer<>(properties);
        this.isAsync = isAsync;
        this.topic = topic;
    }
    @Override
    public void run() {
        int num = 1;
        while (true) {
            String msg = "message_" + num;
            //异步发送
            if (isAsync) {
                prod.send(new ProducerRecord<>(topic, num, msg), (recordMetadata, e) -> System.out.println("#offset:" + recordMetadata.offset()));
            }
            //同步发送
            else {
                prod.send(new ProducerRecord<>(topic, num, msg));
            }
        }
    }
    public static void main(String[] args) {
        new Producer("test1", true).start();
        new Producer("test2", false).start();
    }
}

消费者

public class Consumer {
    static String topic = "test1";
    static volatile boolean running = true;
    public static void main(String[] args) {
        Properties properties = new Properties();
        properties.setProperty("bootstrap.servers", "192.168.10.151:9092");
        properties.setProperty("group.id", "test");
        properties.setProperty("enable.auto.commit", "true");
        properties.setProperty("auto.commit.interval.ms", "1000");
        properties.setProperty("session.timeout.ms", "30000");
        properties.setProperty("heartbeat.interval.ms", "2000");
        //  properties.setProperty("group.min.session.timeout.ms", "10000");
        //  properties.setProperty("group.max.session.timeout.ms", "1000");
        properties.setProperty("key.deserializer", "org.apache.kafka.common.serialization.IntegerDeserializer");
        properties.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        KafkaConsumer<Integer, String> consumer = new KafkaConsumer<>(properties);
        consumer.subscribe(Collections.singleton(topic));
        while (running) {
            ConsumerRecords<Integer, String> record = consumer.poll(1000);
            record.forEach(it -> {
                System.out.println(it.key() + "\t" + it.offset() + "\t" + it.value());
            });
        }
    }
}

主要调用了KafkaConsumer提供的两个方法：订阅与轮询

• subscribe()
• poll() 轮询返回消息集

两种消费方式

• 订阅模式 “动态分配“，分区是动态分配的

        consumer.subscribe(Collections.singleton(topic));

• 分配模式 “静态分配“，分区是静态分配的

        List<TopicPartition> topicPartitions = new ArrayList<>();
        topicPartitions.add(new TopicPartition("test1", 0));
        consumer.assign(topicPartitions);

重置偏移量

        consumer.seek(new TopicPartition("test1", 0), 161999044);

消费者提交偏移量

消费组发生再平衡时分区会被分配给新的消费者，为了保证新消费者能够从分区的上一次消费位置继续拉取并处理消息，每个消费者需要将分区的消费进度，定时的同步给消费组对应的协调者节点。
kafka提供了两种提交偏移量的方式：

• 异步模式

consumer.commitAsync();

• 同步模式

consumer.commitAsync(new OffsetCommitCallback() {
            @Override
            public void onComplete(Map<TopicPartition, OffsetAndMetadata> offsets, Exception exception) {
                //
            }
        });

注意：如果消费者客户端设置了自动提交的(enable.auto.commit=true,默认是开启的)选项，会在客户端的轮询操作中调度定时任务，定时任务也属于异步模式提交偏移量的一种运用场景。

心跳任务