springboot + kafka （二）kafka-client队列消息的发送和监听

1.kafka-client介绍

之前的博客单个队列消息的发送和监听是springboot+sping-kafka的单个队列的发送和监听。最近项目接到一个需求，就是接入腾讯ckafka,而ckafka最高支持到kafka版本V1.1.1。于是便使用了kafka-client的版本1.0.2，便于接入ckafka。而kafka-client是kafka官方调用kafka的客户端版本。

2.自定义配置文件

2.1 配置文件

此处为yml自定义配置文件，一些个性化需求可以修改。

kafka:
  #=============== provider  =======================
  bootstrap-servers: 127.0.0.1:9092
  session-out: 30000
  retry: 3
  # 每次批量发送消息的数量
  batch-size: 232840
  buffer-memory: 33554432
  username:
  password:
  ##=============== consumer  =======================
  consumer-groupId: user-group-0
  auto-commit-interval: 5000
  enable.auto.commit: true

2.2 配置producer 和consumer

此处由于调试的时候使用了sasl安全方式，此处方便两种方式调用，由于作者水平有限，可以自定义配置。字段意义后续博客讲解。此文主要介绍接入springboot，并启动时启动监听。

• 引入自定义配置

    @Value("${kafka.bootstrap-servers}")
    private String servers;
    @Value("${kafka.session-out}")
    private String sessionOut;
    @Value("${kafka.retry}")
    private int retry;
    @Value("${kafka.batch-size}")
    private int batchSize;
    @Value("${kafka.buffer-memory}")
    private String bufferMemory;
    @Value("${kafka.password}")
    private String kafkaPassword;
    @Value("${kafka.username}")
    private String kafkaUserName;
    @Value("${kafka.enable.auto.commit}")
    private String enableAutoCommit;

    //    --------------consumer-----------------
    @Value("${kafka.consumer-groupId}")
    private String groupId;
    @Value("${kafka.auto-commit-interval}")
    private String autoCommitInterval;

• 配置consumer

    @Bean
    @Scope("prototype")
    public KafkaConsumer kafkaConsumer() {
        Properties props = new Properties();
        props.put("bootstrap.servers", servers);
        props.put("group.id", groupId);
        props.put("auto.commit.interval.ms", autoCommitInterval);
        props.put("session.timeout.ms", sessionOut);
        if (!StringUtils.isEmpty(kafkaUserName) && !StringUtils.isEmpty(kafkaPassword)) {
            props.put("sasl.jaas.config",
                    "org.apache.kafka.common.security.plain.PlainLoginModule required username=" + kafkaUserName + " password=" + kafkaPassword + ";");
            props.put("authorizer.class.name", "kafka.security.auth.SimpleAclAuthorizer");
            props.put("security.protocol", "SASL_PLAINTEXT");
            props.put("sasl.mechanism", "PLAIN");
        }

        props.put("enable.auto.commit", "true");
        props.put("auto.offset.reset", "earliest");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        return new KafkaConsumer(props);
    }

• 配置producer

    @Bean
    public KafkaProducer kafkaProducer() {
        Properties props = new Properties();
        props.put("bootstrap.servers", servers);
        props.put("session.timeout.ms", sessionOut);
        props.put("retries", retry);
        props.put("buffer.memory", bufferMemory);
        props.put("batch.size", batchSize);
        if (!StringUtils.isEmpty(kafkaUserName) && !StringUtils.isEmpty(kafkaPassword)) {
            props.put("sasl.jaas.config",
                    "org.apache.kafka.common.security.plain.PlainLoginModule required username=" + kafkaUserName + " password=" + kafkaPassword + ";");
            props.put("security.protocol", "SASL_PLAINTEXT");
            props.put("sasl.mechanism", "PLAIN");
            props.put("authorizer.class.name", "kafka.security.auth.SimpleAclAuthorizer");
        }

        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("acks", "1");
        props.put("linger.ms", 10);
        props.put("max.block.ms", 3000);
        return new KafkaProducer(props);

    }

}

3. Restful 方式发送消息

由于要多次发送消息，保证kafkaPruducer不关闭，此处注掉close。此处简单示例，正确开发应该使用controller对外暴露接口

   @Autowired
   private KafkaProducer kafkaProducer;

   @RequestMapping("/send/{topic}/{message}")
   public String sendMessage(@PathVariable("topic") String topic, @PathVariable("message") String message) {
       asynSendRecord(topic, message);
       return message;
   }

   //异步发送消息
   public void asynSendRecord(String topic, Object message) {
       ProducerRecord<String, Object> record = new ProducerRecord<>(topic, message);
       log.info("record:" + record.value());
       kafkaProducer.send(record, (recordMetadata, e) -> {
           if (e == null) {
               log.info("消息发送: offset: {} timestamp:{}  topic:{}  partition: {} ", recordMetadata.offset(), recordMetadata.timestamp(), recordMetadata.topic(), recordMetadata.partition());
               log.info("消息发送成功");
           } else {
               log.error(String.format("消息发送失败: {}", e.getMessage()));
           }
       });

	// kafkaProducer.close();
   }

4. 监听消费messages

    @Autowired
    private KafkaMessageService kafkaMessageService;

    public void onMessage(KafkaConsumer kafkaConsumer, List<String> topic) {
        kafkaConsumer.subscribe(topic);
        log.info("队列开始监听：topic {}", topic);
        try {
            while (true) {
                ConsumerRecords<String, String> records = kafkaConsumer.poll(1000);
                for (ConsumerRecord<String, String> record : records) {
                    log.info("partition:{} offset = {}, key = {}, value = {}",record.partition(), record.offset(), record.key(), record.value());
                    try {
                        String messageData = new String(record.value().getBytes(), StandardCharsets.UTF_8);
                        log.info("{}解析处理内容为:{}", LOGGER_MSG, messageData);
                        handle(record.topic(), messageData);
                    } catch (Exception e) {
                        log.error("消息处理异常");
                    }
                }
            }
        } finally {
		// kafkaConsumer.close();
        }
    }

5.SpringBoot 启动时加载监听

5.1 Consumer和Listener对象注入

此处实现 org.springframework.boot.ApplicationRunner 接口，在 重写run方法中，启动监听topic的功能。
当考虑到启动监听的时候我是这样注入spring 消费服务和监听业务方法的。

    @Autowired
    KafkaListenMessageHandler kafkaInvoiceHandler;
    @Autowired
    KafkaConsumer kafkaConsumer;

后来由于业务中有多个topic,而我又都写入到一个Handler中，此处需要原型模式来实现多实例。所以在 KafkaConsumer和KafkaListenMessageHandler中添加注解 @Scope(“prototype”)，KafkaListenMessageHandler就是监听消费的实体对象。

• KafkaConsumer对象

  	@Bean
    @Scope("prototype")
    public KafkaConsumer kafkaConsumer() {...}

• KafkaListenMessageHandler对象

  	@Bean
    @Scope("prototype")
	public class KafkaListenMessageHandler {...}

5.2 线程启动时调用Listener

一开始我是这样做的,k-a,k-b就是需要监听的队列。而启动的时候kafkaConsumer和kafkaInvoiceHandler也应该是多实例。以下为错误示例：

new Thread(() -> kafkaListenMessageHandler.onMessage(kafkaConsumer,Arrays.asList("k-a"))).start();
new Thread(() -> kafkaListenMessageHandler.onMessage(kafkaConsumer,Arrays.asList("k-b"))).start();

后续感觉使用线程池比较好，修改成了线程池模式。线程的个数可以根据业务调整。

public static ExecutorService executorService = Executors.newFixedThreadPool(2);
@Override
public void run(ApplicationArguments args) {
    log.info("监听服务启动!");
    executorService.execute(() -> {
        KafkaListenMessageHandler kafkaListenMessageHandler = SpringBeanUtils.getBean(KafkaListenMessageHandler.class);
        kafkaListenMessageHandler.onMessage(SpringBeanUtils.getBean("kafkaConsumer"), Arrays.asList("k-a"));
    });
    executorService.execute(() -> {
        KafkaListenMessageHandler kafkaListenMessageHandler = SpringBeanUtils.getBean("kafkaInvoiceHandler");
        kafkaListenMessageHandler.onMessage(SpringBeanUtils.getBean("kafkaConsumer"), Arrays.asList("k-b"));
    });
}

5.3 获取对象SpringUtils#getBean方法

@SuppressWarnings("unchecked")
@Component
public class SpringBeanUtils implements ApplicationContextAware {

    private static ApplicationContext applicationContext;
    @Override
    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
        SpringBeanUtils.applicationContext = applicationContext;
    }
    public static <T> T getBean(String beanName) {
        if (applicationContext.containsBean(beanName)) {
            return (T) applicationContext.getBean(beanName);
        } else {
            return null;
        }
    }
    public static <T> T getBean(Class<T> clazz) {
        return applicationContext.getBean(clazz);
    }
}

以上为所有内容，由于比较晚，许多内容没有详细解释，如有疑问，可以评论。下文第三篇准备完成上文承诺的多队列监听消费。第四篇准备完成队列的自动创建和config的修改。