使用@KafkaListener配置两个不同的Consumer监听不同kafka集群的消息

我们的项目中很多系统交互使用的kafka,最近遇到一个问题,原来我们的kafka需要监听消费我们自己的kafka生产的消息,但是现在需要使用kafka来消费外部系统的消息,但是我们原来的使用非常简单,直接在application.yml中进行的配置,同时监听两个kafka的话,不能实现,于是就要解决这个问题啦。

下面是我们之前的配置方式:

yml文件中的配置:

spring.kafka.consumer.bootstrap-servers=localhost:9092

消费者方法加上如下注解:

@KafkaListener(topics = "Topic001")

可以看到,这样是无法对kafka服务地址进行选择的,但是我们发现,在@KafkaListener这个注解中,有一个containerFactory参数,而containerFactory中是可以配置kafka的bootstrapServers的,所以问题就简单啦,我们只需要多建几个containerFactory,配置不通的bootstrapServers,然后在KafkaListener注解上传入不同的containerFactory就好啦。

接下来看实现:

这是我的KafkaConfig

@Configuration
@Slf4j
@EnableConfigurationProperties({KafkaProducerProperties.class, KafkaConsumerProperties.class, KafkaLocalConsumerProperties.class})
public class KafkaConfig {
    private final KafkaProducerProperties producerProperties;
    private final KafkaConsumerProperties defaultConsumerProperties;
    private final KafkaLocalConsumerProperties localConsumerProperties;

    public KafkaConfig(KafkaProducerProperties kafkaProperties, KafkaConsumerProperties kafkaConsumerProperties, KafkaLocalConsumerProperties kafkaLocalConsumerProperties) {
        this.producerProperties = kafkaProperties;
        this.defaultConsumerProperties = kafkaConsumerProperties;
        this.localConsumerProperties = kafkaLocalConsumerProperties;
    }

    @Bean
    @Primary
    public ConsumerFactory<Object, Object> defaultConsumerFactory() {
        return createConsumerFactory(defaultConsumerProperties);
    }

    @Bean
    public ConsumerFactory<Object, Object> localConsumerFactory() {
        return createConsumerFactory(localConsumerProperties);
    }

    private DefaultKafkaConsumerFactory<Object, Object> createConsumerFactory(KafkaBaseConsumerProperties kafkaConsumerProperties) {
        Map<String, Object> configs = new HashMap<>();
        configs.put(ConsumerConfig.GROUP_ID_CONFIG, kafkaConsumerProperties.getGroupId());
        configs.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaConsumerProperties.getBootstrapServers());
        configs.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, kafkaConsumerProperties.getAutoCommitInterval());
        configs.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, kafkaConsumerProperties.isEnableAutoCommit());
        configs.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, kafkaConsumerProperties.getAutoOffsetReset());
        configs.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, kafkaConsumerProperties.getValueDeserializer());
        configs.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, kafkaConsumerProperties.getKeyDeserializer());
        configs.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 5);
        return new DefaultKafkaConsumerFactory<>(configs);
    }


    @Bean
    @Primary
    public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory(
            ConsumerFactory<Object, Object> consumerFactory) {
        return createListenerContainerFactory(consumerFactory, defaultConsumerProperties);
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> localListenerContainerFactory(
            @Qualifier("localConsumerFactory") ConsumerFactory<Object, Object> consumerFactory) {
        return createListenerContainerFactory(consumerFactory, localConsumerProperties);
    }


    private ConcurrentKafkaListenerContainerFactory<String, String> createListenerContainerFactory(ConsumerFactory<Object, Object> consumerFactory, KafkaBaseConsumerProperties properties) {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        if (properties.getConcurrency() > 1) {
            //设置并发量,小于或等于Topic的分区数
            factory.setConcurrency(properties.getConcurrency());
            //想要上面的设置生效,必须设置为批量监听
            factory.setBatchListener(true);
        }
        factory.setConsumerFactory(consumerFactory);
        factory.setAutoStartup(properties.isEnableAutoStartup());
        return factory;
    }
}

listener:

@KafkaListener(topics = "testTopic", containerFactory = "localListenerContainerFactory",concurrency = "3",autoStartup = "true")
    private void localTest(String message) {
        log.info("[LocalKafkaConsumer Test API] get message from kafka : [ {} ]", message);
    }

亲测有效。

©️2020 CSDN 皮肤主题: 点我我会动 设计师:上身试试 返回首页