kafka优雅集成(多线程提升性能)


最近项目改造,原先是使用的rabbitMq。此次改造,我们重新调研整体流程,为结合原已有服务,所以使用的Kafka。
我的集成思路简单清晰,不过还有提升的地方,直接上菜。
一、消费者配置


import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.EnableKafka;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.config.KafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
import org.springframework.kafka.listener.ConcurrentMessageListenerContainer;
import org.springframework.kafka.listener.ContainerProperties;

import java.util.HashMap;
import java.util.Map;

/**
 * kafka消费者配置
 * @author zhengwen
 */
@EnableKafka
@Configuration
public class KafkaConsumerConfig {

    /**
     * 服务地址
     */
    @Value("${spring.kafka.bootstrap-servers}")
    private String servers;
    /**
     * 分组id
     */
    @Value("${spring.kafka.consumer.group-id}")
    private String groupId;
    /**
     * 自动提交
     */
    @Value("${spring.kafka.consumer.enable-auto-commit}")
    private boolean enableAutoCommit;
    /**
     * 自动提交的间隔时间,默认值是 5000,单位是毫秒
     */
    @Value("${spring.kafka.consumer.auto-commit-interval}")
    private String autoCommitInterval;
    /**
     * 批量消费一次最大拉取的数据量
     */
    @Value("${spring.kafka.consumer.max-poll-records}")
    private int maxPollRecordsConfig;

    /**
     * 消费者配置
     * @return 配置map
     */
    public Map<String, Object> consumerConfigs() {
        Map<String, Object> propsMap = new HashMap<>(10);
        propsMap.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, servers);
        propsMap.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, enableAutoCommit);
        propsMap.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, autoCommitInterval);
        propsMap.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        propsMap.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        // 组名
        propsMap.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
        propsMap.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, maxPollRecordsConfig);
        return propsMap;
    }

    @Bean
    public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        factory.setConcurrency(10);
        factory.getContainerProperties().setPollTimeout(1500);
        factory.setBatchListener(true);
        factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
        return factory;
    }

    public ConsumerFactory<String, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(consumerConfigs());
    }

}

二、生产者配置


import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.EnableKafka;
import org.springframework.kafka.core.DefaultKafkaProducerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.core.ProducerFactory;

import java.util.HashMap;
import java.util.Map;

/**
 * kafka生产者配置
 * @author zhengwen
 */
@EnableKafka
@Configuration
public class KafkaProducerConfig {

    /**
     * 服务地址
     */
    @Value("${spring.kafka.bootstrap-servers}")
    private String servers;
    /**
     * 重试失败的发送次数
     */
    @Value("${spring.kafka.producer.retries}")
    private int retries;
    /**
     * 批量数据大小
     */
    @Value("${spring.kafka.producer.batch-size}")
    private int batchSize;
    /**
     * 缓冲内存大小
     */
    @Value("${spring.kafka.producer.buffer-memory}")
    private int bufferMemory;

    /**
     * 生产者配置
     * @return 配置map
     */
    public Map<String, Object> producerConfigs() {
        Map<String, Object> props = new HashMap<>();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, servers);
        props.put(ProducerConfig.RETRIES_CONFIG, retries);
        props.put(ProducerConfig.BATCH_SIZE_CONFIG, batchSize);
        props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, bufferMemory);
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        return props;
    }

    @Bean
    public KafkaTemplate<String, String> kafkaTemplate() {
        return new KafkaTemplate(producerFactory());
    }

    public ProducerFactory<String, String> producerFactory() {
        return new DefaultKafkaProducerFactory<>(producerConfigs());
    }
}

三、多线程配置


import lombok.extern.slf4j.Slf4j;
import org.springframework.aop.interceptor.AsyncUncaughtExceptionHandler;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.AsyncConfigurer;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;

import java.lang.reflect.Method;
import java.util.concurrent.Executor;

/**
 * @author zhengwen
 */
@Configuration
@EnableAsync
@Slf4j
public class ThreadConfig implements AsyncConfigurer {

    /**
     * 核心线程树
     */
    @Value("${thread.config.corePoolSize:5}")
    private Integer corePoolSize;
    /**
     * 最大线程池数量
     */
    @Value("${thread.config.maxPoolSize:10}")
    private Integer maxPoolSize;
    /**
     * 队列长度
     */
    @Value("${thread.config.queueCapacity:10}")
    private Integer queueCapacity;

    @Override
    public Executor getAsyncExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        log.info("---------[多线程配置初始化],最大核心线程数:{},最大线程池数量:{},线程处理队列长度:{}", corePoolSize, maxPoolSize, queueCapacity);
        //核心线程数
        executor.setCorePoolSize(corePoolSize);
        //最大线程池数量
        executor.setMaxPoolSize(maxPoolSize);
        //线程处理队列长度
        executor.setQueueCapacity(queueCapacity);
        executor.initialize();
        return executor;
    }

    @Override
    public AsyncUncaughtExceptionHandler getAsyncUncaughtExceptionHandler() {
        log.info("-----------多线程异常handler------------");
        return new SpringAsyncExceptionHandler();
    }

    class SpringAsyncExceptionHandler implements AsyncUncaughtExceptionHandler {
        @Override
        public void handleUncaughtException(Throwable ex, Method method, Object... params) {
            log.error("Asyn返回异常:" + ex.getCause().getMessage() + method.getName());
        }
    }
}

四、监听方式消费


import lombok.extern.slf4j.Slf4j;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.support.Acknowledgment;
import org.springframework.stereotype.Component;

import java.util.List;

/**
 * 消费者listener
 *
 * @author zhengwen
 **/
@Slf4j
@Component
public class AnalyzeConsumer {

    @Autowired
    private AnalyzeService analyzeService;


    /**
     * 车辆抓拍
     * @param records 消费信息
     * @param ack Ack机制
     */
    @KafkaListener(topics = "${fillersmart.analyze.car.topic.consumer}", containerFactory = "kafkaListenerContainerFactory")
    public void carListen(List<ConsumerRecord> records, Acknowledgment ack) {
        log.info("=====车辆carListen消费者接收信息====");
        try {
            for (ConsumerRecord record : records) {
                log.info("---开启线程解析车辆数据:{}",record.toString());
                analyzeService.carAnalyze(record);
            }
        } catch (Exception e) {
            e.printStackTrace();
            log.error("----车辆消费者解析数据异常:{}",e.getMessage(),e);
        } finally {
            //手动提交偏移量
            ack.acknowledge();
        }
    }

}

四、生产者
我这里是消费信息做了处理以后再放回去到另一个topic。如果不需要生产的,可以不要了。


import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Component;
import org.springframework.util.concurrent.FailureCallback;
import org.springframework.util.concurrent.ListenableFuture;
import org.springframework.util.concurrent.SuccessCallback;

/**
 * 解析生产者
 * @author zhengwen
 **/
@Slf4j
@Component
public class AnalyzeProducer {

    @Autowired
    private KafkaTemplate kafkaTemplate;

    /**
     * 发送数据到kafka
     * @param topic topic名称
     * @param msg 发送信息字符串
     * @param sendCallBack 发送回调
     */
    public void send(String topic, String msg, SendCallBack sendCallBack) {

        ListenableFuture listenableFuture = kafkaTemplate.send(topic,msg);
        //发送成功后回调
        SuccessCallback<String> successCallback = new SuccessCallback() {
            @Override
            public void onSuccess(Object result) {
                sendCallBack.sendSuccessCallBack(topic,msg);
            }
        };
        //发送失败回调
        FailureCallback failureCallback = new FailureCallback() {
            @Override
            public void onFailure(Throwable ex) {
                sendCallBack.sendFailCallBack(topic,msg,ex);
            }
        };

        listenableFuture.addCallback(successCallback,failureCallback);
    }
}

五、回调方法函数
定义了回调接口,发送的时候传入。


/**
 * @author zhengwen
 **/
public interface SendCallBack {
    /**
     * 生产成功回调
     * @param topic topic
     * @param msg 信息字符串
     */
    void sendSuccessCallBack(String topic,String msg);

    /**
     * 生产失败回调
     * @param topic topic
     * @param msg 信息字符串
     * @param ex 异常
     */
    void sendFailCallBack(String topic,String msg,Throwable ex);
}

六、使用方法
就这上面几步步就ok了,AnalyzeService就是普通的service,注意它的实现类的方法要体现多线程,需要再实现方法上加 @Async标签,否则无效。多线程配置之前其实我也分享过,我是个怕麻烦的人,都是喜欢简洁的集成方法。实现方法里就可以放飞自我了,这里就看看实现方法怎么调用的。

//解析完成的数据发送到kafka
        String msg = JSON.toJSONString(collectDataDto);
        analyzeProducer.send(carProducerTopic,msg, new SendCallBack() {
            @Override
            public void sendSuccessCallBack(String topic, String msg) {
                log.info("----车辆抓拍kafka记录解析完成放入topic:{},发送成功:{}",topic, msg);
            }

            @Override
            public void sendFailCallBack(String topic, String msg, Throwable ex) {
                log.error("----车辆抓拍kafka记录解析完成放入topic:{},发送失败{}",topic,msg,ex);
            }
        });

实现方法里处理完成后生产到kafka就是这样优雅的调用,回调函数里我这里只打印了topic,没有做其他处理,有需要的就再这里放飞自我。
七、关键配置文件与引包
我是springBoot2.3.0 + springCloud:Hoxton.SR4

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>
//配置文件
spring:
  kafka:
    bootstrap-servers: 127.0.0.1:9092
    consumer:
      auto-commit-interval: 1000
      enable-auto-commit: false
      group-id: test-consumer-group
      max-poll-records: 10
    producer:
      batch-size: 4096
      buffer-memory: 40960
      retries: 1

其他配置文件都就随意了。
使用就是这么优雅,我这个场景同事说更适合使用kafkaStream,下次分享。

©️2020 CSDN 皮肤主题: 编程工作室 设计师:CSDN官方博客 返回首页