目录
问题
线上ELK日志发现kafka消费者消费到重复消息
排查原因
生产者原因
由于生产方本身就发送了重复的消息,导致消费到重复消息
消费者可能原因
消费方采用的是循环poll的模式,具体是在多线程分租户去批量处理的消息
while(true) {
ConsumerRecords consumerRecords;
do {
consumerRecords = this.consumer.poll(60000L);
} while(consumerRecords == null);
if (!consumerRecords.isEmpty()) {
long beginTime = System.currentTimeMillis();
Map<String, List> recordMap = this.consumerHandler.slice(consumerRecords);
List<ConsumerFutureDto> sliceFutureList = this.submitToExecutorService(recordMap);
this.checkReinstanceConsumerExecutorService(unDoneSliceFutures);
sliceFutureList.forEach((item) -> {
unDoneSliceFutures.add(item.getFuture());