关键词:
kafka生产者的缓存机制
Kafka导致OOM
Kafka数据压缩
大量64M
堆外内存泄露
Docker使用内存不断升高直到重启即堆外内存泄露64M
kafka-producer-network-thread
1.背景
线上kafka异步系统一段时间会OOM,但是我们的测试环境部署在某平台容器虚拟机上,
2.排查过程
1.在测试环境能重现服务被打挂,以为重现线上问题,为了拿测试环境的dump文件,增加jvm启动参数:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/app/oom
以及deployment.yaml配置了卷挂载;
结果服务好不容易配置好起来了,能在平台检测到内存一直在飙升,把服务打挂,结果没拿到dump文件,以为配置问题,检查几遍没问题,怀疑不是OOM打挂服务,查询日志,证实服务是内存到达容器设置limit被系统killed了,并非OOM;
只能换思路先解决容器为什么内存飙升,就开始排查内存使用情况,先排查JVM内存情况,相关命令:top,jmap -heap pid ;jstat等等,看到JVM内存正常,怀疑堆外内存,开始排查堆外内存;刚开始以为是kafka开启了压缩导致的,检查参数并未开启压缩;
(kafka压缩相关知识:Kafka数据压缩_kafka中文社区的博客-CSDN博客_kafka 压缩)
转而排查堆外内存;
(堆外内存相关知识:Spring Boot引起的“堆外内存泄漏”排查及经验总结;记一次java native memory增长问题的排查 - Axb的自我修养)
pmap pid;查看堆外内存,发现大量64M的内存块,找了好久排查到是经典的64M问题;跟docker有关以及Java原生内存管理有关;解决:dockerfile增加
ENV MALLOC_ARENA_MAX=2
(相关知识:记录处理Docker使用内存不断升高直到重启即堆外内存泄露64M(已解决)_好好学习/天天向上的博客-CSDN博客_docker容器java内存不断上升;JAVA问题定位大杂汇之java 虚拟机内存占用超出 -Xmx设置_nature502的博客-CSDN博客;
Java原生内存管理(native memory management)(5)-MALLOC_ARENA_MAX - 知乎)
一顿操作下来发现64M问题解决了,但还是有大量的堆外内存,pmap命令查看发现还有大量148k跟12k的内存块,开始怀疑是kafka;手动导出dump文件,用visualVM分析看到大量的kafka相关线程kafka-producer-network-thread
怀疑跟kafka生产者有关,怀疑是kafka客户端缓存问题;
(kafka生产者客户端相关知识:kafka生产者的缓存机制_知食份子.的博客-CSDN博客_kafka 缓存)
开始检查代码;发现问题所在:
修改前代码
@Configuration
public class KafkaProducerConfig {
@Value("${spring.kafka.bootstrap-servers}")
private String servers;
@Value("${spring.kafka.producer.retries}")
private int retries;
@Value("${spring.kafka.producer.batch-size}")
private int batchSize;
@Value("${spring.kafka.producer.properties.linger.ms}")
private int linger;
@Value("${spring.kafka.producer.buffer-memory}")
private int bufferMemory;
@Value("${spring.kafka.producer.acks}")
private String acks;
@Value("${spring.kafka.producer.key-serializer}")
private String keySerializer;
@Value("${spring.kafka.producer.value-serializer}")
private String valueSerializer;
// @Value("${spring.kafka.producer.max.block.ms}")
// private int maxBlockMs;
public Map<String, Object> producerConfigs() {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, servers);
props.put(ProducerConfig.ACKS_CONFIG,acks);
props.put(ProducerConfig.RETRIES_CONFIG, retries);
props.put(ProducerConfig.BATCH_SIZE_CONFIG, batchSize);
props.put(ProducerConfig.LINGER_MS_CONFIG, linger);
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, bufferMemory);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
// props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG,4);
// props.put(ProducerConfig.MAX_BLOCK_MS_CONFIG,maxBlockMs);
props.put(ProducerConfig.MAX_BLOCK_MS_CONFIG,2500);
return props;
}
public KafkaFuture<Set<String>> topicList() {
Map<String, Object> stringObjectMap = this.producerConfigs();
ListTopicsResult result = KafkaAdminClient.create(stringObjectMap).listTopics();
KafkaFuture<Set<String>> names = result.names();
KafkaFuture<Set<String>> set = result.names();
return set;
}
public ProducerFactory<String, Object> producerFactory() {
return new DefaultKafkaProducerFactory<>(producerConfigs());
}
public KafkaTemplate<String, Object> kafkaTemplate() {
return new KafkaTemplate<String, Object>(producerFactory());
}
}
使用:
KafkaTemplate<String, Object> stringObjectKafkaTemplate = kafkaProducerConfig.kafkaTemplate();
ListenableFuture<SendResult<String, Object>> future = stringObjectKafkaTemplate.send(, );
问题出在KafkaProducerConfig类;使用的时候循环数据发送kafka每次都会生成新的KafkaTemplate,生成新的ProducerRecord,KafkaProducer有消息累加器(RecordAccumulator);这里提贴上doSend源码
protected ListenableFuture<SendResult<K, V>> doSend(ProducerRecord<K, V> producerRecord) {
if (this.transactional) {
Assert.state(this.inTransaction(), "No transaction is in process; possible solutions: run the template operation within the scope of a template.executeInTransaction() operation, start a transaction with @Transactional before invoking the template method, run in a transaction started by a listener container when consuming a record");
}
Producer<K, V> producer = this.getTheProducer();
this.logger.trace(() -> {
return "Sending: " + producerRecord;
});
SettableListenableFuture<SendResult<K, V>> future = new SettableListenableFuture();
producer.send(producerRecord, this.buildCallback(producerRecord, producer, future));
if (this.autoFlush) {
this.flush();
}
this.logger.trace(() -> {
return "Sent: " + producerRecord;
});
return future;
}
getTheProducer方法会生成新的Producer;
修改代码,用单例模式生成KafkaTemplate;
思考优化增加吞吐量如下:
kafka producer实例池实现生产者多线程写入_张老七没脾气的博客-CSDN博客_kafka producer 多线程