kafka报错nested exception is org.apache.kafka.common.errors.RecordTooLargeException

最新推荐文章于 2024-03-14 14:21:14 发布
最新推荐文章于 2024-03-14 14:21:14 发布
阅读量5.3w 收藏 9
点赞数 7
文章标签： kafka 分布式
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_38814164/article/details/135206769
版权

前言：之前使用的是feign请求单独的kafka服务，因为项目需要申请防火墙策略，所以直接使用KafkaTemplate，上周五刚改，今天就报错了。

错误信息：

Caused by: org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'public void com.njwx.yq.business.kafka.listener.KafkaTaskListener.listenerTask(org.apache.kafka.clients.consumer.ConsumerRecord<java.lang.Object, java.lang.String>,org.springframework.kafka.support.Acknowledgment)' threw exception; nested exception is org.springframework.kafka.KafkaException: Send failed; nested exception is org.apache.kafka.common.errors.RecordTooLargeException: The message is 1270600 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.; nested exception is org.springframework.kafka.KafkaException: Send failed; nested exception is org.apache.kafka.common.errors.RecordTooLargeException: The message is 1270600 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.decorateException(KafkaMessageListenerContainer.java:2084)
	... 10 common frames omitted
Caused by: org.springframework.kafka.KafkaException: Send failed; nested exception is org.apache.kafka.common.errors.RecordTooLargeException: The message is 1270600 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.
	at org.springframework.kafka.core.KafkaTemplate.doSend(KafkaTemplate.java:573)
	at org.springframework.kafka.core.KafkaTemplate.send(KafkaTemplate.java:388)
	at com.njwx.yq.business.kafka.service.KafkaService.send(KafkaService.java:26)
	at com.njwx.yq.business.kafka.controller.KafkaMessageController.sendMessage(KafkaMessageController.java:37)
	at com.njwx.yq.business.kafka.controller.KafkaMessageController$$FastClassBySpringCGLIB$$873a3c4b.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
	at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:62)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
	at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:55)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
	at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
	at com.njwx.yq.business.kafka.controller.KafkaMessageController$$EnhancerBySpringCGLIB$$6600c573.sendMessage(<generated>)
	at com.njwx.yq.business.kafka.service.impl.TaskQueueServiceImpl.sendAppMessage(TaskQueueServiceImpl.java:388)
	at com.njwx.yq.business.kafka.service.impl.TaskQueueServiceImpl.saveBatchTaskReceiver(TaskQueueServiceImpl.java:351)
	at com.njwx.yq.business.kafka.service.impl.TaskQueueServiceImpl.createTaskReceiver(TaskQueueServiceImpl.java:246)
	at com.njwx.yq.business.kafka.service.impl.TaskQueueServiceImpl.taskManager(TaskQueueServiceImpl.java:98)
	at com.njwx.yq.business.kafka.service.impl.TaskQueueServiceImpl$$FastClassBySpringCGLIB$$9735e29a.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
	at com.njwx.yq.business.kafka.service.impl.TaskQueueServiceImpl$$EnhancerBySpringCGLIB$$3af4caba.taskManager(<generated>)
	at com.njwx.yq.business.kafka.listener.KafkaTaskListener.listenerTask(KafkaTaskListener.java:29)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:171)
	at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:120)
	at org.springframework.kafka.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:48)
	at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:329)
	at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:86)
	at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:51)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:2039)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2021)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:1958)
	... 8 common frames omitted
提示：The message is 1270600 bytes when serialized which is larger than 1048576，原因是因为生产者发送到Kafka的单个请求中的所有消息总大小默认的max.request.size大小是1M，所以需要在yml里面配置producer.properties.max.request.size

  producer:
    properties:
        max.request.size: 10240000
有需要可以加上
batch-size:控制默认批量大小（以字节为单位）
buffer-memory:缓冲等待发送到服务器的记录的内存总字节数

你当像山
关注
  • 7
    点赞
  • 踩
  • 9
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
kafka处理超大消息的配置 org.apache.kafka.common.errors.RecordTooLargeException
01-07
在使用 canal 和kafka处理数据同步时canal日志提示如下异常： java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept   原因是消息过大无法处理 第一步：修改 canal/conf/canal.properties配置
Spring Boot 集成 Kafka 遇到问题
weixin_45626288的博客
06-02 4302
问题1：nested exception is org.apache.kafka.common.errors.RecordTooLargeException: 异常类: nested exception is org.apache.kafka.common.errors.RecordTooLargeException: 很明显，消息体目前过大。需要提高Kafka服务器能够接收的最大数据大小。 解决方法：提高Kafka服务器单条消息最大大小。配置参数 message.max.bytes 我的方法：生产环境大概
Kafka重复消费、Dubbo重复调用问题排查
最新发布
qq_41382215的博客
03-14 2424
本业务为车机流量充值业务，大致流程为：收到微信、支付宝端用户支付成功回调后，将用户订单信息发送至kafka中；消费者接收到kafka中信息后进行解析，处理用户订单信息，为用户订购相关流量包（调用电信相关接口），订购成功/失败后会通过MQTT发送订购成功/失败消息至车机端，若订购失败则为用户退款。
org.apache.kafka kafka-clients 0.10.1.1 的jar包下载
11-03
org.apache.kafka kafka-clients 0.10.1.1
kafkajs：适用于node.js的现代Apache Kafka客户端
02-04
KafkaJS 适用于Node.js的现代ApacheKafka:registered:客户端 · · 目录 关于该项目 KafkaJS是用于Node.js的现代客户端。 它与Kafka 0.10+兼容，并提供对0.11功能的本地支持。 KAFKA是The Apache Software Foundation的注册商标，并已被KafkaJS许可使用。 KafkaJS与Apache软件基金会没有任何隶属关系，也不认可。 产品特点 制片人 具有暂停，恢复和寻找的消费者群体 对生产者和消费者的交易支持 邮件标题 GZIP压缩通过可插入编解码器进行Snappy，LZ4和ZSTD压缩 普通SSL和SASL_SSL实现 支
SpringbootKaka批量消费报Listener method could not be invoked with the incoming message
SDAUguo的博客
12-03 7822
结论：在配置文件中加上spring.kafka.listener.type=batch 。 完整报错信息如下： org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message Endpoint handler details: Method [public void com.kafka.gateway.c
springboot kafka手动设置ack报错
carterslam的专栏
04-14 4319
kafka手动设置ack时报以下错误： org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message Endpoint handler details: Method [public void com.roomdis.micros.kafka.consumer.MessageConsumer.onMess.
线上问题-kafka生产者发送消息总是失败
DRD
08-26 1万+
问题解决过程 线上一个界面发现老是没有数据，排查下来时生产者没有成功发送消息所致，报错如下： org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 0 ms. 我们发现配置里有一个很奇怪的参数 config.p
Kafka - which is larger than the maximum request size you have configured with the max.request
牧码的博客
06-13 1005
which is larger than the maximum request size you have configured with the max.request
kafka提交偏移量失败导致重复消费
lck_csdn的博客
09-06 1万+
问题详情： org.springframework.kafka.KafkaException: Seek to current after exception; nested exception is org.apache.kafka.clients.consumer.CommitFailedException: Offset commit cannot be completed since the consumer is not part of an active group for auto parti
org.apache.kafka kafka_2.10 0.10.2.0 的jar包下载
11-03
依赖方式     org.apache.kafka     kafka_2.10     0.10.0.0 但是没有被中央仓库的任何jar包依赖！您可以在这里点击下载
kafka-2.10-0.8.1.1
12-22
kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8.1.1kafka_2.10-0.8
org.apache.kafka kafka-log4j-appender 的jar包下载
11-03
<groupId>org.apache.kafka <artifactId>kafka-log4j-appender <version>0.10.2.0 </dependency>
Kafka:exception is org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not
有才而性缓，定属大才。有智而气和，斯为大智。
07-24 3011
错误： Failed to send; nested exception is org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. 找不到推送消息服务器的所在分区，那估计就是kafka挂掉了 1.kafka的几个概念： producer：生产者。 consumer：消费者。 topic: 消息以topic为类别记录,
kafka发送消息失败，消息过大
H785503444的博客
10-27 4440
报错如下：org.apache.kafka.common.errors.RecordTooLargeException: The message is 46860 bytes when serialized which is larger than the total memory buffer you have configured with the buffer.memory configuration 生产者的bufferMemory（缓冲大小）设置过小原因 ...
【kafka异常】使用Spring-kafka遇到的坑
热门推荐
AK774S的博客
04-29 1万+
查看一下压缩策略 bin/kafka-topics.sh --describe --zookeeper xxxx:2181 --topic SHI_TOPIC1 Topic:SHI_TOPIC1 PartitionCount:1 ReplicationFactor:1 Configs:cleanup.policy=compact Topic: SHI_TOPIC1 Partition: 0 Leader: 0 Replicas: 0 Isr: 0 Configs:cleanup.policy=compact
Kafka报错org.apache.kafka.common.errors.RecordTooLargeException
蜗牛^_^的博客
01-15 4121
Kafka报错： org.apache.kafka.common.errors.RecordTooLargeException: There are some messages at [Partition=Offset]: {testTopic-0=25537140} whose size is larger than the fetch size 1048576 and hence canno...
springboot使用kafka发送消息，消息过大报错
ab3344556677的博客
10-27 6145
报错信息如下所示： Caused by: org.apache.kafka.common.errors.RecordTooLargeException: The message is 1527150 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration. 从日志可以看出，kafka默认一次接收请求消息的最大量为1M,即1048576字节，从这我们
nested exception is org.apache.kafka.common.KafkaException: Failed create
06-09
这个错误的意思是Kafka创建失败了，可能有多种原因导致这个错误，比如Kafka的配置不正确、Kafka的版本不兼容、Kafka的依赖库缺失等等。你需要检查相关的日志和配置文件，查找具体的错误信息，并逐一排除各种可能的问题，直到解决这个错误为止。如果你无法解决这个问题，建议你寻求一些专业的Kafka技术支持或者咨询相关的社区论坛。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值