一:异常信息
21/09/23 10:39:46 ERROR internals.ErrorLoggingCallback: Error when sending message to topic ad_source_mob_prtsc with key: null, value: 5242233 bytes with error:
org.apache.kafka.common.errors.RecordTooLargeException: The message is 5242321 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.
>21/09/23 10:39:46 ERROR internals.ErrorLoggingCallback: Error when sending message to topic ad_source_mob_prtsc with key: null, value: 6376666 bytes with error:
org.apache.kafka.common.errors.RecordTooLargeException: The message is 6376754 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.
二:处理办法
都知道是配置***max***size的配置项,但细节往往容易忽略:
- server、producer、consumer都需要配置
- server中有两个,其中【message.max.bytes】是个临界,设为aMb,后面的配置需要根据临界调整
- producer中的【max.request.size】要小于等于临界a
- consumer中的【fetch.message.max.bytes】要大于等于临界a
- server中还有另一个【replica.fetch.max.bytes】,这个是每个分区尝试获取的字节数
总结一下就是小心使得万年船。
对了,最后还要记得重启kafka才能生效。