The request included a message larger than the max message size the server will accept.

org.apache.flink.streaming.connectors.kafka.FlinkKafkaException: Failed to send data to Kafka: The request included a message larger than the max message size the server will accept.

    at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.checkErroneous(FlinkKafkaProducer.java:1196)

    at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:766)

    at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:98)

    at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.invoke(TwoPhaseCommitSinkFunction.java:228)

    at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56)

    at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164)

    at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.performDefaultAction(StreamTask.java:276)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:298)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:403)

    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)

    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)

    at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept.

 

解决方法：

修改kafka的 message.max.bytes参数

 

对于已创建的topic，调整max.message.bytes参数方法：

root@fdc01 bin]# /opt/cloudera/parcels/KAFKA/bin/kafka-configs --zookeeper fdc01:2181  --alter --entity-type topics --entity-name factorydata_json_topic --add-config max.message.bytes=1147483640