直接报错 spark Kryo serialization failed: Buffer overflow 错误
Caused by: org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 28. To avoid this, increase spark.kryoserializer.buffer.max value
提示需要调整的参数是 spark.kryoserializer.buffer.max 最少是28
直接在 submit时候配置算了
--conf 'spark.kryoserializer.buffer.max=64'
有时候设置
sparkConf.set("spark.kryoserializer.buffer","64");