Spark报错处理系列之:Kryo serialization failed: Buffer overflow. Available: 0, required: 12678. To avoid this, increase spark.kryoserializer.buffer.max value.
一、完整报错
- org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 12678. To avoid this, increase spark.kryoserializer.buffer.max value.
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:350)
at org.apache.spark.executor.Executor T a s k R u n n e r . r u n ( E x e c u t o r . s c a l a : 455 ) a t j a v a . u t i l . c o n c u r r e n t . T h r e a d P o o l E x e c u t o r . r u n W o r k e r ( T h r e a d P o o l E x e c u t o r . j a v a : 1149 ) a t j a v a . u t i l . c o n c u r r e n t . T h r e a d P o o l E x e c u t o r TaskRunner.run(Executor.scala:455) at java.util.concurrent
本文介绍了Spark运行时遇到的Kryo序列化异常,详细解析了报错原因及如何通过调整`spark.kryoserializer.buffer.max`参数来解决问题,提供了设置参数的方法和官方参考资料链接。
订阅专栏 解锁全文
558

被折叠的 条评论
为什么被折叠?



