2021-01-14

最新推荐文章于 2024-05-07 14:02:37 发布

zhengbeijing

最新推荐文章于 2024-05-07 14:02:37 发布

阅读量229

点赞数

文章标签： spark

本文链接：https://blog.csdn.net/zhengbeijing/article/details/112582915

版权

spark2.3 Kryo serialization failed: Buffer overflow. Available: 0, required: 182950

Kryo serialization failed: Buffer overflow. Available: 0, required: 182950

spark版本2.3
kryo序列化缓冲区溢出任务执行失败，调整缓冲区：spark.kryoserializer.buffer.max=256
可以先调整128，不行再逐渐调整

报错日志如下：

Task 1 in stage 6.0 failed 1 times; aborting job
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 6.0 failed 1 times, most recent failure: Lost task 1.0 in stage 6.0 (TID 28, localhost, executor driver): org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 182950
Serialization trace:
values (org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema). To avoid this, increase spark.kryoserializer.buffer.max value.
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:350)
at org.apache.spark.executor.Executor $T a s k R u n n e r . r u n (E x e c u t o r . s c a l a : 393) a t j a v a . u t i l . c o n c u r r e n t . T h r e a d P o o l E x e c u t o r . r u n W o r k e r (T h r e a d P o o l E x e c u t o r . j a v a : 1149) a t j a v a . u t i l . c o n c u r r e n t . T h r e a d P o o l E x e c u t o r$ Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0, required: 182950
Serialization trace:
values (org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema)
at com.esotericsoftware.kryo.io.Output.require(Output.java:163)
at com.esotericsoftware.kryo.io.Output.writeString_slow(Output.java:462)
at com.esotericsoftware.kryo.io.Output.writeString(Output.java:363)
at com.esotericsoftware.kryo.serializers.DefaultSerializers $S t r i n g S e r i a l i z e r . w r i t e (D e f a u l t S e r i a l i z e r s . j a v a : 191) a t c o m . e s o t e r i c s o f t w a r e . k r y o . s e r i a l i z e r s . D e f a u l t S e r i a l i z e r s$ StringSerializer.write(DefaultSerializers.java:184)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers $O b j e c t A r r a y S e r i a l i z e r . w r i t e (D e f a u l t A r r a y S e r i a l i z e r s . j a v a : 366) a t c o m . e s o t e r i c s o f t w a r e . k r y o . s e r i a l i z e r s . D e f a u l t A r r a y S e r i a l i z e r s$ ObjectArraySerializer.write(DefaultArraySerializers.java:307)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers $O b j e c t A r r a y S e r i a l i z e r . w r i t e (D e f a u l t A r r a y S e r i a l i z e r s . j a v a : 366) a t c o m . e s o t e r i c s o f t w a r e . k r y o . s e r i a l i z e r s . D e f a u l t A r r a y S e r i a l i z e r s$ ObjectArraySerializer.write(DefaultArraySerializers.java:307)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:347)
… 4 more

zhengbeijing

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
2021-01-14

Kryo serialization failed: Buffer overflow. Available: 0, required: 182950(这里写自定义目录标题)Kryo serialization failed: Buffer overflow. Available: 0, required: 182950Caused by: org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Availa
复制链接

扫一扫