spark解决org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow

最新推荐文章于 2024-05-07 14:02:37 发布

灵佑666

最新推荐文章于 2024-05-07 14:02:37 发布

阅读量857

点赞数

分类专栏： Spark

本文链接：https://blog.csdn.net/onway_goahead/article/details/107860657

版权

在使用pyspark和Spark SQL的Thrift JDBC接口查询数据时遇到`org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow`错误。尝试通过增加`spark.kryoserializer.buffer.max`等配置参数来解决问题。首先尝试在`spark-defaults.conf`文件中配置，但未成功，因为Spark SQL Thrift JDBC不受此文件影响。然后，通过`spark-shell`测试确认配置未生效。最终，通过在启动命令行中添加配置参数`--conf spark.kryoserializer.buffer.max=3236381`等解决了问题。

摘要由CSDN通过智能技术生成

pyspark --queue default\
--driver-memory 10G \
--executor-cores 6 \
--executor-memory 10G \
--executor-cores 6 \
--conf spark.kryoserializer.buffer.max=256m \
--conf spark.kryoserializer.buffer=64m \
--conf spark.driver.maxResultSize=4096m \
--conf spark.executor.memoryOverhead=2048m \
--conf spark.driver.memoryOverhead=2048m

使用spark sql的thrift jdbc接口查询数据时报这个错误

Exception in thread "main" java.sql.SQLException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3107 in stage 308.0 failed 4 times, most recent failure: Lost task 3107.3 in stage 308.0 (TID 620318, XXX): org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 1572864, required: 3236381
Serialization trace:
values (org.apache.spark.sql.catalyst.expressions.GenericInternalRow). To avoid this, increase spark.kryoserializer.buffer.max value.
        at org.ap