Spark Shuffle FetchFailedException 内存溢出源码级分析解决

最新推荐文章于 2023-05-18 15:21:24 发布

JaxMa

最新推荐文章于 2023-05-18 15:21:24 发布

阅读量4.3k

点赞数 7

分类专栏： Spark Trouble Shooting 文章标签： spark hadoop 大数据 hive hdfs

本文链接：https://blog.csdn.net/Jaxma/article/details/106827482

版权

本文分析了Spark Shuffle阶段遇到的FetchFailedException内存溢出问题。增大shuffle partitions未能解决问题，反而引发错误。通过源码研究，发现当shuffle partitions超过2000时，Spark使用HighlyCompressedMapStatus，导致内存问题。调整`spark.shuffle.minNumPartitionsToHighlyCompress`、`spark.shuffle.accurateBlockThreshold`和`spark.maxRemoteBlockSizeFetchToMem`参数可优化内存使用。

摘要由CSDN通过智能技术生成

某日遇到一个数据倾斜的SQL, 首先想到的方法就是加大Partition 看看数据hash 之后会不会落得均匀,所以就将spark.sql.shuffle.partitions从原来的500 加大到2700 .
结果反而失败了, 错误如下:

FetchFailed(BlockManagerId(516, nfjd-hadoop02-node352.jpushoa.com, 7337, None), shuffleId=3, mapId=59, reduceId=917, message=
org.apache.spark.shuffle.FetchFailedException: failed to allocate 16777216 byte(s) of direct memory (used: 2147483648, max: 2147483648)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:554)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:485)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:64)
	at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage8.sort_addToSorter_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage8.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(