Spark Shuffle过程报错org.apache.spark.shuffle.MetadataFetchFailedException

具体错误提示：

Most recent failure reason: **org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 18** at org.apache.spark.MapOutputTracker$$anonfun$convertMapStatuses$2.apply(MapOutputTracker.scala:882) at org.apache.spark.MapOutputTracker$$anonfun$convertMapStatuses$2.apply(MapOutputTracker.scala:878) at scala.collection.Iterator$class.foreach(Iterator.scala:891) at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at org.apache.spark.MapOutputTracker$.convertMapStatuses(MapOutputTracker.scala:878) at org.apache.spark.MapOutputTrackerWorker.getMapSizesByExecutorId(MapOutputTracker.scala:691) at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49) at org.apache.spark.sql.execution.ShuffledRowRDD.compute(ShuffledRowRDD.scala:165)

解决方案：
调大executor memory,
例如 .set(“spark.executor.memory”, “10g”)

 spark_conf = SparkConf().setAppName(
        job_name).set(
        "spark.sql.sources.partitionOverwriteMode", "dynamic"
    ).set(
        "fs.s3a.access.key", os.getenv('AWS_ACCESS_KEY_ID')
    ).set(
        "fs.s3a.secret.key", os.getenv('AWS_SECRET_ACCESS_KEY')
    ).set(
        "fs.s3a.endpoint", "xxxxxxx.amazonaws.com"
    ).set(
        "fs.s3a.block.size", "134217728"
    ).set("spark.executor.memory", "10g")

也有文章提到考虑是否有数据倾斜，做repartition，暂无此种尝试。