fetch failed:_Apache Spark中的Fetch Failed Exception：解密最常见的原因

fetch failed:

火花执行指南 (GUIDE TO SPARK EXECUTION)

Shuffle operations are the backbone of almost all Spark Jobs that are aimed at data aggregation, joins, or data restructuring. During a shuffle operation, the data is shuffled across various nodes of the cluster via a two-step process:

随机操作是几乎所有针对数据聚合，联接或数据重组的Spark作业的骨干。 在混洗操作期间，通过两步过程在集群的各个节点之间对数据进行混洗：

a) Shuffle Write: Shuffle map tasks write the data to be shuffled in a disk file, the data is arranged in the file according to shuffle reduce tasks. Bunch of shuffle data corresponding to a shuffle reduce task written by a shuffle map task is called a shuffle block. Further, each of the shuffle map tasks informs the driver about the written shuffle data.

a)随机写入：随机映射任务将要随机播放的数据写入磁盘文件中，数据按照随机减少任务排列在文件中。 与由混洗映射任务写入的混洗减少任务相对应的混洗数据束称为混洗块。 此外，每个混洗图任务将写入的混洗数据通知驾驶员。

b) Shuffle Read: Shuffle reduce tasks queries the driver about the locations of their shuffle blocks. Then these tasks establish connections with the executors hosting their shuffle blocks and start fetching the required shuffle blocks. Once a block is fetched, it is available for further computation in the reduce task.

b)随机读取：随机压缩任务向驱动程序查询随机块的位置。 然后，这些任务将与托管其shuffle块的执行程序建立连接，并开始获取所需的shuffle块。 一旦获取了块，就可以在reduce任务中进行进一步的计算。

To know more about the shuffle process, you can refer to my earlier story titled: Revealing Apache Spark Shuffling Magic.

要了解有关改组过程的更多信息，可以参考我以前的文章： 揭示Apache Spark改组魔术 。

The two-step process of a shuffle although sounds simple, but is operationally intensive as it involves data sorting, disk writes/reads, and network transfers. Therefore, there is always a question mark on the reliability of a shuffle operation, and the evidence of this unreliability is the commonly encountered ‘FetchFailed Exception’ during the shuffle operation. Most Spark developers spend considerable time in troubleshooting this widely encountered exception. First, they try to find out the root cause of the exception, and then accordingly put the right fix for the same.

洗牌的两步过程虽然听起来很简单，但由于涉及数据排序，磁盘写入/读取和网络传输，因此操作很密集。 因此，在混洗操作的可靠性方面始终存在一个问号，这种不可靠性的证据是在混洗操作过程中经常遇到的“ FetchFailed Exception”。 大多数Spark开发人员都花费大量时间来对这种广泛遇到的异常进行故障排除。 首先，他们试图找出异常的根本原因，然后相应地针对该异常进行正确的修