博客探讨了Spark处理大规模数据时遇到的Shuffle FetchFailedException问题。文章分析了shuffle操作的原因，指出shuffle是最耗时的操作，应尽量避免。内容详细讲解了shuffle write和shuffle read的过程，以及内存分配策略，特别是ShuffleMemoryManager如何动态调整内存分配以避免溢出。
spark大规模数据处理中，有个比较常见的错误：
org.apache.spark.shuffle.MetadataFetchFailedException:
Missing an output location for shuffle 0
ERROR shuffle.RetryingBlockFetcher: Failed to fetch block shuffle_0_1300_106, and will not retry (0 retries)
java.lang.RuntimeException: java.lang.RuntimeException: Failed to open file: /data03/yarn/nm/usercache/vendorszry/appcache/application_1533223718021_249359/blockmgr-980fe20e-f8ef-4646-a01c-2e55bb07b333/01/shuffle_0_343_0.index at org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.getSortBasedShuffleBlockData(ExternalShuffleBlockResolver.java:243)
.............
..............
................
Caused by: java.io.FileNotFoundException: /data03/yarn/nm/usercache/vendorszry/appcache/application_1533223718021_249359/blockmgr-980fe20e-f8ef-4646-a01c-2e55bb07b333/01/shuffle_0_343_0.index (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileI
最低0.47元/天 解锁文章
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
