解决MapReduce编程过程中的 java.io.FileNotFoundException 问题

最新推荐文章于 2024-06-22 09:39:54 发布

桂小林

最新推荐文章于 2024-06-22 09:39:54 发布

阅读量2.8k

点赞数 3

分类专栏： Hadoop 文章标签： mapreduce 编程

本文链接：https://blog.csdn.net/quintind/article/details/77983426

版权

Hadoop 专栏收录该内容

9 篇文章 0 订阅

订阅专栏

在运行WordCount程序的过程中出现了如下错误：

DEBUG - LocalFetcher 1 going to fetch: attempt_local938878567_0001_m_000000_0
 WARN - job_local938878567_0001
java.lang.Exception: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#1
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#1
    at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.io.FileNotFoundException: D:/tmp/hadoop-Gui%20Xiaolin/mapred/local/localRunner/Gui%20Xiaolin/jobcache/job_local938878567_0001/attempt_local938878567_0001_m_000000_0/output/file.out.index
    at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:198)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
    at org.apache.hadoop.io.SecureIOUtils.openFSDataInputStream(SecureIOUtils.java:156)
    at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:70)
    at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:62)
    at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:57)
    at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:124)
    at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:102)
    at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:85)
DEBUG - LocalFetcher 1 going to fetch: attempt_local938878567_0001_m_000000_0
DEBUG - LocalFetcher 1 going to fetch: attempt_local938878567_0001_m_000000_0

提示内容说的是找不到如下目录下的一个文件，经过多番周折才获知Hadoop运行过程中的文件路径中不能出现空格键。由于/tmp/hadoop-Gui%20Xiaolin……是根据电脑中的当前用户名产生的，因此只需要将用户名修改成不含空格即可。

D:/tmp/hadoop-Gui%20Xiaolin/mapred/local/localRunner/Gui%20Xiaolin/jobcache/job_local938878567_0001/attempt_local938878567_0001_m_000000_0/output/file.out.index