在本机编写了 Hadoop的 Driver、Mapper、Reducer后,运行时报错:
java.lang.Exception: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#1
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#1
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.FileNotFoundException: File D:/tmp/hadoop-Mr.%20Y/mapred/local/localRunner/Mr.%20Y/jobcache/job_local1029102278_0001/attempt_local1029102278_0001_m_000000_0/output/file.out.index does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:211)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
at org.apache.hadoop.io.SecureIOUtils.openFSDataInputStream(SecureIOUtils.java:152)
at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:71)
at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:62)
。。。。。。。。
=========================================================================
原因:
core-site.xml中有这样一个定义:
hadoop.tmp.dir=/tmp/hadoop-${usr}
即运行的时候,会自动在D盘(?)下生成一个tmp目录,再生成一个 hadoop-${usr}文件夹。
当windows电脑用户名中包含了非法字符(如 Mr.Y包含了 . )就会导致文件路径解析失败,进而报错
解决:
最简单的解决方式就是找到 core-site.xml配置文件,找到hadoop.tmp.dir修改其属性值。
参考文章:
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError - Tekzak.com