window10下hadoop3.1.3 Shuffle$ShuffleError: error in shuffle in localfetcher#1 原因及解决方法

问题

Windows10 下用idea本地调试wordcount产生以下错误:

2021-03-05 11:11:41,311 WARN [org.apache.hadoop.mapred.LocalJobRunner] - job_local1291814541_0001
java.lang.Exception: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#1
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#1
	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.FileNotFoundException: File C:/tmp/hadoop-Tomas%20Wen/mapred/local/localRunner/Tomas%20Wen/jobcache/job_local1291814541_0001/attempt_local1291814541_0001_m_000000_0/output/file.out.index does not exist

各种搜索后发现该错误是因为Windows用户名中带空格,导致map完后储存时因为tmp路径带空格无法存入导致reducer在读取的时filenotfound

解决方法

方法一

在控制面板 -> user Account -> change account name;

方法二

我的电脑 右键 管理 -> 本地用户和组 -> 用户 -> 选中要改的用户右键->重命名

打开cmd,虽然路径显示还是带空格但是whoami后就会发现已经重命名
在这里插入图片描述
再重新run wordcount之后成功

已标记关键词 清除标记
相关推荐
©️2020 CSDN 皮肤主题: 大白 设计师:CSDN官方博客 返回首页