hadoop执行mapreduce任务，能够map，不能reduce，Shuffle阶段报错

最新推荐文章于 2021-06-21 15:14:34 发布
最新推荐文章于 2021-06-21 15:14:34 发布
阅读量7k 收藏 1
点赞数 1
分类专栏： hadoop 文章标签： hadoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/tangzwgo/article/details/25894427
版权
第一次运行：

[root@sjfx jar]# hadoop jar /home/tangzw/jar/GameLoginLogAnalyzeA.jar /tangzw/input /tangzw/output
08/01/25 07:13:43 INFO client.RMProxy: Connecting to ResourceManager at sjfx/192.168.57.127:18040
08/01/25 07:13:43 INFO client.RMProxy: Connecting to ResourceManager at sjfx/192.168.57.127:18040
08/01/25 07:13:43 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
08/01/25 07:13:44 INFO mapred.FileInputFormat: Total input paths to process : 13
08/01/25 07:13:44 INFO mapreduce.JobSubmitter: number of splits:103
08/01/25 07:13:44 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
08/01/25 07:13:44 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
08/01/25 07:13:44 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
08/01/25 07:13:44 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
08/01/25 07:13:44 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
08/01/25 07:13:44 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
08/01/25 07:13:44 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
08/01/25 07:13:44 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
08/01/25 07:13:44 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
08/01/25 07:13:45 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1201209004442_0001
08/01/25 07:13:46 INFO impl.YarnClientImpl: Submitted application application_1201209004442_0001 to ResourceManager at sjfx/192.168.57.127:18040
08/01/25 07:13:46 INFO mapreduce.Job: The url to track the job: http://sjfx:18088/proxy/application_1201209004442_0001/
08/01/25 07:13:46 INFO mapreduce.Job: Running job: job_1201209004442_0001
08/01/25 07:13:57 INFO mapreduce.Job: Job job_1201209004442_0001 running in uber mode : false
08/01/25 07:13:57 INFO mapreduce.Job:  map 0% reduce 0%
08/01/25 07:14:36 INFO mapreduce.Job:  map 2% reduce 0%
08/01/25 07:14:46 INFO mapreduce.Job:  map 3% reduce 0%
08/01/25 07:14:51 INFO mapreduce.Job:  map 4% reduce 0%
08/01/25 07:14:54 INFO mapreduce.Job:  map 5% reduce 0%
08/01/25 07:14:57 INFO mapreduce.Job:  map 6% reduce 0%
08/01/25 07:15:04 INFO mapreduce.Job:  map 7% reduce 0%
08/01/25 07:15:16 INFO mapreduce.Job:  map 8% reduce 0%
08/01/25 07:15:22 INFO mapreduce.Job:  map 9% reduce 0%
08/01/25 07:17:01 INFO mapreduce.Job:  map 10% reduce 0%
08/01/25 07:17:56 INFO mapreduce.Job:  map 11% reduce 0%
08/01/25 07:18:38 INFO mapreduce.Job:  map 12% reduce 0%
08/01/25 07:19:11 INFO mapreduce.Job:  map 13% reduce 0%
08/01/25 07:19:20 INFO mapreduce.Job:  map 14% reduce 0%
08/01/25 07:19:34 INFO mapreduce.Job:  map 15% reduce 0%
08/01/25 07:20:37 INFO mapreduce.Job:  map 16% reduce 0%
08/01/25 07:20:52 INFO mapreduce.Job:  map 17% reduce 0%
08/01/25 07:21:39 INFO mapreduce.Job:  map 18% reduce 0%
08/01/25 07:22:06 INFO mapreduce.Job:  map 19% reduce 0%
08/01/25 07:23:18 INFO mapreduce.Job:  map 20% reduce 0%
08/01/25 07:23:38 INFO mapreduce.Job:  map 21% reduce 0%
08/01/25 07:24:24 INFO mapreduce.Job:  map 24% reduce 0%
08/01/25 07:24:30 INFO mapreduce.Job:  map 25% reduce 0%
08/01/25 07:25:07 INFO mapreduce.Job:  map 26% reduce 0%
08/01/25 07:25:35 INFO mapreduce.Job:  map 27% reduce 0%
08/01/25 07:26:10 INFO mapreduce.Job:  map 28% reduce 0%
08/01/25 07:26:19 INFO mapreduce.Job:  map 29% reduce 0%
08/01/25 07:26:35 INFO mapreduce.Job:  map 29% reduce 1%
08/01/25 07:27:13 INFO mapreduce.Job:  map 30% reduce 1%
08/01/25 07:27:24 INFO mapreduce.Job:  map 31% reduce 1%
08/01/25 07:28:42 INFO mapreduce.Job:  map 32% reduce 1%
08/01/25 07:28:50 INFO mapreduce.Job:  map 33% reduce 1%
08/01/25 07:28:56 INFO mapreduce.Job:  map 34% reduce 1%
08/01/25 07:29:02 INFO mapreduce.Job:  map 35% reduce 1%
08/01/25 07:29:05 INFO mapreduce.Job:  map 36% reduce 1%
08/01/25 07:30:03 INFO mapreduce.Job:  map 37% reduce 1%
08/01/25 07:30:31 INFO mapreduce.Job:  map 38% reduce 1%
08/01/25 07:30:56 INFO mapreduce.Job:  map 39% reduce 1%
08/01/25 07:32:16 INFO mapreduce.Job:  map 40% reduce 1%
08/01/25 07:33:15 INFO mapreduce.Job:  map 41% reduce 1%
08/01/25 07:33:27 INFO mapreduce.Job:  map 42% reduce 1%
08/01/25 07:33:31 INFO mapreduce.Job:  map 43% reduce 1%
08/01/25 07:33:36 INFO mapreduce.Job:  map 44% reduce 1%
08/01/25 07:33:40 INFO mapreduce.Job:  map 45% reduce 1%
08/01/25 07:34:26 INFO mapreduce.Job:  map 46% reduce 1%
08/01/25 07:35:30 INFO mapreduce.Job:  map 46% reduce 2%
08/01/25 07:35:43 INFO mapreduce.Job:  map 47% reduce 2%
08/01/25 07:35:50 INFO mapreduce.Job:  map 48% reduce 2%
08/01/25 07:36:04 INFO mapreduce.Job:  map 49% reduce 2%
08/01/25 07:37:31 INFO mapreduce.Job:  map 50% reduce 2%
08/01/25 07:38:33 INFO mapreduce.Job:  map 51% reduce 2%
08/01/25 07:39:26 INFO mapreduce.Job:  map 52% reduce 2%
08/01/25 07:39:29 INFO mapreduce.Job:  map 53% reduce 2%
08/01/25 07:39:35 INFO mapreduce.Job:  map 54% reduce 2%
08/01/25 07:39:38 INFO mapreduce.Job:  map 55% reduce 2%
08/01/25 07:40:25 INFO mapreduce.Job:  map 56% reduce 2%
08/01/25 07:40:58 INFO mapreduce.Job:  map 56% reduce 3%
08/01/25 07:41:25 INFO mapreduce.Job:  map 57% reduce 3%
08/01/25 07:42:17 INFO mapreduce.Job:  map 58% reduce 3%
08/01/25 07:43:12 INFO mapreduce.Job:  map 59% reduce 3%
08/01/25 07:43:18 INFO mapreduce.Job:  map 60% reduce 3%
08/01/25 07:43:42 INFO mapreduce.Job:  map 61% reduce 3%
08/01/25 07:44:19 INFO mapreduce.Job:  map 62% reduce 3%
08/01/25 07:45:12 INFO mapreduce.Job:  map 63% reduce 3%
08/01/25 07:46:01 INFO mapreduce.Job:  map 64% reduce 3%
08/01/25 07:46:33 INFO mapreduce.Job:  map 64% reduce 4%
08/01/25 07:46:36 INFO mapreduce.Job:  map 65% reduce 4%
08/01/25 07:47:02 INFO mapreduce.Job:  map 66% reduce 4%
08/01/25 07:47:08 INFO mapreduce.Job:  map 67% reduce 4%
08/01/25 07:47:30 INFO mapreduce.Job:  map 68% reduce 4%
08/01/25 07:47:36 INFO mapreduce.Job:  map 69% reduce 4%
08/01/25 07:47:42 INFO mapreduce.Job:  map 70% reduce 4%
08/01/25 07:47:51 INFO mapreduce.Job:  map 71% reduce 4%
08/01/25 07:48:13 INFO mapreduce.Job:  map 72% reduce 4%
08/01/25 07:48:19 INFO mapreduce.Job:  map 73% reduce 4%
08/01/25 07:48:40 INFO mapreduce.Job:  map 74% reduce 4%
08/01/25 07:48:53 INFO mapreduce.Job:  map 75% reduce 4%
08/01/25 07:49:15 INFO mapreduce.Job:  map 76% reduce 4%
08/01/25 07:49:55 INFO mapreduce.Job:  map 77% reduce 4%
08/01/25 07:51:06 INFO mapreduce.Job:  map 78% reduce 4%
08/01/25 07:51:48 INFO mapreduce.Job:  map 78% reduce 5%
08/01/25 07:51:54 INFO mapreduce.Job:  map 79% reduce 5%
08/01/25 07:52:06 INFO mapreduce.Job:  map 80% reduce 5%
08/01/25 07:52:10 INFO mapreduce.Job:  map 81% reduce 5%
08/01/25 07:52:19 INFO mapreduce.Job:  map 82% reduce 5%
08/01/25 07:52:25 INFO mapreduce.Job:  map 83% reduce 5%
08/01/25 07:53:10 INFO mapreduce.Job:  map 84% reduce 5%
08/01/25 07:54:10 INFO mapreduce.Job:  map 85% reduce 6%
08/01/25 07:54:27 INFO mapreduce.Job:  map 86% reduce 6%
08/01/25 07:54:31 INFO mapreduce.Job:  map 87% reduce 6%
08/01/25 07:54:34 INFO mapreduce.Job:  map 88% reduce 6%
08/01/25 07:54:37 INFO mapreduce.Job:  map 89% reduce 6%
08/01/25 07:54:48 INFO mapreduce.Job:  map 90% reduce 6%
08/01/25 07:55:17 INFO mapreduce.Job:  map 91% reduce 6%
08/01/25 07:55:33 INFO mapreduce.Job:  map 93% reduce 6%
08/01/25 07:55:36 INFO mapreduce.Job:  map 94% reduce 6%
08/01/25 07:55:40 INFO mapreduce.Job:  map 95% reduce 6%
08/01/25 07:55:49 INFO mapreduce.Job:  map 96% reduce 6%
08/01/25 07:55:56 INFO mapreduce.Job:  map 97% reduce 6%
08/01/25 07:57:25 INFO mapreduce.Job:  map 98% reduce 6%
08/01/25 07:57:28 INFO mapreduce.Job:  map 99% reduce 6%
08/01/25 07:57:45 INFO mapreduce.Job:  map 100% reduce 6%
08/01/25 07:58:50 INFO mapreduce.Job:  map 100% reduce 7%
08/01/25 08:00:01 INFO mapreduce.Job:  map 100% reduce 8%
08/01/25 08:01:10 INFO mapreduce.Job:  map 100% reduce 9%
08/01/25 08:02:44 INFO mapreduce.Job:  map 100% reduce 10%
08/01/25 08:03:45 INFO mapreduce.Job:  map 100% reduce 11%
08/01/25 08:04:07 INFO mapreduce.Job: Task Id : attempt_1201209004442_0001_r_000000_0, Status : FAILED
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#1
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/attempt_1201209004442_0001_r_000000_0/map_41.out
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:398)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at org.apache.hadoop.mapred.YarnOutputFiles.getInputFileForWrite(YarnOutputFiles.java:213)
at org.apache.hadoop.mapreduce.task.reduce.OnDiskMapOutput.<init>(OnDiskMapOutput.java:61)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:257)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:411)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:341)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)

08/01/25 08:04:08 INFO mapreduce.Job:  map 100% reduce 0%
08/01/25 08:05:01 INFO mapreduce.Job: Task Id : attempt_1201209004442_0001_r_000000_1, Status : FAILED
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#5
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/attempt_1201209004442_0001_r_000000_1/map_11.out
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:398)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at org.apache.hadoop.mapred.YarnOutputFiles.getInputFileForWrite(YarnOutputFiles.java:213)
at org.apache.hadoop.mapreduce.task.reduce.OnDiskMapOutput.<init>(OnDiskMapOutput.java:61)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:257)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:411)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:341)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)

08/01/25 08:05:17 INFO mapreduce.Job:  map 100% reduce 1%
08/01/25 08:05:42 INFO mapreduce.Job:  map 100% reduce 2%
08/01/25 08:05:58 INFO mapreduce.Job:  map 100% reduce 3%
08/01/25 08:06:26 INFO mapreduce.Job:  map 100% reduce 4%
08/01/25 08:07:15 INFO mapreduce.Job:  map 100% reduce 5%
08/01/25 08:07:48 INFO mapreduce.Job:  map 100% reduce 6%
08/01/25 08:08:21 INFO mapreduce.Job: Task Id : attempt_1201209004442_0001_r_000000_2, Status : FAILED
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#4
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/attempt_1201209004442_0001_r_000000_2/map_20.out
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:398)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at org.apache.hadoop.mapred.YarnOutputFiles.getInputFileForWrite(YarnOutputFiles.java:213)
at org.apache.hadoop.mapreduce.task.reduce.OnDiskMapOutput.<init>(OnDiskMapOutput.java:61)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:257)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:411)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:341)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)

08/01/25 08:08:22 INFO mapreduce.Job:  map 100% reduce 0%


第二次运行：


[root@sjfx jar]# hadoop jar GameLoginLogAnalyzeA.jar /tangzw/input /tangzw/output
14/10/04 11:30:36 INFO client.RMProxy: Connecting to ResourceManager at sjfx/192.168.57.127:18040
14/10/04 11:30:36 INFO client.RMProxy: Connecting to ResourceManager at sjfx/192.168.57.127:18040
14/10/04 11:30:37 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
14/10/04 11:30:38 INFO mapred.FileInputFormat: Total input paths to process : 13
14/10/04 11:30:38 INFO mapreduce.JobSubmitter: number of splits:103
14/10/04 11:30:38 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/10/04 11:30:38 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/10/04 11:30:38 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/10/04 11:30:38 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/10/04 11:30:38 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/10/04 11:30:38 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/10/04 11:30:38 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/10/04 11:30:38 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/10/04 11:30:38 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/10/04 11:30:39 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1412391426441_0002
14/10/04 11:30:39 INFO impl.YarnClientImpl: Submitted application application_1412391426441_0002 to ResourceManager at sjfx/192.168.57.127:18040
14/10/04 11:30:39 INFO mapreduce.Job: The url to track the job: http://sjfx:18088/proxy/application_1412391426441_0002/
14/10/04 11:30:39 INFO mapreduce.Job: Running job: job_1412391426441_0002
14/10/04 11:30:48 INFO mapreduce.Job: Job job_1412391426441_0002 running in uber mode : false
14/10/04 11:30:48 INFO mapreduce.Job:  map 0% reduce 0%
14/10/04 11:31:27 INFO mapreduce.Job:  map 1% reduce 0%
14/10/04 11:31:31 INFO mapreduce.Job:  map 4% reduce 0%
14/10/04 11:31:46 INFO mapreduce.Job:  map 5% reduce 0%
14/10/04 11:31:51 INFO mapreduce.Job:  map 7% reduce 0%
14/10/04 11:32:08 INFO mapreduce.Job:  map 9% reduce 0%
14/10/04 11:32:11 INFO mapreduce.Job:  map 10% reduce 0%
14/10/04 11:32:35 INFO mapreduce.Job:  map 11% reduce 0%
14/10/04 11:33:16 INFO mapreduce.Job:  map 12% reduce 0%
14/10/04 11:33:19 INFO mapreduce.Job:  map 13% reduce 0%
14/10/04 11:33:31 INFO mapreduce.Job:  map 14% reduce 0%
14/10/04 11:33:35 INFO mapreduce.Job:  map 15% reduce 0%
14/10/04 11:33:42 INFO mapreduce.Job:  map 16% reduce 0%
14/10/04 11:34:00 INFO mapreduce.Job:  map 17% reduce 0%
14/10/04 11:34:25 INFO mapreduce.Job:  map 18% reduce 0%
14/10/04 11:34:38 INFO mapreduce.Job:  map 19% reduce 0%
14/10/04 11:35:05 INFO mapreduce.Job:  map 20% reduce 0%
14/10/04 11:35:16 INFO mapreduce.Job:  map 21% reduce 0%
14/10/04 11:35:21 INFO mapreduce.Job:  map 22% reduce 0%
14/10/04 11:35:32 INFO mapreduce.Job:  map 23% reduce 0%
14/10/04 11:36:04 INFO mapreduce.Job:  map 24% reduce 0%
14/10/04 11:36:23 INFO mapreduce.Job:  map 24% reduce 1%
14/10/04 11:36:48 INFO mapreduce.Job:  map 25% reduce 1%
14/10/04 11:36:51 INFO mapreduce.Job:  map 26% reduce 1%
14/10/04 11:36:54 INFO mapreduce.Job:  map 27% reduce 1%
14/10/04 11:36:57 INFO mapreduce.Job:  map 28% reduce 1%
14/10/04 11:37:06 INFO mapreduce.Job:  map 29% reduce 1%
14/10/04 11:37:22 INFO mapreduce.Job:  map 31% reduce 1%
14/10/04 11:37:32 INFO mapreduce.Job:  map 32% reduce 1%
14/10/04 11:37:43 INFO mapreduce.Job:  map 33% reduce 2%
14/10/04 11:38:05 INFO mapreduce.Job:  map 34% reduce 2%
14/10/04 11:38:27 INFO mapreduce.Job:  map 35% reduce 2%
14/10/04 11:38:53 INFO mapreduce.Job:  map 36% reduce 2%
14/10/04 11:39:02 INFO mapreduce.Job:  map 37% reduce 2%
14/10/04 11:39:08 INFO mapreduce.Job:  map 38% reduce 2%
14/10/04 11:39:34 INFO mapreduce.Job:  map 39% reduce 2%
14/10/04 11:39:40 INFO mapreduce.Job:  map 39% reduce 3%
14/10/04 11:39:46 INFO mapreduce.Job:  map 40% reduce 3%
14/10/04 11:39:49 INFO mapreduce.Job:  map 41% reduce 3%
14/10/04 11:40:19 INFO mapreduce.Job:  map 43% reduce 3%
14/10/04 11:40:49 INFO mapreduce.Job:  map 44% reduce 3%
14/10/04 11:41:11 INFO mapreduce.Job:  map 45% reduce 3%
14/10/04 11:41:13 INFO mapreduce.Job:  map 45% reduce 4%
14/10/04 11:41:33 INFO mapreduce.Job:  map 46% reduce 4%
14/10/04 11:41:44 INFO mapreduce.Job:  map 47% reduce 4%
14/10/04 11:41:50 INFO mapreduce.Job:  map 48% reduce 4%
14/10/04 11:41:58 INFO mapreduce.Job:  map 49% reduce 4%
14/10/04 11:42:14 INFO mapreduce.Job:  map 50% reduce 4%
14/10/04 11:42:27 INFO mapreduce.Job:  map 51% reduce 4%
14/10/04 11:42:40 INFO mapreduce.Job:  map 52% reduce 4%
14/10/04 11:43:02 INFO mapreduce.Job:  map 53% reduce 4%
14/10/04 11:43:17 INFO mapreduce.Job:  map 54% reduce 4%
14/10/04 11:43:41 INFO mapreduce.Job:  map 55% reduce 4%
14/10/04 11:43:55 INFO mapreduce.Job:  map 55% reduce 5%
14/10/04 11:44:05 INFO mapreduce.Job:  map 56% reduce 5%
14/10/04 11:44:16 INFO mapreduce.Job:  map 57% reduce 5%
14/10/04 11:44:32 INFO mapreduce.Job:  map 58% reduce 5%
14/10/04 11:44:47 INFO mapreduce.Job:  map 59% reduce 5%
14/10/04 11:44:51 INFO mapreduce.Job:  map 60% reduce 5%
14/10/04 11:45:09 INFO mapreduce.Job:  map 61% reduce 5%
14/10/04 11:45:25 INFO mapreduce.Job:  map 62% reduce 5%
14/10/04 11:45:48 INFO mapreduce.Job:  map 63% reduce 5%
14/10/04 11:46:06 INFO mapreduce.Job:  map 64% reduce 5%
14/10/04 11:46:15 INFO mapreduce.Job:  map 64% reduce 6%
14/10/04 11:46:36 INFO mapreduce.Job:  map 65% reduce 6%
14/10/04 11:47:10 INFO mapreduce.Job:  map 66% reduce 6%
14/10/04 11:47:24 INFO mapreduce.Job:  map 67% reduce 6%
14/10/04 11:47:38 INFO mapreduce.Job:  map 68% reduce 6%
14/10/04 11:47:42 INFO mapreduce.Job:  map 69% reduce 6%
14/10/04 11:47:50 INFO mapreduce.Job:  map 70% reduce 6%
14/10/04 11:47:57 INFO mapreduce.Job:  map 71% reduce 6%
14/10/04 11:48:03 INFO mapreduce.Job:  map 72% reduce 6%
14/10/04 11:48:07 INFO mapreduce.Job:  map 73% reduce 6%
14/10/04 11:48:24 INFO mapreduce.Job:  map 74% reduce 6%
14/10/04 11:48:52 INFO mapreduce.Job:  map 75% reduce 6%
14/10/04 11:48:58 INFO mapreduce.Job:  map 75% reduce 7%
14/10/04 11:49:17 INFO mapreduce.Job:  map 76% reduce 7%
14/10/04 11:49:43 INFO mapreduce.Job:  map 77% reduce 7%
14/10/04 11:50:06 INFO mapreduce.Job:  map 78% reduce 7%
14/10/04 11:50:18 INFO mapreduce.Job:  map 79% reduce 7%
14/10/04 11:50:25 INFO mapreduce.Job:  map 79% reduce 8%
14/10/04 11:50:55 INFO mapreduce.Job:  map 80% reduce 8%
14/10/04 11:50:58 INFO mapreduce.Job:  map 81% reduce 8%
14/10/04 11:51:18 INFO mapreduce.Job:  map 82% reduce 8%
14/10/04 11:51:24 INFO mapreduce.Job:  map 83% reduce 8%
14/10/04 11:51:45 INFO mapreduce.Job:  map 84% reduce 8%
14/10/04 11:52:05 INFO mapreduce.Job:  map 85% reduce 8%
14/10/04 11:52:17 INFO mapreduce.Job:  map 85% reduce 9%
14/10/04 11:52:28 INFO mapreduce.Job:  map 86% reduce 9%
14/10/04 11:52:31 INFO mapreduce.Job:  map 87% reduce 9%
14/10/04 11:52:34 INFO mapreduce.Job:  map 88% reduce 9%
14/10/04 11:52:38 INFO mapreduce.Job:  map 89% reduce 9%
14/10/04 11:52:42 INFO mapreduce.Job:  map 90% reduce 9%
14/10/04 11:52:59 INFO mapreduce.Job:  map 91% reduce 9%
14/10/04 11:53:28 INFO mapreduce.Job:  map 92% reduce 9%
14/10/04 11:53:55 INFO mapreduce.Job:  map 93% reduce 9%
14/10/04 11:53:58 INFO mapreduce.Job: Task Id : attempt_1412391426441_0002_m_000097_0, Status : FAILED
14/10/04 11:54:00 INFO mapreduce.Job: Task Id : attempt_1412391426441_0002_m_000099_0, Status : FAILED
14/10/04 11:54:00 INFO mapreduce.Job: Task Id : attempt_1412391426441_0002_m_000098_0, Status : FAILED
14/10/04 11:54:00 INFO mapreduce.Job: Task Id : attempt_1412391426441_0002_m_000100_0, Status : FAILED
14/10/04 11:54:00 INFO mapreduce.Job: Task Id : attempt_1412391426441_0002_m_000101_0, Status : FAILED
14/10/04 11:54:05 INFO mapreduce.Job: Task Id : attempt_1412391426441_0002_m_000097_1, Status : FAILED
14/10/04 11:54:05 INFO mapreduce.Job: Task Id : attempt_1412391426441_0002_r_000000_0, Status : FAILED
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#5
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/attempt_1412391426441_0002_r_000000_0/map_20.out
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:398)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at org.apache.hadoop.mapred.YarnOutputFiles.getInputFileForWrite(YarnOutputFiles.java:213)
at org.apache.hadoop.mapreduce.task.reduce.OnDiskMapOutput.<init>(OnDiskMapOutput.java:61)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:257)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:411)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:341)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)

14/10/04 11:54:06 INFO mapreduce.Job:  map 93% reduce 0%
14/10/04 11:54:09 INFO mapreduce.Job: Task Id : attempt_1412391426441_0002_m_000099_1, Status : FAILED
14/10/04 11:54:09 INFO mapreduce.Job: Task Id : attempt_1412391426441_0002_m_000098_1, Status : FAILED
14/10/04 11:54:18 INFO mapreduce.Job:  map 94% reduce 0%
14/10/04 11:54:41 INFO mapreduce.Job:  map 96% reduce 0%
14/10/04 11:54:41 INFO mapreduce.Job: Task Id : attempt_1412391426441_0002_m_000101_1, Status : FAILED
Error: java.io.IOException: Spill failed
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTask.java:1540)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1447)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1412391426441_0002_m_000101_1_spill_0.out
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:398)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFiles.java:159)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1573)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:852)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1510)

14/10/04 11:54:41 INFO mapreduce.Job: Task Id : attempt_1412391426441_0002_m_000100_1, Status : FAILED
Error: java.io.IOException: Spill failed
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTask.java:1540)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1447)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1412391426441_0002_m_000100_1_spill_0.out
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:398)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFiles.java:159)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1573)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:852)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1510)

14/10/04 11:54:41 INFO mapreduce.Job: Task Id : attempt_1412391426441_0002_m_000099_2, Status : FAILED
Error: java.io.IOException: Spill failed
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTask.java:1540)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1447)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1412391426441_0002_m_000099_2_spill_0.out
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:398)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFiles.java:159)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1573)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:852)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1510)

14/10/04 11:54:41 INFO mapreduce.Job: Task Id : attempt_1412391426441_0002_m_000097_2, Status : FAILED
Error: java.io.IOException: Spill failed
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTask.java:1540)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1447)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1412391426441_0002_m_000097_2_spill_0.out
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:398)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFiles.java:159)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1573)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:852)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1510)

14/10/04 11:54:42 INFO mapreduce.Job:  map 94% reduce 0%
14/10/04 11:54:42 INFO mapreduce.Job: Task Id : attempt_1412391426441_0002_r_000000_1, Status : FAILED
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#3
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/attempt_1412391426441_0002_r_000000_1/map_4.out
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:398)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at org.apache.hadoop.mapred.YarnOutputFiles.getInputFileForWrite(YarnOutputFiles.java:213)
at org.apache.hadoop.mapreduce.task.reduce.OnDiskMapOutput.<init>(OnDiskMapOutput.java:61)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:257)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:411)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:341)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)

14/10/04 11:54:44 INFO mapreduce.Job:  map 100% reduce 100%
14/10/04 11:54:51 INFO mapreduce.Job: Job job_1412391426441_0002 failed with state FAILED due to: Task failed task_1412391426441_0002_m_000097
Job failed as tasks failed. failedMaps:1 failedReduces:0

14/10/04 11:54:52 INFO mapreduce.Job: Counters: 34
File System Counters
FILE: Number of bytes read=13102192174
FILE: Number of bytes written=26212068108
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=12608837345
HDFS: Number of bytes written=0
HDFS: Number of read operations=288
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
Job Counters 
Failed map tasks=14
Failed reduce tasks=2
Killed map tasks=5
Launched map tasks=115
Launched reduce tasks=2
Other local map tasks=12
Data-local map tasks=103
Total time spent by all maps in occupied slots (ms)=18567852
Total time spent by all reduces in occupied slots (ms)=1205016
Map-Reduce Framework
Map input records=246865790
Map output records=246865790
Map output bytes=12608458902
Map output materialized bytes=13102191058
Input split bytes=9984
Combine input records=0
Spilled Records=493731580
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=180426
CPU time spent (ms)=3391620
Physical memory (bytes) snapshot=25874554880
Virtual memory (bytes) snapshot=86517563392
Total committed heap usage (bytes)=19113443328
File Input Format Counters 
Bytes Read=12608827361
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836)
at com.fiveone.game.GameLoginLog.main(GameLoginLog.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)


解决办法：
在mapred-site.xml配置文件中加上一条配置：
<property>
                <name>mapreduce.reduce.shuffle.memory.limit.percent</name>
                <value>0.20</value>
                <description>Expert: Maximum percentage of the in-memory limit that a single shuffle can consume</description> 
</property>
默认mapreduce.reduce.shuffle.memory.limit.percent配置的值为：0.25    ，可以根据自己的情况适量降低这个值。


出现这个问题的原因有很多种，要视情况去解决，下面是在网上找的一些解决方案：


2、hadoop能map不能reduce（shuffle阶段报错，fetch#1错误）

日志报错：attempt_201207232344_0002_r_000000_0, Status : FAILED
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#1
  Error reading task outputConnection refused
No job jar file set.   User classes may not be found. See Job or Job#setJar(String).
12/07/23 23:51:42 INFO mapreduce.JobSubmitter: Cleaning up the staging area hdfs://127.0.0.1:9000/tmp/hadoop-root/mapred/staging/root/.staging/job_201207232344_0003
在网上找了下解决办法，大致有几种可能的原因：
1、 Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out. 系统设置有关，文件较大，无法分配大内存
2、机器位数是64位导致这个问题
3、/etc/hosts 里面没有配master的hostname 
4、防火墙没关
             问题1我安装网上方法，对配置文件做了修改： vim /etc/security/limits.conf   加入 * soft nofile 102400   * hard nofile 409600
vim /etc/pam.d/login   加入：session      required       /lib/security/pam_limits.so但是还是没有办法解决
问题三：由于我是伪分布式，所以只要配本机就行，看了下配置了hostname
问题四：setup下看了我的防火墙，发现防火墙有两个，一个是firework一个是selinux，第一个关了，第二个没关
最后在我重启虚拟机后，终于可以正常运行，但是是不是就是第四步弄好的呢，在我重新打开selinux后，再次运行，依然正常，我想应该不是selinux没关的可能，至于到底什么原因，可能是问题1

 

5、hadoop能map不能reduce（shuffle阶段报错，fetch#1错误）

Too many fetch-failures
13/02/25 21:20:48 INFO mapreduce.Job: Task Id : attempt_201302230043_0022_r_000000_0, Status : FAILED
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#4
            at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124)
            at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362)
            at org.apache.hadoop.mapred.Child$4.run(Child.java:223)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:396)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
            at org.apache.hadoop.mapred.Child.main(Child.java:217)
Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
            at org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.checkReducerHealth(ShuffleScheduler.java:253)
            at org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.copyFailed(ShuffleScheduler.java:187)
            at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:347)
            at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:251)
            at org.apache.hado
      几个可能原因：1、hostname 配置    2、我的错误（选用的数据压缩方法不对，造成错误）：
job.getConfiguration().setClass("mapred.map.runner.class", MultithreadedMapRunner.class, MapRunnable.class);
job.getConfiguration().setBoolean("mapreduce.map.output.compress", true);


6、hadoop执行JOB在reduce阶段报了下面的错误： 
  1. Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#4   
  2. at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)   
  3. at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)   
  4. at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:165)   
  5. at java.security.AccessController.doPrivileged(Native Method)   
  6. at javax.security.auth.Subject.doAs(Subject.java:415)   
  7. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)   
  8. at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:160)   
  9. Caused by: java.lang.OutOfMemoryError: Java heap space   
  10. at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)   
  11. at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)   
  12. at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)   
  13. at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:297)   
  14. at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:287)   
  15. at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:411)   
  16. at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:341)   
  17. at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)  
解决方法： 

修改mapred-site.xml配置中的mapreduce.reduce.shuffle.memory.limit.percent参数 
  1. <property>  
  2.    <name>mapreduce.reduce.shuffle.memory.limit.percent</name>  
  3.   <value>0.25</value>  
  4.   <description>Expert: Maximum percentage of the in-memory limit that a  
  5.   single shuffle can consume</description>  
  6. </property>  


这个值默认是0.25，可以根据需要适当的减小这个值。


成汐
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 1
    评论
专栏目录
hadoop自带wordcount 卡在mapreduce.Job: map 0% reduce 0%，求指点谢谢！！！
qq_38932407的博客
11-13 5945
我是分布式集群，但是电脑配置低，就配置了一个master以及一个slave1。 运行hadoop命令完全没有问题，上课使用hadoop自带的wordcount，统计一个文件里面的登陆次数命令时，简直要气炸了【大哭】。   统计了N个钟头都不会动！！！   1. 尝试过的方法（均失败）： 1.1  将master和slave1的内存都改到3G，没有作用！！！，甚至增加处理器的位数，也...
运行wordcount的时候显示INFO mapreduce.Job: map 0% reduce 0%
Joy
01-05 6353
错误提示： [xiaoqiu@s150 /home/xiaoqiu]$ hadoop jar wordcounter.jar com.cr.wordcount.WordcountApp hdfs://s150/user/xiaoqiu/data/wc.txt hdfs://s150/user/xiaoqiu/data/out 18/01/05 09:12:52 INFO client.RMPro
关于Hadoop下运行MapReduce任务卡在Map0%和Reduce0%的解决办法
_Ashore的博客
05-25 5772
关于Hadoop下运行MapReduce任务卡在Map0%和Reduce0%的解决办法 检查核心配置 mapred-site.xml文件 检查以下配置信息，如果没有该信息，则添加上去 ![在这里插入图片描述](https://img-blog.csdnimg.cn/20200525163616428.png 放个对比图吧（这里的端口不一定要和我的一样，只要不冲突的都可以） 在这里插入代码片 <property> <name>mapreduce.framework.name</
关于在eclipse上能运行Map但无法运行Reduce的解决方法
yhzalan的博客
07-13 763
今日我在eclipse上运行Map/Reduce框架进行数据分析的时候遇到了一个很奇怪的错误： 一开始我发现运行程序之后也没在控制台报错，也生成了目标目录，但一直出不来统计好的数据文件。 我先通过位置标记输出来判断各个类的加载是否正常， 发现Map是可以正常加载执行的，但是Reduce一直无法加载执行。 然后我通过设置，让其在运行时显示日志信息，再运行。 文末会分享设置显示日志信息的方法。
hadoop3 任务卡在map 0% reduce 0%的解决方案
Akari0216的博客
08-31 4945
在VMWare上面配置好了hadoop3之后，当然是先试跑一下案例。然而在master上面运行wordcount的时候却出现了卡在map 0% reduce 0%的情况，搜索了这个问题的相关解决方案，基本是修改yarn-site.xml的配置，如 1.增大cpu数，yarn.nodemanager.resource.cpu-vcores 2.增大每个节点可用内存，yarn.scheduler.maximum-allocation-mb 3.调整虚拟内存值，yarn.nodemanager.vmem-pme
shuffle的关键阶段sort(Map端和Reduce端)源码分析
08-26
shuffle 的关键阶段 sort（Map 端和 Reduce 端）源码分析 在 Hadoop 的 MapReduce 框架中，排序是 shuffle 过程中的一个关键阶段。排序的目的是将数据按照特定的顺序排列，以便于后续的处理。今天，我们将深入分析 ...
HadoopMapReduce:数据集链接的Hadoop MapReduce实践问题
05-10
- "hadoop-mapreduce combiner"：Combiner是可选的MapReduce组件，可以在Map阶段本地聚合数据，减少网络传输，提高效率。 - "drwhoami"：这可能是项目或例子的特定部分，但在这里没有详细信息，我们不作深入讨论。 -...
Hadoop-mapreduce过程.doc
最新发布
06-26
5. **Reduce任务执行** reduce任务的执行方式类似，但数据来源是从各个map任务的输出中。TaskTracker会从其他TaskTracker处拉取属于本reduce任务的中间结果，经过shuffle和sort阶段，再交给用户定义的reduce函数...
03-Hadoop-MapReduce.docx
06-18
它将大规模数据处理任务分解为两个主要阶段：Map阶段和Reduce阶段，这两个阶段通过中间结果进行通信。 **1.1 优点** - **可扩展性**：MapReduce能够轻松地扩展到数千台服务器，处理PB级别的数据。 - **容错性**：...
Hive 任务卡在 map = 0%, reduce = 0%
Alex
06-21 3628
Hive 卡在map = 0%, reduce = 0%阶段 解决：增加map个数，设置mapreduce.input.fileinputformat.split.maxsize 小于系统默认值，需要综合考虑调用更多map时候的消耗 定位时发现任务卡在map = 0%, reduce = 0%阶段，而且发现map分配数量很少，所以猜测分配map较慢，增加map数量可解决问题。 ...
hadoop运行计算pi或running job卡住或 map 0% reduce 0%
空空说技术的博客
04-15 4607
1.出现MapReduce任务运行到running job卡住 一：如下异常： Starting Job 16/06/30 01:15:34 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.10.50:8032 16/06/30 01:15:35...
遇到的情况--hive---动态创建分区报错dynamic.partitions.pernode. Maximum was set to: 100
直到世界的尽头
03-03 5646
遇到的情况 我们在hive shell中使用 insert overwrite table table_p partition(time) select id,time from table_info; 进行动态分区写入数据，然后在reduce的时候报错 hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 10...
map100% reduce0%&error in shuffle in localfetcher#1
qq_42568323的博客
04-05 1764
内存溢出&map100% reduce0% 第一个错误 org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#1 解决办法：在Hadoop安装目录下添加mapred-site.xml文件的属性，如下： <property> <name>mapreduce.admin.reduce.child.java.opts</name>
关于运行PI和wordcount的map0%，reduce0%问题解决方法
PeixinYe的博客
04-23 3011
记得：没金刚钻别揽瓷器活！当然怎么说呢。。4个site.xml文件配置别搞的太高大上，用默认的就好了，这样就不会导致你机器尤其是master节点卡在map0%和reduce0%了。。。。为了明白这问题，一搞这问题又是一天啊。。。我的电脑配置：3台腾讯云：1核2G，50G；以下为我的4个site文件配置：mapred-site.xml&lt;configuration&gt;&lt;property...
Hadoop运行mapreduce卡住或者一直停在map 0% reduce 0%的解决方案
热门推荐
家里没矿
10-20 2万+
首先先要说明我的系统的一个配置 主机名 主机IPV4地址 配置 hadoop001 192.168.120.100 RecourceManager、NodeManager、DateNode、NameNode hadoop002 192.168.120.101 NodeManager、DateNode、SecondaryNameNode hadoop003 192.168.1...
hadoop中运行map-reduce程序时，java.net.connectionException
一路向北
10-11 2514
about云开发 标题: mapreduce报错 java.net.ConnectException: Connection refused [打印本页] 作者: Wyy_Ck    时间: 2016-10-31 15:13标题: mapreduce报错 java.net.ConnectException: Connection refused弄了半天，系统是centos 7，本想执行一个测试...
js reduce实现中间件_MapReduce 模型
weixin_29710403的博客
01-12 140
MapReduce 模型是 Map 模型的扩展，新增 Reduce接口，需要实现 MapReduceJobProcessor。注意事项MapReduce 模型只有一个 Reduce，所有子任务完成后会执行 Reduce 方法，可以在 Reduce 方法中返回该任务示例的执行结果，作为工作流的上下游数据传递。如果有子任务失败，Reduce不会执行。Reduce 失败，整个任务示例也失败。Schedu...
探讨MapReduce失败的几种情形
andyguan01_2的博客
04-22 3476
在现实情况中，程序运行可能遇到各种故障：用户代码错误不断、进程崩溃、机器故障等等。使用Hadoop最主要的好处之一是它能处理此类故障并让你能够成功完成作业。五门需要考虑以下实体的失败：任务、application master、节点管理器和资源管理器。 1、任务运行失败 首先考虑任务失败的情况。最常见的情况是map任务或reduce任务中的用户代码抛出运行异常。如果发生这种情况，任务JVM会在退出...
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

成汐

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值