mapreduce运行过程反复卡在[org.apache.hadoop.mapred.LocalJobRunner] - reduce ＞ copy原因以及解决办法

先贴具体info:

INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6c57fea9
2017-08-15 15:51:35,642 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1503238528, maxSingleShuffleLimit=375809632, mergeThreshold=992137472, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2017-08-15 15:51:35,643 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local19506925_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2017-08-15 15:51:36,238 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 0%
2017-08-15 15:51:41,595 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > copy
2017-08-15 15:51:44,596 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > copy
2017-08-15 15:51:47,597 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > copy
2017-08-15 15:51:50,597 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > copy
2017-08-15 15:51:53,598 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > copy
2017-08-15 15:51:56,598 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > copy
2017-08-15 15:51:59,599 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > copy
2017-08-15 15:52:02,601 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > copy
2017-08-15 15:52:05,601 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > copy
2017-08-15 15:52:08,601 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > copy
2017-08-15 15:52:11,602 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > copy
2017-08-15 15:52:14,602 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > copy
2017-08-15 15:52:17,602 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > copy
2017-08-15 15:52:20,603 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > copy

......................

一直循环跳出，并且在HDFS上会有创建目录操作，当然目录操作里是没有结果的。

网络上说是因为我们编写的mapreduce代码中reduce出现死循环，这个是没错的、

但是我的代码里并没有包含reduce函数，我直接用map函数作为结果输出，于是可以想到是因为没有告诉程序我们不执行reduce函数

所以程序一直在找reduce作为输入？（是否可以这么说呢，我不能百分百确定）

于是我在程序中加入如下代码：job.setNumReduceTasks(0);          //设置不进行reduce任务，否则任务会卡在map 100% reduce0 % 后

程序跑通：

以上，希望能帮到你。