mapreduce运行遇到的问题-1

1.ShuffleError: error in shuffle in fetcher

Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#1 at 
org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) at 
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376) at 
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Unknown Source) at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at 
org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: 
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for 
output/attempt_1530235952408_0005_r_000102_0/map_149.out at 
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:402) at 
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at 
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at 
org.apache.hadoop.mapred.YarnOutputFiles.getInputFileForWrite(YarnOutputFiles.java:213) at 
org.apache.hadoop.mapreduce.task.reduce.OnDiskMapOutput.<init>(OnDiskMapOutput.java:61) at 
org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:257) at 
org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:411) at 
org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:341) at 
org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165) 

因为hadoopdata的空间不够，伴随Unhealthy Nodes 出现

2.ShuffleError: error in shuffle in InMemoryMerger

Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs at 
org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) at 
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376) at 
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Unknown Source) at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at 
org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/attempt_1530493351637_0003_r_000002_0/map_34.out at 
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:402) at
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at 
org.apache.hadoop.mapred.YarnOutputFiles.getInputFileForWrite(YarnOutputFiles.java:213) at 
org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl$InMemoryMerger.merge(MergeManagerImpl.java:450) at 
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run(MergeThread.java:94) 

这个是因为reduce做shuffle时候临时文件目录不够，但是是配置在hdfs上，看了下磁盘，的确有100%的而且Unhealthy Nodes没有节点。所以也很奇怪。说明shuffle时候临时文件在本地还是有存的。