[hadoop@hadoop01 ~]$ hadoop jar mr.jar cn.hadoop.mr.WCRunner
16/07/24 16:52:08 INFO client.RMProxy: Connecting to ResourceManager at hadoop01/192.168.56.200:8032
16/07/24 16:52:09 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/07/24 16:52:09 INFO input.FileInputFormat: Total input paths to process : 1
16/07/24 16:52:09 INFO mapreduce.JobSubmitter: number of splits:1
16/07/24 16:52:09 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1469343376698_0001
16/07/24 16:52:10 INFO impl.YarnClientImpl: Submitted application application_1469343376698_0001
16/07/24 16:52:10 INFO mapreduce.Job: The url to track the job: http://hadoop01:8088/proxy/application_1469343376698_0001/
16/07/24 16:52:10 INFO mapreduce.Job: Running job: job_1469343376698_0001
16/07/24 16:52:20 INFO mapreduce.Job: Job job_1469343376698_0001 running in uber mode : false
16/07/24 16:52:20 INFO mapreduce.Job: map 0% reduce 0%
16/07/24 16:52:25 INFO mapreduce.Job: Task Id : attempt_1469343376698_0001_m_000000_0, Status : FAILED
Error: java.io.IOException: Unable to initialize any output collector
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:412)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:695)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
16/07/24 16:52:30 INFO mapreduce.Job: Task Id : attempt_1469343376698_0001_m_000000_1, Status : FAILED
Error: java.io.IOException: Unable to initialize any output collector
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:412)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:695)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
16/07/24 16:52:35 INFO mapreduce.Job: Task Id : attempt_1469343376698_0001_m_000000_2, Status : FAILED
Error: java.io.IOException: Unable to initialize any output collector
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:412)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:695)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
16/07/24 16:52:42 INFO mapreduce.Job: map 100% reduce 100%
16/07/24 16:52:42 INFO mapreduce.Job: Job job_1469343376698_0001 failed with state FAILED due to: Task failed task_1469343376698_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
16/07/24 16:52:42 INFO mapreduce.Job: Counters: 9
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=15512
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=15512
Total vcore-seconds taken by all map tasks=15512
Total megabyte-seconds taken by all map tasks=15884288
错误原因如下:
public class WCRunner {
public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
Configuration conf = new Configuration();
Job wcjob = Job.getInstance(conf);
wcjob.setJarByClass(WCRunner.class);
wcjob.setMapperClass(WCMapper.class);
wcjob.setReducerClass(WCReducer.class);
wcjob.setOutputKeyClass(Text.class);
wcjob.setOutputValueClass(LongWritable.class);
wcjob.setMapOutputKeyClass(Text.class);
wcjob.setMapOutputValueClass(LongWritable.class);
FileInputFormat.setInputPaths(wcjob, "/wc/inputdata/");
FileOutputFormat.setOutputPath(wcjob, new Path("/wc/output/"));
wcjob.waitForCompletion(true);
}
}
这里的Text是import com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider.Text;
正确的类型是import org.apache.hadoop.io.Text;