在使用自定义序列化类型时,job程序出现的错误。
Unable to initialize MapOutputCollector org.apache.hadoop.mapred.MapTask$MapOutputBuffer
java.lang.ClassCastException: class cn.wintersun.bean.FlowBean
at java.base/java.lang.Class.asSubclass(Class.java:3613)
at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:887)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1011)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:402)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:704)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:776)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.base/java.lang.Thread.run(Thread.java:844)
[INFO ] 2019-03-27 09:13:17,964 method:org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:485)
map task executor complete.
[WARN ] 2019-03-27 09:13:17,966 method:org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:589)
job_local1956632227_0001
java.lang.Exception: java.io.IOException: Initialization of all the collectors failed. Error in last collector was:java.lang.ClassCastException: class cn.wintersun.bean.FlowBean
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:491)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:551)
Caused by: java.io.IOException: Initialization of all the collectors failed. Error in last collector was:java.lang.ClassCastException: class cn.wintersun.bean.FlowBean
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:416)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:704)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:776)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.lang.ClassCastException: class cn.wintersun.bean.FlowBean
at java.base/java.lang.Class.asSubclass(Class.java:3613)
at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:887)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1011)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:402)
... 10 more
最后发现是我的job程序设置setMapOutputValueClass时设置错了
//错误
job.setMapOutputKeyClass(Text.class);
job.setMapOutputKeyClass(FlowBean.class);
//正确
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(FlowBean.class);