踩坑:Unable to initialize MapOutputCollector org.apache.hadoop.mapred.MapTask$MapOutputBuffer

今天在做MapReduce的时候,为了做个降序排列,所以定义了一个自定义序列,然后运行的时候就出现了这个问题,查了好多教程,最后发现错在了一个比较离谱的地方,记录一下,顺带写上其他几种可能。(1-3来源网络,4亲身经历)
先放个报错信息以供参考

2022-04-29 01:04:34,361 WARN  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:createSortingCollector(418)) - Unable to initialize MapOutputCollector org.apache.hadoop.mapred.MapTask$MapOutputBuffer
java.lang.RuntimeException: java.lang.NoSuchMethodException: model.classes.<init>()
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
        at org.apache.hadoop.io.WritableComparator.newKey(WritableComparator.java:144)
        at org.apache.hadoop.io.WritableComparator.<init>(WritableComparator.java:130)
        at org.apache.hadoop.io.WritableComparator.get(WritableComparator.java:65)
        at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:903)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1019)
        at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:409)
        at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:83)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:711)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:783)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodException: model.classes.<init>()
        at java.lang.Class.getConstructor0(Class.java:3082)
        at java.lang.Class.getDeclaredConstructor(Class.java:2178)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
        ... 16 more
2022-04-29 01:04:34,364 INFO  [Thread-21] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(486)) - map task executor complete.
2022-04-29 01:04:34,370 WARN  [Thread-21] mapred.LocalJobRunner (LocalJobRunner.java:run(590)) - job_local1220408697_0001
java.lang.Exception: java.io.IOException: Initialization of all the collectors failed. Error in last collector was:java.lang.RuntimeException: java.lang.NoSuchMethodException: model.classes.<init>()
        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.io.IOException: Initialization of all the collectors failed. Error in last collector was:java.lang.RuntimeException: java.lang.NoSuchMethodException: model.classes.<init>()
        at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:424)
        at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:83)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:711)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:783)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException: model.classes.<init>()
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
        at org.apache.hadoop.io.WritableComparator.newKey(WritableComparator.java:144)
        at org.apache.hadoop.io.WritableComparator.<init>(WritableComparator.java:130)
        at org.apache.hadoop.io.WritableComparator.get(WritableComparator.java:65)
        at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:903)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1019)
        at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:409)
        ... 10 more
Caused by: java.lang.NoSuchMethodException: model.classes.<init>()
        at java.lang.Class.getConstructor0(Class.java:3082)
        at java.lang.Class.getDeclaredConstructor(Class.java:2178)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
        ... 16 more

情况1

包导入错了,这是网上最多情况的报错原因,比如Text应该要导入org.apache.hadoop.io.Text之类的,仔细检查一下,看看有没有。

情况2

自定义序列的时候没有没有完全继承WritableComparable, 这种情况的话,只要继承一下WritableComparable就ok了, WritableComparable已经继承了WritableComparable。比如我的自定义序列:

public class classes implements WritableComparable<classes>{...}

情况3

如果是自定义序列的话,再MapReduce操作中是要求其可迭代的,也就是说需要重写一个compareTo方法,比如我的

public int compareTo(classes o) {
        //降序
       return (int) (-this.avgsorce+o.getAvgsorce());
    }

如果没写的话也可能出现报错

情况4

最后就是我的问题了,很简单,在自定义序列的时候,我只定义了一个有参的构造器,总所周知,在Java里如果已经定义了构造器,系统就不再有默认构造器了,这就导致我定义的这个自定义序列没有无参构造器,好巧不巧,MapReduce排序的时候就偏偏要用到这个无参构造器,得嘞,这可不就报错了,到此我定义了一个无参构造器,问题解决

public classes(){

    }

这坑有点深,找了我好久,再加上现在是北京时间1:21,半夜啦!!忍不住发一下吐槽一下(这句纯吐槽,不想看可以略过哈哈哈)

转载请注明出处,谢谢!

评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

T&King

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值