hadoop pipes 中遇到问题

c++实现的mapreduce代码没有问题,执行指令时候提示 如下


16/05/30 17:43:33 INFO mapreduce.Job: Task Id : attempt_1464593740608_0013_m_000002_0, Status : FAILED
Error: java.io.IOException: pipe child exception
        at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:226)
        at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:102)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.EOFException
        at java.io.DataInputStream.readByte(DataInputStream.java:267)
        at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
        at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)
        at org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:125)


这个是因为命令指定不对引起的:

正确命令:

        hadoop pipes -D hadoop.pipes.java.recordreader=true -D hadoop.pipes.java.recordwriter=true -input /input -output /output12 -program /cppexe/wordcount


注意点: 1、大小写问题

                  2、两个   -D

                  3、hadoop.pipes.java.recordreaderhadoop.pipes.java.recordwriter 的设置 值及大小写

阅读更多
文章标签: hadoop
个人分类: 技术文档
想对作者说点什么? 我来说一句

没有更多推荐了,返回首页

加入CSDN,享受更精准的内容推荐,与500万程序员共同成长!
关闭
关闭