hadoop报错: java.lang.ArrayIndexOutOfBoundsException

每次都是跑到map 99%   reduce 33%的时候挂掉


2013-12-04 10:40:45,128 INFO org.apache.hadoop.mapred.Merger: Merging 16 sorted segments
2013-12-04 10:40:45,172 INFO org.apache.hadoop.mapred.Merger: Merging 2 intermediate segments out of a total of 14
2013-12-04 10:40:45,314 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 8 segments left of total size: 255981 bytes
2013-12-04 10:40:45,324 INFO org.apache.hadoop.mapred.Merger: Merging 16 sorted segments
2013-12-04 10:40:45,385 INFO org.apache.hadoop.mapred.Merger: Merging 2 intermediate segments out of a total of 14
2013-12-04 10:40:45,527 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 8 segments left of total size: 163242 bytes
2013-12-04 10:40:45,532 INFO org.apache.hadoop.mapred.Merger: Merging 16 sorted segments
2013-12-04 10:40:45,577 INFO org.apache.hadoop.mapred.Merger: Merging 2 intermediate segments out of a total of 16
2013-12-04 10:40:46,984 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-12-04 10:40:46,986 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.ArrayIndexOutOfBoundsException
	at java.lang.System.arraycopy(Native Method)
	at org.apache.hadoop.mapred.IFile$Reader.rejigDataOffset(IFile.java:460)
	at org.apache.hadoop.mapred.IFile$Reader.readNextBlockAndReserveKey(IFile.java:424)
	at org.apache.hadoop.mapred.IFile$Reader.nextValue(IFile.java:599)
	at org.apache.hadoop.mapred.Merger$Segment.nextValue(Merger.java:243)
	at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:388)
	at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:157)
	at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:534)
	at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:414)
	at org.apache.hadoop.mapred.Merger.merge(Merger.java:78)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1563)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1180)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:392)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
	at org.apache.hadoop.mapred.Child.main(Child.java:260)
2013-12-04 10:40:46,989 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task


 解决方法: 

custom param中加入 mapred.ifile.buffer.reset.size.mb:256

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 3
    评论
评论 3
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值