hadoop报错: java.lang.ArrayIndexOutOfBoundsException

原创 2013年12月04日 13:56:33

每次都是跑到map 99%   reduce 33%的时候挂掉


2013-12-04 10:40:45,128 INFO org.apache.hadoop.mapred.Merger: Merging 16 sorted segments
2013-12-04 10:40:45,172 INFO org.apache.hadoop.mapred.Merger: Merging 2 intermediate segments out of a total of 14
2013-12-04 10:40:45,314 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 8 segments left of total size: 255981 bytes
2013-12-04 10:40:45,324 INFO org.apache.hadoop.mapred.Merger: Merging 16 sorted segments
2013-12-04 10:40:45,385 INFO org.apache.hadoop.mapred.Merger: Merging 2 intermediate segments out of a total of 14
2013-12-04 10:40:45,527 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 8 segments left of total size: 163242 bytes
2013-12-04 10:40:45,532 INFO org.apache.hadoop.mapred.Merger: Merging 16 sorted segments
2013-12-04 10:40:45,577 INFO org.apache.hadoop.mapred.Merger: Merging 2 intermediate segments out of a total of 16
2013-12-04 10:40:46,984 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-12-04 10:40:46,986 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.ArrayIndexOutOfBoundsException
	at java.lang.System.arraycopy(Native Method)
	at org.apache.hadoop.mapred.IFile$Reader.rejigDataOffset(IFile.java:460)
	at org.apache.hadoop.mapred.IFile$Reader.readNextBlockAndReserveKey(IFile.java:424)
	at org.apache.hadoop.mapred.IFile$Reader.nextValue(IFile.java:599)
	at org.apache.hadoop.mapred.Merger$Segment.nextValue(Merger.java:243)
	at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:388)
	at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:157)
	at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:534)
	at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:414)
	at org.apache.hadoop.mapred.Merger.merge(Merger.java:78)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1563)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1180)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:392)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
	at org.apache.hadoop.mapred.Child.main(Child.java:260)
2013-12-04 10:40:46,989 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task


 解决方法: 

custom param中加入 mapred.ifile.buffer.reset.size.mb:256

java.lang.ArrayIndexOutOfBoundsException异常分析及解决

这是一个非常常见的异常,从名字上看是数组下标越界错误,解决方法就是查看为什么下标越界。下面是一个错误示例: Exception in thread "main" java.lang.ArrayInde...

hive中的 数组越界异常 IndexOutOfBoundsException

java.lang.RuntimeException: Hive Runtime Error while closing operators at org.apache.hadoop.hive.ql...

ArrayList在多线程调用Add()添加元素时的下标越界问题(java.lang.ArrayIndexOutOfBoundsException)

最近在看《实战Java虚拟机》一书,看到有关锁与并发章节时,看到如下一个多线程使用ArrayList的例子:        两个线程t1和t2同时向numberList中添加数据,由于Arra...

Solution of ArrayIndexOutOfBoundsException while extractor Microsoft Format Files with POI

这两天在用POI提取 Microsoft 格式文档的内容的时候莫名其妙的遇到了一个问题,困扰了我3天: 我的POI版本为:poi-3.0.2-FINAL-20080204.jar;poi-contri...
  • rxr1st
  • rxr1st
  • 2008年03月26日 01:04
  • 2208

如何自定义mapreduce在输出的key和value之间自动添加的分隔符

背景:我在用mahout做一个推荐

Hadoop2.2.0 中错误总结之(org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /test._COPYING)

错误: [root@xiajie01 sbin]# hadoop fs -put /root/20131210110122880.doc  hdfs://192.168.30.169:9000/te...

hadoop常见问题及其解决办法

转自Hadoop使用常见问题以及解决方法(转载),保存在此以学习。 1:Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-ou...

Hadoop异常 hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException

转自:http://blog.csdn.net/shenshouer/article/details/7360262 按照http://hadoop.apache.org/commo...
  • jokes000
  • jokes000
  • 2012年07月01日 23:01
  • 12566

Spark On CDH5.6.0 报错java.lang.ArrayIndexOutOfBoundsException: 1

Spark On CDH5.6.0 报错java.lang.ArrayIndexOutOfBoundsException: 1 问题的解决过程分享

eclipse运行mapreduce 提示:Caused by: java.lang.ArrayIndexOutOfBoundsException

eclipse运行mapreduce 提示:Caused by: java.lang.ArrayIndexOutOfBoundsException
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:hadoop报错: java.lang.ArrayIndexOutOfBoundsException
举报原因:
原因补充:

(最多只允许输入30个字)