Hive多表连接异常，java.lang.ArrayIndexOutOfBounds ：140，官方Bug，在3.0.0版本已经被解决了

最新推荐文章于 2024-05-09 21:59:20 发布
最新推荐文章于 2024-05-09 21:59:20 发布
阅读量5.1k 收藏 4
点赞数 2
分类专栏： Hive 大数据 文章标签： Hive ArrayIndexOutOfBounds
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_38822045/article/details/88050203
版权

后续官方解决方案：https://issues.apache.org/jira/browse/HIVE-14564

异常详细情况

2019-02-28 16:33:44,429 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
2019-02-28 16:33:44,429 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e25_1551269222015_0034_01_000005, NodeId: bigdata001:45454, NodeHttpAddress: bigdata001:8042, Resource: <memory:2048, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 192.168.30.230:45454 }, ] to fast fail map
2019-02-28 16:33:44,431 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
2019-02-28 16:33:44,432 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e25_1551269222015_0034_01_000005 to attempt_1551269222015_0034_m_000000_3
2019-02-28 16:33:44,432 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:43008, vCores:1>
2019-02-28 16:33:44,432 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 1
2019-02-28 16:33:44,432 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:4 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:0 RackLocal:1
2019-02-28 16:33:44,432 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved bigdata001 to /default-rack
2019-02-28 16:33:44,433 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1551269222015_0034_m_000000_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2019-02-28 16:33:44,433 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e25_1551269222015_0034_01_000005 taskAttempt attempt_1551269222015_0034_m_000000_3
2019-02-28 16:33:44,433 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1551269222015_0034_m_000000_3
2019-02-28 16:33:44,434 INFO [ContainerLauncher #6] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : bigdata001:45454
2019-02-28 16:33:44,441 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1551269222015_0034_m_000000_3 : 13562
2019-02-28 16:33:44,441 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1551269222015_0034_m_000000_3] using containerId: [container_e25_1551269222015_0034_01_000005 on NM: [bigdata001:45454]
2019-02-28 16:33:44,442 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1551269222015_0034_m_000000_3 TaskAttempt Transitioned from ASSIGNED to RUNNING
2019-02-28 16:33:45,434 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1551269222015_0034: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:43008, vCores:1> knownNMs=3
2019-02-28 16:33:45,785 INFO [Socket Reader #1 for port 35318] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1551269222015_0034 (auth:SIMPLE)
2019-02-28 16:33:45,796 INFO [IPC Server handler 4 on 35318] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1551269222015_0034_m_27487790694405 asked for a task
2019-02-28 16:33:45,796 INFO [IPC Server handler 4 on 35318] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1551269222015_0034_m_27487790694405 given task: attempt_1551269222015_0034_m_000000_3
2019-02-28 16:33:47,740 INFO [IPC Server handler 1 on 35318] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1551269222015_0034_m_000000_3 is : 0.0
2019-02-28 16:33:47,743 FATAL [IPC Server handler 2 on 35318] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1551269222015_0034_m_000000_3 - exited : java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row [Error getting row data with exception java.lang.ArrayIndexOutOfBoundsException: -1746617499
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.readVInt(LazyBinaryUtils.java:314)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:183)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:202)
    at org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
    at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:354)
    at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:198)
    at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:184)
    at org.apache.hadoop.hive.ql.exec.MapOperator.toErrorMessage(MapOperator.java:588)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:557)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
 ]
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row [Error getting row data with exception java.lang.ArrayIndexOutOfBoundsException: -1746617499
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.readVInt(LazyBinaryUtils.java:314)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:183)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:202)
    at org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
    at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:354)
    at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:198)
    at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:184)
    at org.apache.hadoop.hive.ql.exec.MapOperator.toErrorMessage(MapOperator.java:588)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:557)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
 ]
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:562)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
    ... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: -1746617499
    at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:403)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
    at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
    at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:167)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
    ... 9 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1746617499
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.readVInt(LazyBinaryUtils.java:314)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:183)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:202)
    at org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
    at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94)
    at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
    at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
    at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.computeHashCode(ReduceSinkOperator.java:480)
    at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:368)
    ... 13 more

2019-02-28 16:33:47,743 INFO [IPC Server handler 2 on 35318] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1551269222015_0034_m_000000_3: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row [Error getting row data with exception java.lang.ArrayIndexOutOfBoundsException: -1746617499
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.readVInt(LazyBinaryUtils.java:314)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:183)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:202)
    at org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
    at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:354)
    at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:198)
    at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:184)
    at org.apache.hadoop.hive.ql.exec.MapOperator.toErrorMessage(MapOperator.java:588)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:557)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
 ]
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row [Error getting row data with exception java.lang.ArrayIndexOutOfBoundsException: -1746617499
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.readVInt(LazyBinaryUtils.java:314)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:183)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:202)
    at org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
    at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:354)
    at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:198)
    at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:184)
    at org.apache.hadoop.hive.ql.exec.MapOperator.toErrorMessage(MapOperator.java:588)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:557)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
 ]
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:562)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
    ... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: -1746617499
    at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:403)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
    at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
    at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:167)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
    ... 9 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1746617499
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.readVInt(LazyBinaryUtils.java:314)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:183)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:202)
    at org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
    at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94)
    at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
    at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
    at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.computeHashCode(ReduceSinkOperator.java:480)
    at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:368)
    ... 13 more

2019-02-28 16:33:47,744 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1551269222015_0034_m_000000_3: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row [Error getting row data with exception java.lang.ArrayIndexOutOfBoundsException: -1746617499
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.readVInt(LazyBinaryUtils.java:314)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:183)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:202)
    at org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
    at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:354)
    at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:198)
    at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:184)
    at org.apache.hadoop.hive.ql.exec.MapOperator.toErrorMessage(MapOperator.java:588)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:557)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
 ]
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row [Error getting row data with exception java.lang.ArrayIndexOutOfBoundsException: -1746617499
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.readVInt(LazyBinaryUtils.java:314)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:183)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:202)
    at org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
    at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:354)
    at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:198)
    at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:184)
    at org.apache.hadoop.hive.ql.exec.MapOperator.toErrorMessage(MapOperator.java:588)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:557)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
 ]
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:562)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
    ... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: -1746617499
    at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:403)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
    at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
    at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:167)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
    ... 9 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1746617499
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.readVInt(LazyBinaryUtils.java:314)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:183)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142)
    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:202)
    at org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
    at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94)
    at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
    at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
    at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.computeHashCode(ReduceSinkOperator.java:480)
    at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:368)
    ... 13 more

上图显示数据数组下表越界异常！

具体解释：

是因为序列化和反序列化不匹配。
LazyBinarySerDe从之前的MapReduce作业进行的序列化使用了不同的列顺序。当前MapReduce作业对前一个MapReduce作业生成的中间序列文件进行反序列化时，它将使用LazyBinaryStruct使用错误的列顺序从反序列化中获取损坏的数据。序列化和反序列化之间的不匹配列由SelectOperator的Column Pruning ColumnPrunerSelectProc引起。

查询一条数据进行详细查看

2019-02-28 18:57:46,328 FATAL [IPC Server handler 2 on 43230] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1551269222015_0068_m_000000_0 - exited : java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row 
{"_col0":"ab1065d1ca224499a2d78c0cd18eafe3","_col1":"伏自河","_col2":"10100080022","_col3":"10100070009","_col4":"10100030005","_col5":"1975-12-14 15:09:37","_col6":null,"_col7":"0040007\u001coVcvgshLl-C35lN4UAuB6OqQKrAU�@E�\u0000\u0000\u0000\u0000\u0000\u000b10","_col8":"0020002\u00011\u001coftlquLmek909kESJYmVT6BVqAJs\u0006东城\u0006�","_col9":null,"_col10":null,"_col11":null,"_col12":"","_col13":"\u000b10100040007\u001coVcvgshLl-C35lN4UAuB6OqQKrAU�@E�\u0000\u0000\u0000\u0000\u0000\u000b10","_col14":null,"_col21":null,"_col22":"0020002\u00011\u001coftlquLmek909kESJYmVT6BVqAJs\u0006东城\u0006�","_col24":null,"_col30":null,"_col32":null}
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"_col0":"ab1065d1ca224499a2d78c0cd18eafe3","_col1":"伏自河","_col2":"10100080022","_col3":"10100070009","_col4":"10100030005","_col5":"1975-12-14 15:09:37","_col6":null,"_col7":"0040007\u001coVcvgshLl-C35lN4UAuB6OqQKrAU�@E�\u0000\u0000\u0000\u0000\u0000\u000b10","_col8":"0020002\u00011\u001coftlquLmek909kESJYmVT6BVqAJs\u0006东城\u0006�","_col9":null,"_col10":null,"_col11":null,"_col12":"","_col13":"\u000b10100040007\u001coVcvgshLl-C35lN4UAuB6OqQKrAU�@E�\u0000\u0000\u0000\u0000\u0000\u000b10","_col14":null,"_col21":null,"_col22":"0020002\u00011\u001coftlquLmek909kESJYmVT6BVqAJs\u0006东城\u0006�","_col24":null,"_col30":null,"_col32":null}
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:562)
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
	... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException
	at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:403)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
	at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
	at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:167)
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
	... 9 more
Caused by: java.lang.ArrayIndexOutOfBoundsException
	at java.lang.System.arraycopy(Native Method)
	at org.apache.hadoop.io.Text.set(Text.java:225)
	at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48)
	at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:267)
	at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:204)
	at org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
	at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94)
	at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
	at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
	at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.makeValueWritable(ReduceSinkOperator.java:558)
	at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:383)
	... 13 more

2019-02-28 18:57:46,328 INFO [IPC Server handler 2 on 43230] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1551269222015_0068_m_000000_0: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"_col0":"ab1065d1ca224499a2d78c0cd18eafe3","_col1":"伏自河","_col2":"10100080022","_col3":"10100070009","_col4":"10100030005","_col5":"1975-12-14 15:09:37","_col6":null,"_col7":"0040007\u001coVcvgshLl-C35lN4UAuB6OqQKrAU�@E�\u0000\u0000\u0000\u0000\u0000\u000b10","_col8":"0020002\u00011\u001coftlquLmek909kESJYmVT6BVqAJs\u0006东城\u0006�","_col9":null,"_col10":null,"_col11":null,"_col12":"","_col13":"\u000b10100040007\u001coVcvgshLl-C35lN4UAuB6OqQKrAU�@E�\u0000\u0000\u0000\u0000\u0000\u000b10","_col14":null,"_col21":null,"_col22":"0020002\u00011\u001coftlquLmek909kESJYmVT6BVqAJs\u0006东城\u0006�","_col24":null,"_col30":null,"_col32":null}
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"_col0":"ab1065d1ca224499a2d78c0cd18eafe3","_col1":"伏自河","_col2":"10100080022","_col3":"10100070009","_col4":"10100030005","_col5":"1975-12-14 15:09:37","_col6":null,"_col7":"0040007\u001coVcvgshLl-C35lN4UAuB6OqQKrAU�@E�\u0000\u0000\u0000\u0000\u0000\u000b10","_col8":"0020002\u00011\u001coftlquLmek909kESJYmVT6BVqAJs\u0006东城\u0006�","_col9":null,"_col10":null,"_col11":null,"_col12":"","_col13":"\u000b10100040007\u001coVcvgshLl-C35lN4UAuB6OqQKrAU�@E�\u0000\u0000\u0000\u0000\u0000\u000b10","_col14":null,"_col21":null,"_col22":"0020002\u00011\u001coftlquLmek909kESJYmVT6BVqAJs\u0006东城\u0006�","_col24":null,"_col30":null,"_col32":null}
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:562)
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
	... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException
	at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:403)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
	at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
	at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:167)
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
	... 9 more
Caused by: java.lang.ArrayIndexOutOfBoundsException
	at java.lang.System.arraycopy(Native Method)
	at org.apache.hadoop.io.Text.set(Text.java:225)
	at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48)
	at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:267)
	at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:204)
	at org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
	at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94)
	at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
	at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
	at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.makeValueWritable(ReduceSinkOperator.java:558)
	at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:383)
	... 13 more

2019-02-28 18:57:46,330 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1551269222015_0068_m_000000_0: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"_col0":"ab1065d1ca224499a2d78c0cd18eafe3","_col1":"伏自河","_col2":"10100080022","_col3":"10100070009","_col4":"10100030005","_col5":"1975-12-14 15:09:37","_col6":null,"_col7":"0040007\u001coVcvgshLl-C35lN4UAuB6OqQKrAU�@E�\u0000\u0000\u0000\u0000\u0000\u000b10","_col8":"0020002\u00011\u001coftlquLmek909kESJYmVT6BVqAJs\u0006东城\u0006�","_col9":null,"_col10":null,"_col11":null,"_col12":"","_col13":"\u000b10100040007\u001coVcvgshLl-C35lN4UAuB6OqQKrAU�@E�\u0000\u0000\u0000\u0000\u0000\u000b10","_col14":null,"_col21":null,"_col22":"0020002\u00011\u001coftlquLmek909kESJYmVT6BVqAJs\u0006东城\u0006�","_col24":null,"_col30":null,"_col32":null}
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"_col0":"ab1065d1ca224499a2d78c0cd18eafe3","_col1":"伏自河","_col2":"10100080022","_col3":"10100070009","_col4":"10100030005","_col5":"1975-12-14 15:09:37","_col6":null,"_col7":"0040007\u001coVcvgshLl-C35lN4UAuB6OqQKrAU�@E�\u0000\u0000\u0000\u0000\u0000\u000b10","_col8":"0020002\u00011\u001coftlquLmek909kESJYmVT6BVqAJs\u0006东城\u0006�","_col9":null,"_col10":null,"_col11":null,"_col12":"","_col13":"\u000b10100040007\u001coVcvgshLl-C35lN4UAuB6OqQKrAU�@E�\u0000\u0000\u0000\u0000\u0000\u000b10","_col14":null,"_col21":null,"_col22":"0020002\u00011\u001coftlquLmek909kESJYmVT6BVqAJs\u0006东城\u0006�","_col24":null,"_col30":null,"_col32":null}
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:562)
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
	... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException
	at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:403)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
	at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
	at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:167)
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
	... 9 more
Caused by: java.lang.ArrayIndexOutOfBoundsException
	at java.lang.System.arraycopy(Native Method)
	at org.apache.hadoop.io.Text.set(Text.java:225)
	at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48)
	at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:267)
	at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:204)
	at org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
	at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94)
	at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
	at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
	at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.makeValueWritable(ReduceSinkOperator.java:558)
	at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:383)
	... 13 more

2019-02-28 18:57:46,331 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1551269222015_0068_m_000000_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2019-02-28 18:57:46,332 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e25_1551269222015_0068_01_000002 taskAttempt attempt_1551269222015_0068_m_000000_0
2019-02-28 18:57:46,332 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1551269222015_0068_m_000000_0
2019-02-28 18:57:46,332 INFO [ContainerLauncher #1] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : bigdata002:45454
2019-02-28 18:57:46,346 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1551269222015_0068_m_000000_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2019-02-28 18:57:46,347 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2019-02-28 18:57:46,348 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1551269222015_0068_m_000000_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2019-02-28 18:57:46,353 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved bigdata002 to /default-rack
2019-02-28 18:57:46,353 INFO [Thread-53] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node bigdata002
2019-02-28 18:57:46,354 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1551269222015_0068_m_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED
2019-02-28 18:57:46,355 INFO [Thread-53] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1551269222015_0068_m_000000_1 to list of failed maps
2019-02-28 18:57:46,757 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:4 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:1 RackLocal:0
2019-02-28 18:57:46,762 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1551269222015_0068: ask=1 release= 0 newContainers=0 finishedContainers=1 resourcelimit=<memory:45056, vCores:1> knownNMs=3
2019-02-28 18:57:46,762 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e25_1551269222015_0068_01_000002
2019-02-28 18:57:46,763 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:45056, vCores:1>
2019-02-28 18:57:46,763 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1551269222015_0068_m_000000_0: Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

将第二行ETL后的数据进行查询出来，由原先条件可知，数据字段为 32字段

{
"_col0":"ab1065d1ca224499a2d78c0cd18eafe3",
"_col1":"伏自河",
"_col2":"10100080022",
"_col3":"10100070009",
"_col4":"10100030005",
"_col5":"1975-12-14 15:09:37",
"_col6":null,
"_col7":"0040007\u001coVcvgshLl-C35lN4UAuB6OqQKrAU�@E�\u0000\u0000\u0000\u0000\u0000\u000b10",
"_col8":"0020002\u00011\u001coftlquLmek909kESJYmVT6BVqAJs\u0006东城\u0006�",
"_col9":null,
"_col10":null,
"_col11":null,
"_col12":"",
"_col13":"\u000b10100040007\u001coVcvgshLlC35lN4UAuB6OqQKrAU�@E�\u0000\u0000\u0000\u0000\u0000\u000b10",
"_col14":null,
"_col21":null,
"_col22":"0020002\u00011\u001coftlquLmek909kESJYmVT6BVqAJs\u0006东城\u0006�",
"_col24":null,
"_col30":null,
"_col32":null}

由上图正向序列化可知： 序列化时，列裁剪字段丢失，从col15 ——> col21之间字段丢失，数据同时也丢失了！！！

 

发生原因：使用Hive做多表连接(本人使用5表连接)，同时使用limit 方法，即发生了如下问题，可以有一间接解决方案，将5表连接缩小，使用中间表方式，创建2-3表连接情况，即可间接解决此问题，但是并不能完全解决这个Bug!!!

详情见官方说明：https://issues.apache.org/jira/browse/HIVE-14564

wunanliu
关注
  • 2
    点赞
  • 踩
  • 4
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Error解决：hive中的数组越界异常IndexOutOfBoundsException
小平子的专栏
12-12 7985
在spark中执行sql语句有时会出现IndexOutOfBoundsException的错误，如下： File "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera2-1.cdh5.13.3.p0.316101/lib/spark2/python/lib/pyspark.zip/pyspark/sql/session.py", line 708, in sql...
hive中的 数组越界异常 IndexOutOfBoundsException
sinat_27047149的博客
07-21 1万+
java.lang.RuntimeException: Hive Runtime Error while closing operators at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.jav
hive：Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processin
qq_43688472的博客
10-18 3104
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row at org.apache.hadoop.hive.ql.exec.MapOperator.process(
已解决java.lang.IndexOutOfBoundsException异常的正确解决方法，亲测有效！！！
最新发布
小明的Java问道之路
05-09 3194
已解决java.lang.IndexOutOfBoundsException异常的正确解决方法，亲测有效！！！
hive添加字段后的查询异常ArrayIndexOutOfBoundsException
qq_30868737的博客
06-15 2573
hive添加字段 alter table ods_wsp_dw_fact_point add columns (ods_datecreated STRING COMMENT ‘ods数据创建时间’); 客户端查询报错 hue端查询报错 Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask 解决方案： 执行sql前，
ArrayIndexOutOfBoundsException 异常报错原因及解决方案
weixin_46030002的博客
08-07 6万+
ArrayIndexOutOfBoundsException 异常报错原因及解决方案
hive ArrayIndexOutOfBoundsException
zhouzhoumi的专栏
03-09 1952
环境：hive on mr 异常： 2021-03-09 10:12:36,807 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row at org.apache.hado
HIVE执行过程中java.lang.ArrayIndexOutOfBoundsException【待解决】
zoe_ranxiaosu的博客
05-17 2778
HIVE执行过程中java.lang.ArrayIndexOutOfBoundsException
FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeExcepti
01-07
FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient 解决方法 ...
启动tomcat时 错误: 代理抛出异常 : java.rmi.server.ExportException: Port already in use: 1099的解决办法
01-10
　错误: 代理抛出异常错误: java.rmi.server.ExportException: Port already in use: 1099; nested exception is: java.net.BindException: Address already in use: JVM_Bind 　这里说的是1099端口被其它进程占用...
java.net.SocketException: Connection reset 解决方法
09-05
Java中的`java.net.SocketException: Connection reset`是一个常见的网络编程错误，通常表示在TCP/IP通信过程中，连接突然中断。这个异常可能在客户端或服务器端发生，通常与数据传输的异常中断有关。 首先，我们来...
hive-1.1.0-cdh5.14.2.tar.gz和mysql-connector-java-5.1.38.jar连接包
06-26
3. **CDH**: Cloudera Data Hub（CDH）是一个预打包的Hadoop发行版，包含多个开源项目，如Hadoop、Hive、Pig、HBase等，提供了一站式的解决方案，简化了大数据集群的部署和管理。 4. **MySQL元数据存储**: 在Hive中...
hive执行sql异常 ------ java.lang.ArrayIndexOutOfBoundsException: 28
wflh323的专栏
02-24 5225
用户在集群上执行sql报如下异常： 2018-06-06 16:01:29,176 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive ...
hive 读取sparksql的orc文件报ArrayIndexOutOfBoundsException：6
wang972779876的博客
10-28 1764
报错日志如下 可以看出报错的地方再OrcFile的WriterVersion的from方法，定位代码发生在下面代码的values[val]代码段。values的值从代码可以看出values的数据应该为5，见下图代码 而传过来的val是6，向上看代码，version来源是文件的meta，应该是spark版本的原因导致hive不支持的version。 解决 修改OrcFile文件的WriterVersion的内部类的from方法： public static WriterVersio
hive 分区表全量count(1)出现错误，单独查分区没问题
a312428841的博客
12-04 746
hive 分区表全量count(1)出现错误，单独查分区没问题 CDH5 中 hive1.1 版本，当修改表格的元数据信息后，使用ORC格式的文件与Hive的矢量化特性不兼容 ，需要先设置矢量化为false set hive.vectorized.execution.enabled=false; set hive.vectorized.execution.reduce.enabled=false; ...
ArrayIndexOutOfBoundsException-数组越界问题
清流的博客
04-15 8596
1.int []a = new int[10];    a.length = 10,a的最大下标为9    在进行循环输出时，循环控制部分应为i&lt;a.length 2.int []a = new int[10];  int []b = new int[10];        if(i&lt;L.length&amp;&amp;j&lt;R.length&amp;&amp;L[i]&lt;=R...
华为OD机试C、D卷 - 快递员的烦恼（Java & JS & Python & C & C++）
热门推荐
03-22 8万+
快递公司每日早晨，给每位快递员推送需要送到客户手中的快递以及路线信息，快递员自己又查找了一些客户与客户之间的路线距离信息，请你依据这些信息，给快递员设计一条最短路径，告诉他最短路径的距离。 注意： 不限制快递包裹送到客户手中的顺序，但必须保证都送到客户手中 用例保证一定存在投递站到每位客户之间的路线，但不保证客户与客户之间有路线，客户位置及投递站均允许多次经过 所有快递送完后，快递员需回到投递站 ## 输入描述 首行输入两个正整数n、m 接下来 n 行，输入快递公司发布的客户快递信息，
关于执行SQL报错：java.lang.ArrayIndexOutOfBoundsException: 6
weixin_45901516的博客
04-14 4389
关于执行SQL报错：java.lang.ArrayIndexOutOfBoundsException: 6 由于工作需要，需创建一个表C，表数据由表A left join 表B生成，表A数据量大概300万行，表B数据量大概600万行左右，字段不多，但是脚本执行一直报错 一开始报错，查询日志提示信息为：return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask 百度查询具体原因要查看详细日志，打印日志发现提示内存溢出，但是2表的数据量不应该导
虚拟机hive报错Caused by: java.lang.ClassNotFoundException: scala.collection.Iterable的具体解决步骤
06-09
在使用虚拟机中的 Hive 时，如果出现 "Caused by: java.lang.ClassNotFoundException: scala.collection.Iterable" 的错误，可能是因为缺少 Scala 库的依赖导致的。以下是一些可能的解决步骤： 1. 确认 Scala 是否已安装：在虚拟机中执行 scala -version 命令，如果显示了 Scala 的版本信息，则说明已经安装了 Scala。 2. 确认 Hive 配置文件中的 Scala 库路径是否正确：在 Hive 配置文件 hive-site.xml 中查找 scala-library.jar 所在的路径，确保路径正确。 3. 下载并安装 Scala 库：如果在虚拟机中未安装 Scala，可以通过以下命令安装： ``` sudo apt-get update sudo apt-get install scala ``` 4. 添加 Scala 库的依赖：在 Hive 配置文件 hive-site.xml 中添加以下配置： ``` <property> <name>hive.aux.jars.path</name> <value>/usr/share/java/scala-library.jar</value> </property> ``` 这将在 Hive 启动时将 Scala 库添加到 CLASSPATH 中。 5. 重新启动 Hive：在进行上述操作后，需要重新启动 Hive 才能使更改生效。 希望这些步骤可以帮助你解决问题。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值