今天在通过jdbc执行hql时出现了一下错误:
2015-11-23 19:55:39,515 FATAL [IPC Server handler 9 on 46637] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1446869087133_1399_m_000000_0 - exited : java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:449) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 17 more Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:147) ... 22 more Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyLongObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.primitive.SettableLongObjectInspector at org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory.genVectorExpressionWritable(VectorExpressionWriterFactory.java:410) at org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory.processVectorInspector(VectorExpressionWriterFactory.java:1102) at org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.initializeOp(VectorReduceSinkOperator.java:55) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425) at org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.initializeOp(VectorFilterOperator.java:77) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126) ... 22 more
是说类型转换错误,但是如果我把hql直接copy到hive-cli中运行没有问题,查了一下午也没有找到原因。
这条语句之前运行的很好,因为hql中查询的表都是分区表,之前没有加分区的条件,今天加上分区的限制条件之后就出现了以上的问题,因为这条hql中有多个表连接,我就把各个字句中的分区条件先去掉,然后在一次加一个,加完后通过jdbc执行,终于定位到是那一条字句有问题:
select unique_id,getAccount(additional_type,additional_value) as accounts from (select unique_id,additional_type,additional_value from(select unique_id,additional_type,additional_value,row_number() over (distribute by unique_id,additional_type sort by time_upd DESC) rownum from dws_i_appserver_cust_additional_info where date='20151123')addinfo where rownum<=1)ac group by unique_id 其中红色部分既有分区限制date='20151123',又有row_number,按说这样写是没有问题的,因为在hive-cli中跑没有问题,可能是hiveserver2中某个配置有问题,但是没有找到原因,改成下列形式后就没有问题了:
select unique_id,concat_ws('&&',collect_set(concat(additional_type,'||',additional_value))) as accounts from (select unique_id,additional_type,additional_value from(select unique_id,additional_type,additional_value,row_number() over (distribute by unique_id,additional_type sort by time_upd DESC) rownum from (select unique_id,additional_type,additional_value,time_upd from dws_i_appserver_cust_additional_info where date='20151123')rowinfo )addinfo where rownum<=1)ac group by unique_id。
不知道有没有大神能给解答一下?!