线上一个ETL Job不能跑了,报异常,这里为了说明问题简化表结构:
hive>desc void_t;
OK
x int None
z void None
而
select* from void_t
确实会抛出异常:
14/03/0201:28:58 ERROR CliDriver: Failed with exceptionjava.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: Errorevaluating x
看到这个异常很疑惑,和x字段有什么关系呢,继续看详细日志:
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating x
atorg.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150)
atorg.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1412)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
atorg.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
atorg.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
atorg.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
atsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
atjava.lang.reflect.Method.invoke(Method.java:597)
atorg.apache.hadoop.util.RunJar.main(RunJar.java:208)
Causedby: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating x
atorg.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:80)
atorg.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
atorg.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
atorg.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
atorg.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:490)
atorg.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
... 11 more
Causedby: java.lang.RuntimeException: Internal error: no LazyObject for VOID
atorg.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyPrimitiveClass(LazyFactory.java:119)
atorg.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObject(LazyFactory.java:155)
at org.apache.hadoop.hive.serde2.lazy.LazyStruct.parse(LazyStruct.java:108)
atorg.apache.hadoop.hive.serde2.lazy.LazyStruct.getField(LazyStruct.java:190)
atorg.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldData(LazySimpleStructObjectInspector.java:188)
atorg.apache.hadoop.hive.serde2.objectinspector.DelegatedStructObjectInspector.getStructFieldData(DelegatedStructObjectInspector.java:79)
atorg.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.evaluate(ExprNodeColumnEvaluator.java:98)
atorg.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:76)
看到这个noLazyObject for VOID才知道原来问题出现在这里,也就是字段z上;查看ETL Job里的Query发现里面一个建表的语句用到了create table xxx as select null as z from xxx这样的方式,进而生成了一个VOID类型的字段,但是Hive本身却无法处理该字段,在jira里确实也有这么一个unresolved的Bug:HIVE-2615
Workaround也比较简单:1.先建表再insert select 2.在ctas时cast(null as <type>) z来强制指定类型.
转载于:https://blog.51cto.com/boylook/1365747