Hive数据仓库--HiveUDF异常解析与其中的坑

使用Hive进行分析的过程中遇到了这个异常的信息，一开始看到比较惊讶，因为感觉代码并没有什么不太合理的地方，不过还是抛出了一堆的异常信息，我把日志信息读了一下，发现导致最终异常的是FastJson在封装的时候找不到无参构造函数，虽然最终原因不是Hive引起的，但是上面的一堆异常也的确是唬人的。

异常信息

Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"sourceid":"4649475029B8","sourcelabelw":"[{\"Name\":\"XX\",\"Weight\":6},{\"Name\":\"XX\",\"Weight\":6}]"}
        at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"sourceid":"4649475029B8","sourcelw":"[{\"Name\":\"XX\",\"Weight\":6},{\"Name\":\"XXX\",\"Weight\":6}]"}
        at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:550)
        at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
        ... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String cn.siti.main.MergeUDF.evaluate(java.lang.String,java.lang.String)  on object cn.siti.main.MergeUDF@9fd062 of class cn.siti.main.MergeUDF with arguments {[{"Name":"XXX","Weight":6},{"Name":"XXX","Weight":6}]:java.lang.String, XXXX,7:java.lang.String} of size 2
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1243)
        at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:182)
        at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
        at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
        at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
        at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
        at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.internalForward(CommonJoinOperator.java:638)
        at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
        at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:754)
        at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:242)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
        at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
        at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:540)
        ... 9 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1219)
        ... 23 more
Caused by: com.alibaba.fastjson.JSONException: default constructor not found. class cn.siti.domain.TagInfo
        at com.alibaba.fastjson.util.JavaBeanInfo.build(JavaBeanInfo.java:212)
        at com.alibaba.fastjson.parser.ParserConfig.createJavaBeanDeserializer(ParserConfig.java:486)
        at com.alibaba.fastjson.parser.ParserConfig.getDeserializer(ParserConfig.java:433)
        at com.alibaba.fastjson.parser.ParserConfig.getDeserializer(ParserConfig.java:288)
        at com.alibaba.fastjson.parser.DefaultJSONParser.parseArray(DefaultJSONParser.java:651)
        at com.alibaba.fastjson.parser.DefaultJSONParser.parseArray(DefaultJSONParser.java:630)
        at com.alibaba.fastjson.parser.DefaultJSONParser.parseArray(DefaultJSONParser.java:625)
        at com.alibaba.fastjson.JSON.parseArray(JSON.java:491)
        at cn.siti.main.MergeUDF.evaluate(MergeUDF.java:21)
        ... 28 more

在使用的过程中请注意这一点，Hive中报错的时候，先看日志，从上往下找最终导致异常的地方，然后进行解决。

此外有时候会遇到一个很无奈的问题

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask 

 There are 3 datanode(s) running and no node(s) are excluded in this operation

本次运行中出现了这样的问题，我这边造成问题的主要原因是：nodemanager停止了，然后任务分发的时候出现问题。

重新启动yarn即可

转载注明出处：Hive数据仓库--HiveUDF异常解析与其中的坑