spark调用hiveUDTF踩坑记录

最新推荐文章于 2023-08-03 13:15:00 发布
最新推荐文章于 2023-08-03 13:15:00 发布
阅读量2.2k 收藏
点赞数 4
文章标签： spark 大数据 hive
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_40206335/article/details/106591726
版权

由于业务需求需要使用UDTF处理数据
公司spark版本为2.2

udtf代码如下：

package com.sankuai;

import com.google.common.collect.Lists;
import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

public class UserInPairsUdtf extends GenericUDTF {
	private List<String> colName = Lists.newLinkedList();
    private List<ObjectInspector> resType = Lists.newLinkedList();
     @Override
    public StructObjectInspector initialize(StructObjectInspector argOIs) throws UDFArgumentException {
        colName.add("fieldName1");
        colName.add("fieldName2");
        colName.add("fieldName3");
        resType.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
        resType.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
        resType.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
        // 返回分别为列名 和 列类型
        return ObjectInspectorFactory.getStandardStructObjectInspector(colName, resType);
    }

    /**
     * 此方法入参为两个
     * 由于涉及公司业务不方便展示各位看客凑活着看
    * */
    @Override
    public void process(Object[] objects) throws HiveException {
       .................................................................
            }

        }
    }

    @Override
    public void close() throws HiveException {

    }
}

将此代码打包到HDFS
**当执行引擎为spark时执行命令：
add jar /user_path/udtf.jar;
CREATE TEMPORARY FUNCTION founction AS ‘/com/sankuai/udtf’;
报如下错误：

20/06/06 15:23:49 ERROR Client: User class threw exception: org.apache.spark.sql.AnalysisException: No handler for Hive UDF 'com.sankuai.Udtf': java.lang.IllegalStateException: Should not be called directly; line 4 pos 7org.apache.spark.sql.AnalysisException: No handler for Hive UDF 'com.sankuai.Udtf': java.lang.IllegalStateException: Should not be called directly; line 4 pos 7
	at org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.initialize(GenericUDTF.java:72)
	at org.apache.spark.sql.hive.HiveGenericUDTF.outputInspector$lzycompute(hiveUDFs.scala:207)
	at org.apache.spark.sql.hive.HiveGenericUDTF.outputInspector(hiveUDFs.scala:207)
	at org.apache.spark.sql.hive.HiveGenericUDTF.elementSchema$lzycompute(hiveUDFs.scala:215)
	at org.apache.spark.sql.hive.HiveGenericUDTF.elementSchema(hiveUDFs.scala:215)
	at org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$makeFunctionBuilder$1.apply(HiveSessionCatalog.scala:96)
	at org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$makeFunctionBuilder$1.apply(HiveSessionCatalog.scala:72)
	at org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry.lookupFunction(FunctionRegistry.scala:93)
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction(SessionCatalog.scala:1366)
	at org.apache.spark.sql.hive.HiveSessionCatalog.org$apache$spark$sql$hive$HiveSessionCatalog$$super$lookupFunction(HiveSessionCatalog.scala:130)
	at org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$3.apply(HiveSessionCatalog.scala:130)
	at org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$3.apply(HiveSessionCatalog.scala:130)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.spark.sql.hive.HiveSessionCatalog.lookupFunction0(HiveSessionCatalog.scala:130)
	at org.apache.spark.sql.hive.HiveSessionCatalog.lookupFunction(HiveSessionCatalog.scala:123)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$16$$anonfun$applyOrElse$7$$anonfun$applyOrElse$54.apply(Analyzer.scala:1198)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$16$$anonfun$applyOrElse$7$$anonfun$applyOrElse$54.apply(Analyzer.scala:1198)
	at org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:48)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$16$$anonfun$applyOrElse$7.applyOrElse(Analyzer.scala:1197)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$16$$anonfun$applyOrElse$7.applyOrElse(Analyzer.scala:1180)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:266)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:272)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:272)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:272)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsDown$1.apply(QueryPlan.scala:79)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsDown$1.apply(QueryPlan.scala:79)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:100)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:110)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1$1.apply(QueryPlan.scala:114)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.immutable.List.map(List.scala:285)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:114)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:119)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:119)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsDown(QueryPlan.scala:79)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressions(QueryPlan.scala:70)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$16.applyOrElse(Analyzer.scala:1180)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$16.applyOrElse(Analyzer.scala:1178)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:62)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:62)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:61)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$1.apply(LogicalPlan.scala:59)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$1.apply(LogicalPlan.scala:59)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:59)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$.apply(Analyzer.scala:1178)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$.apply(Analyzer.scala:1170)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:85)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:82)
	at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
	at scala.collection.immutable.List.foldLeft(List.scala:84)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:82)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:74)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:74)
	at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:71)
	at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:69)
	at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:52)
	at org.apache.spark.sql.Dataset$$anonfun$ofRows$1.apply(Dataset.scala:68)
	at org.apache.spark.sql.Dataset$$anonfun$ofRows$1.apply(Dataset.scala:66)
	at org.apache.spark.sql.execution.SQLExecution$.withNewQueryId(SQLExecution.scala:56)
	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:66)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:639)
	at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:691)
	at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
	at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:377)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:386)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:321)
	at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:419)
	at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:435)
	at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:207)
	at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:660)

20/06/06 15:23:49 INFO Client: resolveAppExceptionMsg, msg start
20/06/06 15:23:49 INFO Client: resolveAppExceptionMsg, msg:User class threw exception: org.apache.spark.sql.AnalysisException: No handler for Hive UDF 'com.sankuai.Udtf': java.lang.IllegalStateException: Should not be called directly; line 4 pos 7org.apache.spark.sql.AnalysisException: No handler for Hive UDF 'com.sankuai.Udtf': java.lang.IllegalStateException: Should not be called directly; line 4 pos 7
	at org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.initialize(GenericUDTF.java:72)
	at org.apache.spark.sql.hive.HiveGenericUDTF.outputInspector$lzycompute(hiveUDFs.scala:207)
	at org.apache.spark.sql.hive.HiveGenericUDTF.outputInspector(hiveUDFs.scala:207)
	at org.apache.spark.sql.hive.HiveGenericUDTF.elementSchema$lzycompute(hiveUDFs.scala:215)
	at org.apache.spark.sql.hive.HiveGenericUDTF.elementSchema(hiveUDFs.scala:215)
	at org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$makeFunctionBuilder$1.apply(HiveSessionCatalog.scala:96)
	at org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$makeFunctionBuilder$1.apply(HiveSessionCatalog.scala:72)
	at org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry.lookupFunction(FunctionRegistry.scala:93)
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction(SessionCatalog.scala:1366)
	at org.apache.spark.sql.hive.HiveSessionCatalog.org$apache$spark$sql$hive$HiveSessionCatalog$$super$lookupFunction(HiveSessionCatalog.scala:130)
	at org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$3.apply(HiveSessionCatalog.scala:130)
	at org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$3.apply(HiveSessionCatalog.scala:130)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.spark.sql.hive.HiveSessionCatalog.lookupFunction0(HiveSessionCatalog.scala:130)
	at org.apache.spark.sql.hive.HiveSessionCatalog.lookupFunction(HiveSessionCatalog.scala:123)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$16$$anonfun$applyOrElse$7$$anonfun$applyOrElse$54.apply(Analyzer.scala:1198)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$16$$anonfun$applyOrElse$7$$anonfun$applyOrElse$54.apply(Analyzer.scala:1198)
	at org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:48)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$16$$anonfun$applyOrElse$7.applyOrElse(Analyzer.scala:1197)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$16$$anonfun$applyOrElse$7.applyOrElse(Analyzer.scala:1180)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:266)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:272)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:272)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:272)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsDown$1.apply(QueryPlan.scala:79)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsDown$1.apply(QueryPlan.scala:79)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:100)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:110)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1$1.apply(QueryPlan.scala:114)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.immutable.List.map(List.scala:285)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:114)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:119)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:119)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsDown(QueryPlan.scala:79)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressions(QueryPlan.scala:70)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$16.applyOrElse(Analyzer.scala:1180)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$16.applyOrElse(Analyzer.scala:1178)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:62)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:62)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:61)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$1.apply(LogicalPlan.scala:59)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$1.apply(LogicalPlan.scala:59)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:59)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$.apply(Analyzer.scala:1178)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$.apply(Analyzer.scala:1170)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:85)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:82)
	at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
	at scala.collection.immutable.List.foldLeft(List.scala:84)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:82)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:74)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:74)
	at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:71)
	at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:69)
	at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:52)
	at org.apache.spark.sql.Dataset$$anonfun$ofRows$1.apply(Dataset.scala:68)
	at org.apache.spark.sql.Dataset$$anonfun$ofRows$1.apply(Dataset.scala:66)
	at org.apache.spark.sql.execution.SQLExecution$.withNewQueryId(SQLExecution.scala:56)
	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:66)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:639)
	at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:691)
	at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
	at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:377)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:386)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:321)
	at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:419)
	at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:435)
	at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:207)
	at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:660)

20/06/06 15:23:49 INFO Client: resolveAppExceptionMsg, msg end
Exception in thread "main" org.apache.spark.SparkException: Application application_1591139964203_1781709 finished with failed status
	at org.apache.spark.deploy.yarn.Client.run(Client.scala:1266)
	at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1378)
	at org.apache.spark.deploy.yarn.Client.main(Client.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1242)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:207)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:232)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:146)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

当执行引擎为hive时，没有有报错
仔细研究发现
GenericUDTF中有两个initialize方法:
1.spark中用的已经过时的initialize(ObjectInspector[] argOIs),且如调用该方法会直接抛异常，所以必须实现
2.我实现的确实initialize(StructObjectInspector argOIs)，因此，我的方法并不会被调用。
在这里插入图片描述
将代码修改如下：

package com.sankuai;

import com.google.common.collect.Lists;
import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

public class UserInPairsUdtf extends GenericUDTF {
	private List<String> colName = Lists.newLinkedList();
    private List<ObjectInspector> resType = Lists.newLinkedList();
     @Override
    public StructObjectInspector initialize(ObjectInspector[] argOIs) throws UDFArgumentException {
        colName.add("fieldName1");
        colName.add("fieldName2");
        colName.add("fieldName3");
        resType.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
        resType.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
        resType.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
        // 返回分别为列名 和 列类型
        return ObjectInspectorFactory.getStandardStructObjectInspector(colName, resType);
    }

    /**
     * 此方法入参为两个
     * 由于涉及公司业务不方便展示各位看客凑活着看
    * */
    @Override
    public void process(Object[] objects) throws HiveException {
       .................................................................
            }

        }
    }

    @Override
    public void close() throws HiveException {

    }
}

解决问题

MT_LIDELIN
关注
  • 4
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
Spark二次调用Hive UDF异常问题解决办法
dingyufei的博客
02-21 7474
异常如下： No handler for UDF/UDAF/UDTF 'com.******.***': java.lang.ClassNotFoundException: com.***; line 1 pos 7 org.apache.spark.sql.AnalysisException: No handler for UDF/UDAF/UDTF 'com.***': java.lang...
SparkSql之UDF、UDAF、UDTF
莫慌 吃口药
10-11 601
UDF---------------------------------------- 完整的示例：object SparkSQL { def main(args:Array[String]):Unit = { //创建SparkConf()并设置App名称 val conf = new SparkConf().setAppName("SparkSQLDemo").setMa...
hive-udf:NexR Hive UDF
05-17
NexR Hive UDF 关于 NexR Hive UDF是Hive用户定义功能的集合。 执照 快速开始 $ git clone https://github.com/nexr/hive-udf.git $ cd hive-udf $ mvn clean package or $ ant -Dhive.install.dir=../hive/build/dist clean package 使用Hive PDK运行测试 $ ant -Dhive.install.dir=../hive/build/dist test 安装和配置 hive-site.xml <property> <name>hive.aux.jars.path</name> <value>file:///path/to/nexr-hive-udf-[VERSION].jar</value> </pr
HIVE UDTF JAVA开发
hxtog的博客
08-21 641
1. Hive UDF编程 继承GenericUDTF类 实现三个方法(initialize, process, 和 close) 结果写入forward方法 2. 需实现的三个方法说明 initialize 方法说明: 用于校验输入参数类型, 指定输出结果类型. 运行: 在单节点运行周期中内执行一次, 且是在最开始执行. 两个initialize比较: StructObjectInspector initialize(StructObjectInspector argOIs) publi
简单谈谈hive，spark-sql，hive on spark中udf自定义函数的使用
Bayerngu的博客
06-19 3515
hive中使用udf 首先如果在hive中使用udf函数的话，如果使用java编写udf，则需要导入的包： import org.apache.hadoop.hive.ql.exec.UDF; //这里的1代表参数个数 import org.apache.spark.sql.api.java.UDF1; 比如创建一个udf： import org.apache.hadoop.hive.serde2.io.DoubleWritable; import org.apache.hadoop.io.IntWri
Spark踩坑记——SparkStreaming+Kafka
02-25
在WeTest舆情项目中，需要对每天千万级的游戏评论信息进行词频统计，在生产者一端，我们将数据按照每天的拉取时间存入了Kafka当中，而在消费者一端，我们利用了sparkstreaming从kafka中不断拉取数据进行词频统计。...
Spark python API 函数调用学习
08-11
2. **RDD（弹性分布式数据集）**：RDD是Spark中最基本的数据抽象，它是不可变的、分区的记录集合。通过`sc.parallelize()`或`sc.textFile()`方法可以从现有数据源创建RDD。 3. **DataFrame和Dataset**：DataFrame是...
Spark 外部数据源调用代码
05-08
Spark 外部数据源调用代码，CSV文件 和HIVE读取方式。
opencvspark:OpenCV + Spark 示例
06-05
《OpenCV + Spark 实例解析》 在当前的数字化时代，大数据处理与计算机视觉技术的结合日益成为研究和开发的重点。OpenCV（开源计算机视觉库）是计算机视觉领域的重要工具，而Apache Spark则是一个用于大规模数据...
Hive的UDF与UDTF浅析，TopN案例
醉与浮的博客
08-17 410
UDF是User-Define-Function，一般是指一个输入一个输出，UDTF是UDF变种，可一个输入多个输出。UDAF是用户聚合，可以多行输入，一个输出。需要注意的是这些函数写好之后需要在平台或者脚本里配合sql使用。...
spark-hive-udf:Spark Hive UDF示例
04-22
Spark Hive UDF示例 建立项目 mvn clean package 将spark-hive-udf-1.0.0-SNAPSHOT.jar复制到边缘节点临时目录 spark-hive-udf]# cp target/spark-hive-udf-1.0.0-SNAPSHOT.jar /tmp 通过提供罐子来启动火花壳 spark-shell --master yarn --jars /tmp/spark-hive-udf-1.0.0-SNAPSHOT.jar 创建名称为大写的函数并列出该函数 spark.sql("CREATE OR REPLACE FUNCTION uppercase AS 'com.ranga.spark.hive.udf.UpperCaseUDF' USING JAR '/tmp/spark-hive-udf-1.0.0-SNAPSHOT.jar'") sp
SparkStreaming流式计算中使用SparkSQL,并结合HIVE的UDF、UDAF、UDTF的使用
qiangzi_lg的博客
03-17 892
思路: 编写UDF函数,实现hive的UDF接口 创建hiveContext对象,注册编写的UDF函数 编写表的结构 DStream==&gt;RDD==&gt;rowRDD==&gt;DataFrame==&gt;SQL查询 UDF 以监听端口号为例,每隔五秒一个批次 object Test_UDF { def main(args: Array[String]): Unit = { ...
Hive UDF开发教程
ALX3li的博客
05-07 325
Hive UDF(User Defined Function 用户自定义函数)，是在Hive内置的处理函数无法满足用户需求时，由用户自行继承接口实现的单行数据处理函数。Hive 有两个不同的接口编写 UDF 程序。1 简单数据类型的 UDF 接口基础 UDF 的函数读取和返回基本类型，即 Hadoop 和 Hive 的基本类型。如 Text、IntWritable、LongWritable、DoubleWritable 等。2 复杂数据类型的 GenericUDF 接口。GenericUDF 可以。
UDF开发手册 - UDTF
qq_38783098的博客
04-24 750
表生产函数 一行输入，输出多行结果 实现 自定义UDTF 需要继承 GenericUDTF，并且实现其三个方法： initialize() process() close() 其中 process()、close() 为 GenericUDTF 中的抽象方法，必须实现。initialize() 虽然不是抽象方法，但必须手动覆盖实现该方法，因为 GenericUDTF 的 initialize() 最终会抛出一个异常： throw new IllegalStateException("Should not
深入解析java.lang.IllegalStateException异常
最新发布
博客
08-03 4万+
是Java中的一个非受检异常，它表示在不适当的时间或状态下调用方法。日志记录工具：如Log4j、Logback等，可以记录异常信息和错误日志，方便排查和分析问题。断言工具：如Assert类，可以在程序中添加断言语句，用于验证代码中的假设条件是否满足，帮助我们在开发和测试阶段快速发现问题。异常处理框架：如Spring框架中的异常处理机制，可以统一处理应用程序中的异常，提供统一的异常处理逻辑和错误信息返回。异常是程序中常见的错误和异常情况，合理处理异常对于保证程序的正确性和稳定性非常重要。
Spark 训练机器学习模型莫名报错（java.lang.stackoverflow）
记录学习痕迹的公众号：Piper蛋窝
02-09 912
归根结底是当时不了解 Spark 原理。
scala出现异常：Exception in thread "main" java.lang.StackOverflowError
njiang的专栏
01-10 5892
出现异常：Exception in thread "main" java.lang.StackOverflowError
java.lang.IllegalStateException异常:简单分析和简单解决方案
自由时飞扬 搬家拉 http://www.meiriyouke.net
03-03 3099
今天写java文件下载程序，完成后使用一切正常，但是总抛出java.lang.IllegalStateException异常，虽然并不影响正常使用，但看了总让人觉得很不舒服，检查代码并没有错，最后上网查了不少资料，终于发现原因之所在。 我们在做文件上传或者下载，或者过滤等操作时，可能要用到页面的输出流.例如在action中使用：　　　　response.reset();　　　　response....
java.lang.IllegalStateException: safe guard client , should not be called ,must have a bug.
哪惧明天，风高路斜
03-02 1万+
问题： java.lang.IllegalStateException: safe guard client , should not be called ,must have a bug. 解决： 第一种方法： 升级dubbo的版本 第二种方法： 不改变dubbo的版本，修改代码 参考下面的链接 fix consumer warning “safe guard client , should not be called ,must have a bug.” ...
spark调用数据库
02-15
Spark 可以通过 JDBC 连接数据库，并调用数据库中的数据。Spark 可以通过 SparkSession 的 read 方法读取数据库中的数据，然后对数据进行处理。下面是一个从 PostgreSQL 数据库读取数据的示例代码： ``` import org.apache.spark.sql.SparkSession val spark = SparkSession.builder().appName("SparkJDBCExample").getOrCreate() val jdbcDF = spark.read .format("jdbc") .option("url", "jdbc:postgresql://<hostname>:<port>/<dbname>") .option("dbtable", "<tablename>") .option("user", "<username>") .option("password", "<password>") .load() jdbcDF.show() ``` 这段代码会创建一个 SparkSession，然后通过 JDBC 连接到 PostgreSQL 数据库，读取数据库中的数据，并将数据以 DataFrame 的形式展示出来。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值