hive udf kyroexception unable to find class

最新推荐文章于 2024-05-07 16:37:17 发布
最新推荐文章于 2024-05-07 16:37:17 发布
阅读量2.5k 收藏 1
点赞数 1
分类专栏： HADOOP 文章标签： hive udf
原文链接：http://jason4zhu.blogspot.com/2018/04/hive-on-spark-unable-to-find-class.html
版权

H

ive-On-Spark配置成功后，准备试用下Hive UDF是否能在Spark-on-Hive环境下正常使用：

 

set hive.execution.engine=spark;

add jar viewfs:///dirs/brickhouse-0.7.1-SNAPSHOT-jar-with-dependencies.jar;

create temporary function to_json AS 'brickhouse.udf.json.ToJsonUDF';

select to_json(app_metric) as tt from tbl_name where dt = '20180417' limit 10;

但在yarn-cluster模式下执行后会complain如下错误：

 

org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: brickhouse.udf.json.ToJsonUDF

Serialization trace:

genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)

colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)

childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)

aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)

invertedWorkGraph (org.apache.hadoop.hive.ql.plan.SparkWork)

at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)

at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)

at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)

at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:181)

at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)

at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)

at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)

at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176)

at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161)

at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)

at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)

at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:214)

at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)

at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)

at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)

at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176)

at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)

at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)

at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)

at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:214)

at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)

at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)

at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)

at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176)

at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161)

at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)

at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)

at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:214)

at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)

at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)

at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)

at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176)

at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153)

at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)

at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)

at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:214)

at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)

at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)

at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:686)

at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:206)

at org.apache.hadoop.hive.ql.exec.spark.KryoSerializer.deserialize(KryoSerializer.java:60)

at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:329)

at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:358)

at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:323)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.ClassNotFoundException: brickhouse.udf.json.ToJsonUDF

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)

... 47 more

此时如果只看stacktrace，感觉可能是jar文件没有分发到对应的RemoteDriver，导致类找不到。

MARK: debug除了看stacktrace，另一个思路是看看stacktrace上面的INFO。因为有些错误可能是fail-silent, 所以INFO里已经暴露出一些不正确的日志内容可以用于排查问题

拉取部分INFO日志内容如下：

 

18/04/23 18:51:39 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done

18/04/23 18:51:39 INFO spark.SparkContext: Added JAR viewfs://hadoop-lt-cluster/tmp/hive/dp/_spark_session_dir/bc42b5c8-f183-4088-b238-2c3a75725d06/hive-exec-2.3.2.jar at viewfs://hadoop-lt-cluster/tmp/hive/dp/_spark_session_dir/bc42b5c8-f183-4088-b238-2c3a75725d06/hive-exec-2.3.2.jar with timestamp 1524480699144

18/04/23 18:51:39 INFO spark.SparkContext: Added JAR viewfs://hadoop-lt-cluster/tmp/hive/dp/_spark_session_dir/bc42b5c8-f183-4088-b238-2c3a75725d06/kuaishou-analytics-auth-1.0.0.jar at viewfs://hadoop-lt-cluster/tmp/hive/dp/_spark_session_dir/bc42b5c8-f183-4088-b238-2c3a75725d06/kuaishou-analytics-auth-1.0.0.jar with timestamp 1524480699162

18/04/23 18:51:39 INFO spark.SparkContext: Added JAR viewfs://hadoop-lt-cluster/tmp/hive/dp/_spark_session_dir/bc42b5c8-f183-4088-b238-2c3a75725d06/brickhouse-0.7.1-SNAPSHOT-jar-with-dependencies-2.jar at viewfs://hadoop-lt-cluster/tmp/hive/dp/_spark_session_dir/bc42b5c8-f183-4088-b238-2c3a75725d06/brickhouse-0.7.1-SNAPSHOT-jar-with-dependencies-2.jar with timestamp 1524480699165

18/04/23 18:51:39 INFO storage.BlockManagerMasterEndpoint: Registering block manager bjlt-h1180.sy:37446 with 7.0 GB RAM, BlockManagerId(53, bjlt-h1180.sy, 37446)

18/04/23 18:51:39 INFO client.RemoteDriver: Received job request 2d71a807-c512-4032-8c9e-71a378d3168b

18/04/23 18:51:39 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.48.74.35:59232) with ID 41

18/04/23 18:51:39 INFO spark.ExecutorAllocationManager: New executor 41 has registered (new total is 53)

18/04/23 18:51:39 INFO storage.BlockManagerMasterEndpoint: Registering block manager bjlt-h1864.sy:36387 with 7.0 GB RAM, BlockManagerId(41, bjlt-h1864.sy, 36387)

18/04/23 18:51:39 INFO client.SparkClientUtilities: Added jar[file:/media/disk3/yarn_data/usercache/dp/appcache/application_1523431310007_1301182/container_e95_1523431310007_1301182_01_000001/viewfs:/hadoop-lt-cluster/tmp/hive/dp/_spark_session_dir/bc42b5c8-f183-4088-b238-2c3a75725d06/hive-exec-2.3.2.jar] to classpath.

18/04/23 18:51:39 INFO client.SparkClientUtilities: Added jar[file:/media/disk3/yarn_data/usercache/dp/appcache/application_1523431310007_1301182/container_e95_1523431310007_1301182_01_000001/viewfs:/hadoop-lt-cluster/tmp/hive/dp/_spark_session_dir/bc42b5c8-f183-4088-b238-2c3a75725d06/brickhouse-0.7.1-SNAPSHOT-jar-with-dependencies-2.jar] to classpath.

18/04/23 18:51:39 INFO client.SparkClientUtilities: Added jar[file:/media/disk3/yarn_data/usercache/dp/appcache/application_1523431310007_1301182/container_e95_1523431310007_1301182_01_000001/viewfs:/hadoop-lt-cluster/tmp/hive/dp/_spark_session_dir/bc42b5c8-f183-4088-b238-2c3a75725d06/kuaishou-analytics-auth-1.0.0.jar] to classpath.

18/04/23 18:51:39 INFO client.RemoteDriver: Failed to run job 2d71a807-c512-4032-8c9e-71a378d3168b

发现SparkContext的"Added JAR"语句对应的路径是正确的，但SparkClientUtilities对应的"Added JAR"是类似"file:/media/disk3/yarn_data/usercache/dp/appcache/application_1523431310007_1301182/container_e95_1523431310007_1301182_01_000001/viewfs:/hadoop-lt-cluster/tmp/hive/dp/_spark_session_dir/bc42b5c8-f183-4088-b238-2c3a75725d06/hive-exec-2.3.2.jar", 可见路径解析有问题。此时，去Hive源代码中review下发现code snippet如下：

 

private static URL urlFromPathString(String path, Long timeStamp,

Configuration conf, File localTmpDir) {

URL url = null;

try {

if (StringUtils.indexOf(path, "file:/") == 0) {

url = new URL(path);

} else if (StringUtils.indexOf(path, "hdfs:/") == 0) {

Path remoteFile = new Path(path);

Path localFile =

new Path(localTmpDir.getAbsolutePath() + File.separator + remoteFile.getName());

Long currentTS = downloadedFiles.get(path);

if (currentTS == null) {

currentTS = -1L;

}

if (!new File(localFile.toString()).exists() || currentTS < timeStamp) {

LOG.info("Copying " + remoteFile + " to " + localFile);

FileSystem remoteFS = remoteFile.getFileSystem(conf);

remoteFS.copyToLocalFile(remoteFile, localFile);

downloadedFiles.put(path, timeStamp);

}

return urlFromPathString(localFile.toString(), timeStamp, conf, localTmpDir);

} else {

url = new File(path).toURL();

}

} catch (Exception err) {

LOG.error("Bad URL " + path + ", ignoring path", err);

}

return url;

}

可见当前版本的Hive（0.23）只有对hdfs和file两种scheme的支持，不支持viewfs。增加适配viewfs的代码判断：else if (StringUtils.indexOf(path, "hdfs:/") == 0 || StringUtils.indexOf(path, "viewfs:/") == 0 ) {, 问题解决。

玉羽凌风
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Hive UDF开发
12-27
详细介绍如何开发hive自定义永久函数，配套有测试数据
HiveUDF:Apache Hive UDF（用户定义函数）
07-01
Hive UDF UDF 聚合 UDF Finds MIN, MAX and SUM from array of Struct Objects based on a field. 排序 UDF Returns sorted array of Struct objects for an array of Struct Objects based on a field. 日期 UDF（用于查找日期维度、数据仓库概念） Returns date key field, e.g. __FUNC__('2014-04-05T01:30:34Z') ISO 8601 format, __FUNC__('2014-04-05 01:30:34') and __FUNC__(UNIX_TIMESTAMP('2014-04-05 01:30:34')) returns 20140405 时间
hive自定义UDF编写函数.docx
01-26
hive的udf函数实现
Spring Boot Maven Plugin打包异常及三种解决方法：Unable to find main class
05-19 4955
【背景】spring-boot项目，打包成可执行jar，项目内有两个带有main方法的类并且都使用了@SpringBootApplication注解（或者另一种情形：你有两个main方法并且所在类都没有使用@SpringBootApplication注解），pom.xml如下 <plugin> <groupId>org.springframew...
Spark 使用Joda DateTime报序列化异常KryoException
m0_37813354的博客
09-04 1156
我的spark代码里有如下面一段使用了org.joda.time.DateTime var rDate = new LocalDateTime(day, dtz) if (dtz.isLocalDateTimeGap(rDate)) { rDate = rDate.plusHours(1) } val regDay: DateTime = rDate.toDateTime() 运行时偶尔报序列化问题，比较诡异的是不是每次都报，而且调度重.
brickhouse-0.7.1-SNAPSHOT.jar
05-19
brickhouse-0.7.1-SNAPSHOT
dubbo反序列化问题 Unable to find class: path
rainbow_lasia的博客
07-07 2285
最近线上环境和测试环境总是时不时报找不到dubbo服务的错，或者是路径错误。 java.io.IOException:com.esotericsoftware.kryo.KryoException:Unabletofindclass:path atcom.alibaba.dubbo.common.serialize.support.kryo.KryoObjectInput.readObject(KryoObjectInput.java:127) atcom.alibaba.d...
livy提交任务报错com.cloudera.livy.shaded.kryo.kryo.KryoException: Unable to find class: GATest.ConJob
春华秋实
08-08 5100
今天在测试的时候，出现这个错误，找了好久网上找不到，最近尝试多次终于成功了。问题就在于：Unable to find class你要把你已经uploadJar的jar包中包含的类，引入工程。我用spring-boot+livy+spark的一个例子:public class Contains extends Kryo{ @RequestMapping("/contains") @R...
使用kryo进行序列化各种异常分析解决
sudo5zzb的博客
08-29 6494
使用kryo进行分布式开发时出现如下异常: com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 104, Size: 9 Serialization trace: datas (com.bonc.text.entity.cluster.InputData)     at c
`Unable to find class` error in Drools
diyinqian的博客
01-11 1311
问题描述：当编写drools规则文件时，function中的参数如果是List类型或者是arrayList类型，在已经import java.util.ArrayList包的前提下，执行的时候会报错。报错信息如下： Exception in thread “main” java.lang.RuntimeException: [ function sumLengths (line:5):Un
hive UDF需要jar包
12-09
hive编写 udf 至少需要引入的jar包：hive-exec-xxx.jar 和 hadoop-core-xxx.jar
hive-udf:NexR Hive UDF
05-17
NexR Hive UDF 关于 NexR Hive UDF是Hive用户定义功能的集合。 执照 快速开始 $ git clone https://github.com/nexr/hive-udf.git ... <value>file:///path/to/nexr-hive-udf-[VERSION].jar </pr
Unable to find main class
小猿备忘录
01-03 1559
报错信息：大概意思是找不到主类。 Failed to execute goal org.springframework.boot:spring-boot-maven-plugin: 2.1.0.RELEASE:repackage (repackage) on project project_domain: Execution repackage of goal org.springframe...
java.io.IOException: java.lang.RuntimeException: unable to find class for code 253
相约滦北来看你
04-10 1963
原因： 跟踪了一下所调用的底层接口，发现是redis缓存的问题，登录redis，刷新redis redis 127.0.0.1:6379> FLUSHDB #清空当前数据库中的所有 key。 OK 然后重启服务。
Maven项目打包报错：Unable to find main class
m0_63926154的博客
09-19 6397
在我使用Maven聚合工程进行打包的父工程进行打包时候报错：Unable to find main class 大概意思是找不到 main 启动类。
Unable to find main class无法找到主类
weixin_45881674的博客
01-14 467
一.检查代码是否有重复注解@SpringApplication,如果有注掉多余的,保留一个 二.如果项目继承于spring-boot-starter-parent,在 properties 标签中指定启动类 <properties> <start-class>com.*.SpringbootApplication</start-class> </properties> ...
Redis之反序列化失败问题追踪解决
辉哥的博客
12-05 5608
事故起因 由于业务需要数据库需要多加两个字段，然后就一顿操作猛如虎。重启项目当get redis存放的的对象时报错了。报错示例如下： com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 1388917379 Serialization trace: at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClass
Hive3.0新特性：Materialized Views 物化视图
最新发布
喻师傅的学习笔记
05-07 322
Hive3.0新特性
Hive 作业报错 unable to rename output
06-13
这个错误通常是由于输出路径已经存在导致的。Hive 在执行查询作业时，需要将查询结果写入到指定的输出路径中。如果这个输出路径已经存在，就会出现这个错误。 解决方法： 1. 修改输出路径：在运行 Hive 作业时，可以指定一个新的输出路径，例如： ``` INSERT OVERWRITE DIRECTORY '/new/output/path' SELECT * FROM my_table; ``` 2. 删除输出路径：如果不需要保留之前的输出数据，可以直接删除输出路径，例如： ``` hadoop fs -rm -r /output/path ``` 3. 强制执行查询：如果查询只是读取数据而不需要写入到输出路径中，可以在查询语句前加上 `SET hive.exec.mode.local.auto=true;`，这样就可以强制在本地执行查询，不需要写入到 HDFS 中。 4. 修改配置文件：在 `hive-site.xml` 中添加以下配置： ``` <property> <name>hive.exec.mode.local.auto</name> <value>true</value> </property> ``` 这样就可以在默认情况下强制执行查询，不需要写入到 HDFS 中。 如果以上方法不能解决问题，可以尝试重新启动 Hive 服务或者联系管理员进行进一步排查。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值