记录hive on tez报错

最新推荐文章于 2023-12-08 11:44:35 发布

琅天溪

最新推荐文章于 2023-12-08 11:44:35 发布

阅读量1.1k

点赞数 3

分类专栏：大数据组件文章标签： hive tez 大数据

本文链接：https://blog.csdn.net/qq_28089993/article/details/109250129

版权

大数据组件专栏收录该内容

2 篇文章 1 订阅

订阅专栏

记录hive on tez报错

我使用的hadoop以及hive还有tez的版本是：
hadoop-2.6.0-cdh5.16.2
hive-1.1.0-cdh5.16.2
tez-0.8.5

上测试成功结果

Query ID = root_20201023210303_3bda0a6a-6f65-41fe-9c18-64b582c5d2d8
Total jobs = 1
Launching Job 1 out of 1


Status: Running (Executing on YARN cluster with App id application_1603453946560_0006)

--------------------------------------------------------------------------------
        VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
--------------------------------------------------------------------------------
Map 1 ..........   SUCCEEDED      1          1        0        0       0       0
--------------------------------------------------------------------------------
VERTICES: 01/01  [==========================>>] 100%  ELAPSED TIME: 5.90 s     
--------------------------------------------------------------------------------
Loading data to table default.student
Table default.student stats: [numFiles=2, numRows=1, totalSize=50, rawDataSize=10]
OK
_col0	_col1
Time taken: 11.422 seconds

问题描述

在配置好tez后，开启hive执行sql报错，报错如下

Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1603453946560_0004_1_00, diagnostics=[Vertex vertex_1603453946560_0004_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: values__tmp__table__1 initializer failed, vertex=vertex_1603453946560_0004_1_00 [Map 1], java.lang.NoClassDefFoundError: com/esotericsoftware/kryo/Serializer
	at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:107)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.esotericsoftware.kryo.Serializer
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 12 more
]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask

解决：将kryo-2.22.jar 放置到/opt/module/hive-1.1.0-cdh5.16.2/auxlib/ 下。

最后

希望帮到你们，这个问题困惑了我将近6个小时，网上寻找遇到此问题的博客也没找到解决方案，期间更换tez版本、使用cdh hadoop版本编译tez等等，搞了很久，差点放弃。最后还是回到了原点，我是搞java的既然报错找不到这个类，那我帮他找到不就行了，最后测试发现将kryo-2.22.jar包放置到hive的auxlib是可行的，如果对你有帮助，记得点赞哈，哈哈。