该帖子主要记录Hive On Tez安装及验证测试,并详细记录验证过程中的各种坑;
Tez安装介绍请参考
http://blog.csdn.net/hqwang4/article/details/72773654
Hive运行在Tez上有2种方式:
一种是修改mapreduce-site.xml文件将yarn修改为yarn-tez(将所有运行在Yarn上的MR全部修改为了引擎为Tez);
另一种修改Hive的执行引擎为Tez,set hive.execution.engine=tez;
修改hive-env.sh
exportHIVE_AUX_JARS_PATH=$HADOOP_HOME/share/hadoop/mapreduce1/hadoop-core-2.6.0-mr1-cdh5.5.0.jar:$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0-cdh5.5.0.jar:$HADOOP_HOME/share/hadoop/common/hadoop-common-2.6.0-cdh5.5.0.jar
验证测试
#启动hive
$HIVE_HOME/bin/hive
#设置Hive执行引擎为Tez
sethive.execution.engine=tez;
#执行SQL
select count(*) from student;
测试结果如下:
常见问题
1、问题:
For moredetailed output, check application trackingpage:http://node01:8088/proxy/application_1495800987987_0003/Then, click on linksto logs of each attempt.
Diagnostics:Exception from container-launch.
Container id:container_1495800987987_0003_02_000001
Exit code: 1
Stack trace:ExitCodeException exitCode=1:
atorg.apache.hadoop.util.Shell.runCommand(Shell.java:543)
atorg.apache.hadoop.util.Shell.run(Shell.java:460)
atorg.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:720)
atorg.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:210)
解决方法:
需要将${TEZ_HOME}/share/tez.tar.gz文件上传到Hdfs上,并在tez-site.xml文件tez.lib.uris属性中配置。
2、问题
Vertex failed,ver