hive集成tez

hive1集成引擎Tez
3.2.1 安装包准备
1）下载tez的依赖包：http://tez.apache.org
2）拷贝apache-tez-0.9.1-bin.tar.gz到hadoop213的/root/soft目录
[root@hadoop213 soft]$ ls
apache-tez-0.9.1-bin.tar.gz
3）将apache-tez-0.9.1-bin.tar.gz上传到HDFS的/tez目录下。
[root@hadoop213 conf]$ hadoop fs -mkdir /tez
[root@hadoop213 conf]$ hadoop fs -put /root/soft/apache-tez-0.9.1-bin.tar.gz/ /tez
4）解压缩apache-tez-0.9.1-bin.tar.gz
[root@hadoop213 soft]$ tar -zxvf apache-tez-0.9.1-bin.tar.gz -C /root/module
5）修改名称
[root@hadoop213 module]$ mv apache-tez-0.9.1-bin/ tez-0.9.1
3.2.2 集成Tez 
1）进入到hive1的配置目录：/root/module/hive1/conf
[root@hadoop213 conf]$ pwd
/root/module/hive1/conf
2）在hive1的/root/module/hive1/conf下面创建一个tez-site.xml文件
[root@hadoop213 conf]$ pwd
/root/module/hive1/conf
[root@hadoop213 conf]$ vim tez-site.xml
添加如下内容
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>tez.lib.uris</name>
        <value>${fs.defaultFS}/tez/apache-tez-0.9.1-bin.tar.gz</value>
    </property>
    <property>
         <name>tez.use.cluster.hadoop-libs</name>
         <value>true</value>
    </property>
    <property>
         <name>tez.history.logging.service.class</name>        
         <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
    </property>
</configuration>
2）在hive-env.sh文件中添加tez环境变量配置和依赖包环境变量配置
[root@hadoop213 conf]$ mv hive-env.sh.template hive-env.sh
[root@hadoop213 conf]$ vim hive-env.sh
添加如下配置
# Set HADOOP_HOME to point to a specific hadoop install directory
export HADOOP_HOME=/root/module/hadoop-2.7.2

# hive1 Configuration Directory can be controlled by:
export hive_CONF_DIR=/root/module/hive1/conf

# Folder containing extra libraries required for hive1 compilation/execution can be controlled by:
export TEZ_HOME=/root/module/tez-0.9.1    #是你的tez的解压目录
export TEZ_JARS=""
for jar in `ls $TEZ_HOME |grep jar`; do
    export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/$jar
done
for jar in `ls $TEZ_HOME/lib`; do
    export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/lib/$jar
done

export hive_AUX_JARS_PATH=/root/module/hadoop-2.7.2/share/hadoop/common/hadoop-lzo-0.4.20.jar$TEZ_JARS
3）在hive-site.xml文件中添加如下配置，更改hive计算引擎
<property>
    <name>hive.execution.engine</name>
    <value>tez</value>
</property>
3.2.3 测试
1）启动hive1
[root@hadoop213 hive1]$ bin/hive
2）创建表
hive1 (default)> create table student(
id int,
name string);
3）向表中插入数据
hive1 (default)> insert into student values(1,"zhangsan");
4）如果没有报错就表示成功了
hive1 (default)> select * from student;
1       zhangsan
3.2.4 注意事项
1）运行Tez时检查到用过多内存而被NodeManager杀死进程问题：
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1546781144082_0005 failed 2 times due to AM Container for appattempt_1546781144082_0005_000002 exited with  exitCode: -103
For more detailed output, check application tracking page:http://hadoop103:8088/cluster/app/application_1546781144082_0005Then, click on links to logs of each attempt.
Diagnostics: Container [pid=11116,containerID=container_1546781144082_0005_02_000001] is running beyond virtual memory limits. Current usage: 216.3 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.
这种问题是从机上运行的Container试图使用过多的内存，而被NodeManager kill掉了。
[摘录] The NodeManager is killing your container. It sounds like you are trying to use hadoop streaming which is running as a child process of the map-reduce task. The NodeManager monitors the entire process tree of the task and if it eats up more memory than the maximum set in mapreduce.map.memory.mb or mapreduce.reduce.memory.mb respectively, we would expect the Nodemanager to kill the task, otherwise your task is stealing memory belonging to other containers, which you don't want.
2）解决方法：
（1）关掉虚拟内存检查，修改yarn-site.xml，
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>
（2）修改后一定要分发，并重新启动h


集成后报如下错误，待解决
Error: java.lang.NoClassDefFoundError: org/apache/tez/runtime/api/Event (state=,code=0)