Hive -- Hive2.3.0集成Tez

安装依赖

1. 安装hadoop

Hadoop的分布式搭建
Hadoop的高可用HA搭建

2. 安装hive

Hive – Hive2.3.0的部署安装

安装Tez

1. 下载tez的依赖包：http://tez.apache.org
2. 解压修改名称

tar -zxvf apache-tez-0.9.1-bin.tar.gz -C /opt/module
mv apache-tez-0.9.1-bin/ tez-0.9.1

3. 将tez的tar包上传hdfs

hadoop fs -mkdir /tez
hadoop fs -put /opt/software/apache-tez-0.9.1-bin.tar.gz/ /tez

集成Tez

1. 在Hive的/opt/module/hive/conf下面创建一个tez-site.xml文件

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>tez.lib.uris</name>
        <value>${fs.defaultFS}/tez/apache-tez-0.9.1-bin.tar.gz</value>
    </property>
    <property>
         <name>tez.use.cluster.hadoop-libs</name>
         <value>true</value>
    </property>
    <property>
         <name>tez.history.logging.service.class</name>        
         <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
    </property>
</configuration>

2. 在hive-env.sh文件中添加tez环境变量配置和依赖包环境变量配置

vim hive-env.sh

# Set HADOOP_HOME to point to a specific hadoop install directory
export HADOOP_HOME=/opt/module/hadoop-2.7.2

# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/opt/module/hive/conf

# Folder containing extra libraries required for hive compilation/execution can be controlled by:
export TEZ_HOME=/opt/module/tez-0.9.1    #是你的tez的解压目录
export TEZ_JARS=""
for jar in `ls $TEZ_HOME |grep jar`; do
    export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/$jar
done
for jar in `ls $TEZ_HOME/lib`; do
    export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/lib/$jar
done

export HIVE_AUX_JARS_PATH=/opt/module/hadoop-2.7.2/share/hadoop/common/hadoop-lzo-0.4.20.jar$TEZ_JARS

3. 在hive-site.xml文件中添加如下配置，更改hive计算引擎

<property>
    <name>hive.execution.engine</name>
    <value>tez</value>
</property>

4. 测试

bin/hive

create table student(id int, name string);
insert into student values(1,"zhangsan");
select * from student;

问题

2.6 GB of 2.1 GB virtual memory used. Killing container.

  运行Tez时检查到用过多内存而被NodeManager杀死进程问题，这种问题是从机上运行的Container试图使用过多的内存，而被NodeManager kill掉了。

Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. 
Application application_1546781144082_0005 failed 2 times due to AM Container for 
appattempt_1546781144082_0005_000002 exited with  exitCode: -103

Container [pid=11116,containerID=container_1546781144082_0005_02_000001] is running beyond 
virtual memory limits. Current usage: 216.3 MB of 1 GB physical memory used; 2.6 GB of 
2.1 GB virtual memory used. Killing container.

  解决方法是：关掉虚拟内存检查，修改yarn-site.xml，修改后需要分发并重启

<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>