Hive 运行在 Tez 上

Tez 介绍

Tez 是一种基于内存的计算框架，速度比 MapReduce 要快

解释： 浅蓝色方块表示 Map 任务，绿色方块表示 Reduce 任务，蓝色边框的云朵表示中间结果落地磁盘。

Tez 下载

Tez 官网

Tez 在 Hive 上的运用

前提要有 Hadoop 集群

上传 Tez 压缩包到 Hive 节点上

tar -zxvf apache-tez-0.9.1-bin.tar.gz -C  /opt/module/tez-0.9.1

修改 $HIVE_HOME/conf/hive-env.sh 文件

export TEZ_HOME=/opt/module/tez-0.9.1    #是你的tez的解压目录
export TEZ_JARS=""
for jar in `ls $TEZ_HOME |grep jar`; do
    export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/$jar
done
for jar in `ls $TEZ_HOME/lib`; do
    export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/lib/$jar
done

export HIVE_AUX_JARS_PATH=/opt/module/hadoop-2.7.2/share/hadoop/common/$TEZ_JARS

修改 $HIVE_HOME/conf/hive-site.xml 文件

<property>
    <name>hive.execution.engine</name>
    <value>tez</value>
</property>

在 $HIVE_HOME/conf 目录下新建 tez-site.xml 文件

vi tez-site.xml

# tez-site.xml 文件内容
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
	<name>tez.lib.uris</name>    <value>${fs.defaultFS}/tez/tez-0.9.1,${fs.defaultFS}/tez/tez-0.9.1/lib</value>
</property>
<property>
	<name>tez.lib.uris.classpath</name>    	<value>${fs.defaultFS}/tez/tez-0.9.1,${fs.defaultFS}/tez/tez-0.9.1/lib</value>
</property>
<property>
     <name>tez.use.cluster.hadoop-libs</name>
     <value>true</value>
</property>
<property>
     <name>tez.history.logging.service.class</name>        <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
</property>
</configuration>

将 Tez 相关内容上传到 HDFS 集群

hadoop fs -mkdir /tez
hadoop fs -put /opt/module/tez-0.9.1/ /tez
hadoop fs -ls /tez/tez/tez-0.9.1

报错信息处理

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:535)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1550269792637_0001 failed 2 times due to AM Container for appattempt_1550269792637_0001_000002 exited with  exitCode: -103
For more detailed output, check application tracking page:http://hadoop103:8088/cluster/app/application_1550269792637_0001Then, click on links to logs of each attempt.
Diagnostics: Container [pid=74999,containerID=container_1550269792637_0001_02_000001] is running beyond virtual memory limits. Current usage: 73.1 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used. Killing container.

# 报错原因
由于 NodeManager 上的虚拟内存不足，导致 container 被强制终止，最终导致启动 Hive 客户端报错失败

解决方案

# 在 yarn-site.xml 添加以下配置内容
<!-- 关闭 NodeManager 上的虚拟内存检查 -->
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>

配置完成后保存退出后，将 Hadoop 集群重启，使得刚配置的 yarn-site.xml 配置文件内容生效。