Hive3.x集成Tez和Tez-ui详细步骤

Hive3.x集成Tez和Tez-ui详细步骤

1. Hive集成Tez

下载合适版本的tez:https://dlcdn.apache.org/tez/0.10.1/
我这里使用0.10.1版本，并上传到指定目录

1. 安装TEZ：

# 解压到安装文件夹
tar -zxvf apache-tez-0.10.1-bin.tar.gz -C /opt/modules/

# 进入到安装目录
cd  /opt/modules

# 文件夹重命名为TEZ
mv apache-tez-0.10.1-bin/ tez/

2. 在Hive中配置Tez

1. 编辑hive-env.sh 文件

添加如下内容：

# Set HADOOP_HOME to point to a specific hadoop install directory
export HADOOP_HOME=/opt/modules/hadoop-3.1.3

# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/opt/modules/hive/conf

# Folder containing extra libraries required for hive compilation/execution can be controlled by:
# tez的解压目录
export TEZ_HOME=/opt/modules/tez
export TEZ_JARS=""
for jar in `ls $TEZ_HOME |grep jar`; do
    export TEZ_JARS=$TEZ_JARS,$TEZ_HOME/$jar
done
for jar in `ls $TEZ_HOME/lib`; do
    export TEZ_JARS=$TEZ_JARS,$TEZ_HOME/lib/$jar
done

# 需要设置 HIVE_AUX_JARS_PATH
export HIVE_AUX_JARS_PATH=${TEZ_JARS:1}
echo $HIVE_AUX_JARS_PATH

2. 编辑hive-site.xml文件

添加如下内容：

    <property>
        <name>hive.execution.engine</name>
        <value>tez</value>
    </property>

3.配置tez-site.xml

在hive/conf文件夹下新建tez-site.xml ,注意文件夹名称的变化，上面我重命名过！！！ 并添加如下内容：

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>tez.lib.uris</name>
        <value>${fs.defaultFS}/tez/tez,${fs.defaultFS}/tez/tez/lib</value>
    </property>
    <property>
        <name>tez.lib.uris.classpath</name>
        <value>${fs.defaultFS}/tez/tez,${fs.defaultFS}/tez/tez/lib</value>
    </property>
    <property>
        <name>tez.use.cluster.hadoop-libs</name>
        <value>true</value>
    </property>
    <property>
        <name>tez.history.logging.service.class</name>
        <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
    </property>
    <property>
        <name>hive.tez.container.size</name>
        <value>1024</value>
    </property>
    <property>
        <name>tez.am.resource.memory.mb</name>
        <value>1024</value>
    </property>

    <property>
        <description>Enable Tez to use the Timeline Server for History Logging</description>
        <name>tez.history.logging.service.class</name>
        <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
    </property>

    <property>
        <description>URL for where the Tez UI is hosted</description>
        <name>tez.tez-ui.history-url.base</name>
        <value>http://hadoop102:8080/tez-ui-0.9.2/</value>
    </property>
</configuration>

4.上传Tez到hadoop

# hdfs上新建tez文件夹
hdfs dfs -mkdir /tez

# 上传tez到hdfs
hdfs dfs -put /opt/modules/tez /tez

# 查看文件夹
hdfs dfs -ls /tez

5.启动hadoop和hive测试

create table test(id integer,name string);
insert into test values (1,'1');

2.Hive集成Tez-ui

官方下载地址：https://tez.apache.org/tez-ui.html

1. 下载war包并部署到tomcat下

下载地址：https://tomcat.apache.org/download-90.cgi
将tez-ui.war报部署到tomcat的webapps下，然后启动tomcat会自动解压，解压后修改config下的configs.env配置文件:

timeline: "http://hadoop102:8188",
rm: "http://hadoop102:8088",

2. 修改tez-site.xml添加如下内容

<property>
  <description>Enable Tez to use the Timeline Server for History Logging</description>
  <name>tez.history.logging.service.class</name>
  <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
</property>

<property>
  <description>URL for where the Tez UI is hosted</description>
  <name>tez.tez-ui.history-url.base</name>
  <value>http://hadoop102:8080/tez-ui-0.9.2/</value>
</property>

2. 编辑yarn-site.xml添加timeline

<!-- 配置timeline -->
<property>
  <description>Indicate to clients whether Timeline service is enabled or not.
  If enabled, the TimelineClient library used by end-users will post entities
  and events to the Timeline server.</description>
  <name>yarn.timeline-service.enabled</name>
  <value>true</value>
</property>

<property>
  <description>The hostname of the Timeline service web application.</description>
  <name>yarn.timeline-service.hostname</name>
  <value>hadoop102</value>
</property>

<property>
  <description>Enables cross-origin support (CORS) for web services where
  cross-origin web response headers are needed. For example, javascript making
  a web services request to the timeline server.</description>
  <name>yarn.timeline-service.http-cross-origin.enabled</name>
  <value>true</value>
</property>

<property>
  <description>Publish YARN information to Timeline Server</description>
  <name> yarn.resourcemanager.system-metrics-publisher.enabled</name>
  <value>true</value>
</property>
 <property>
	<name>yarn.timeline-service.generic-application-history.enabled</name>
	<value>true</value>
   </property>
   <property>
        <description>Address for the Timeline server to start the RPC server.</description>
        <name>yarn.timeline-service.address</name>
        <value>hadoop102:10201</value>
   </property>
   <property>
        <description>The http address of the Timeline service web application.</description>
        <name>yarn.timeline-service.webapp.address</name>
        <value>hadoop102:8188</value>
   </property>
   <property>
        <description>The https address of the Timeline service web application.</description>
        <name>yarn.timeline-service.webapp.https.address</name>
        <value>hadoop102:2191</value>
   </property>
   <property>
        <name>yarn.timeline-service.handler-thread-count</name>
        <value>24</value>
   </property>

3.开启timeline和tomcat

# 开启timeline
yarn-daemon.sh start timelineserver

# 开启tomcat
cd /opt/modules/apache-tomcat-9.0.73/bin
./startup.sh

执行任务并访问yarn的Tracking ui 将自动跳转到tez-ui

3. 遇到的问题

1. Container [pid=43502,containerID=container_1681453895005_0003_02_000001] is running 1823116544B beyond the ‘VIRTUAL’ memory limit. Current usage: 329.1 MB of 512 MB physical memory used; 2.7 GB of 1.0 GB virtual memory used. Killing container.

解决办法：
编辑mapred-site.xml文件，并添加如下配置

    <property>
    <!-- 是否对容器强制执行虚拟内存限制 -->
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
        <description>Whether virtual memory limits will be enforced for containers</description>
    </property>
    <property>
    <!-- 为容器设置内存限制时虚拟内存与物理内存之间的比率 -->
        <name>yarn.nodemanager.vmem-pmem-ratio</name>
        <value>4</value>
        <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
    </property>

2. TEZUI访问一直提示找不到timeline，并且提示访问的是localhost:8081

将configs.env的权限改成777

3.Tez一直在运行中