安装依赖
1. 安装hadoop
2. 安装hive
安装Tez
- 下载tez的依赖包:http://tez.apache.org
- 解压修改名称
tar -zxvf apache-tez-0.9.1-bin.tar.gz -C /opt/module
mv apache-tez-0.9.1-bin/ tez-0.9.1
- 将tez的tar包上传hdfs
hadoop fs -mkdir /tez
hadoop fs -put /opt/software/apache-tez-0.9.1-bin.tar.gz/ /tez
集成Tez
- 在Hive的/opt/module/hive/conf下面创建一个tez-site.xml文件
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>tez.lib.uris</name>
<value>${fs.defaultFS}/tez/apache-tez-0.9.1-bin.tar.gz</value>
</property>
<property>
<name>tez.use.cluster.hadoop-libs</name>
<value>true</value>
</property>
<property>
<name>tez.history.logging.service.class</name>
<value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
</property>
</configuration>
- 在hive-env.sh文件中添加tez环境变量配置和依赖包环境变量配置
vim hive-env.sh
# Set HADOOP_HOME to point to a specific hadoop install directory
export HADOOP_HOME=/opt/module/hadoop-2.7.2
# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/opt/module/hive/conf
# Folder containing extra libraries required for hive compilation/execution can be controlled by:
export TEZ_HOME=/opt/module/tez-0.9.1 #是你的tez的解压目录
export TEZ_JARS=""
for jar in `ls $TEZ_HOME |grep jar`; do
export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/$jar
done
for jar in `ls $TEZ_HOME/lib`; do
export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/lib/$jar
done
export HIVE_AUX_JARS_PATH=/opt/module/hadoop-2.7.2/share/hadoop/common/hadoop-lzo-0.4.20.jar$TEZ_JARS
- 在hive-site.xml文件中添加如下配置,更改hive计算引擎
<property>
<name>hive.execution.engine</name>
<value>tez</value>
</property>
- 测试
bin/hive
create table student(id int, name string);
insert into student values(1,"zhangsan");
select * from student;
问题
2.6 GB of 2.1 GB virtual memory used. Killing container.
运行Tez时检查到用过多内存而被NodeManager杀死进程问题,这种问题是从机上运行的Container试图使用过多的内存,而被NodeManager kill掉了。
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown.
Application application_1546781144082_0005 failed 2 times due to AM Container for
appattempt_1546781144082_0005_000002 exited with exitCode: -103
Container [pid=11116,containerID=container_1546781144082_0005_02_000001] is running beyond
virtual memory limits. Current usage: 216.3 MB of 1 GB physical memory used; 2.6 GB of
2.1 GB virtual memory used. Killing container.
解决方法是:关掉虚拟内存检查,修改yarn-site.xml,修改后需要分发并重启
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>