配置tez

配置tez

1.上传tez的tar包到HDFS
1.1创建/tez目录
hadoop fs -mkdir /tez
1.2 上传tez的tar包到该目录
hadoop fs -put /opt/software/apache-tez-0.9.1-bin.tar.gz/ /tez
2.解压tez安装包
tar -zxvf apache-tez-0.9.1-bin.tar.gz -C /opt/module/
3.修改目录名
mv apache-tez-0.9.1-bin/ tez
4.在Hive中配置Tez
4.1 在hive的conf目录创建tez-site.xml文件

<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> tez.lib.uris ${fs.defaultFS}/tez/apache-tez-0.9.1-bin.tar.gz tez.use.cluster.hadoop-libs true tez.history.logging.service.class org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService hive.execution.engine tez 4.2 在hive-env.sh文件中添加tez环境变量配置和依赖包环境变量配置 export TEZ_HOME=/opt/module/tez #是你的tez的解压目录 export TEZ_JARS="" for jar in `ls $TEZ_HOME |grep jar`; do export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/$jar done for jar in `ls $TEZ_HOME/lib`; do export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/lib/$jar done # Folder containing extra ibraries required for hive compilation/execution can be controlled by: #hive在运行时需要加载的lzo和tez的jar包的路径 export HIVE_AUX_JARS_PATH=/opt/module/hadoop/share/hadoop/common/hadoop-lzo-0.4.20.jar$TEZ_JARS 4.3 在yarn-site.xml中，关掉虚拟内存检查 yarn.nodemanager.vmem-check-enabled false