Ochadoop之hive on tez搭建[转自 AIMP平台wiki]

1.软件版本

och 3.1.0
hive 0.13.0
tez 0.4.0-incubating

2.部署tez

2.1编译

$tar zxvf tez-0.4.0-incubating.tar.gz
$cd tez-0.4.0-incubating/
#修改tez关联的hadoop.version为2.3.0
$vim pom.xml
$mvn clean install -DskipTests=true -Dmaven.javadoc.skip=true

生成的jar在目录tez-dist/target/tez-0.4.0-incubating下；

2.2上传jar

$hdfs dfs -mkdir /apps
$hdfs dfs -put tez-dist/target/tez-0.4.0-incubating/tez-0.4.0-incubating /apps/

2.3配置tez

1. vim tez-site.xml
   配置属性tez.lib.uris，指定值包含上传的jar包根目录及lib目录：${fs.defaultFS}/apps/tez-0.4.0-incubating,${fs.defaultFS}/apps/tez-0.4.0-incubating/lib
   将tez-site.xml放置到$HADOOP_CONF_DIR目录下
2. vim mapred-site.xml
   修改mapreduce.framework.name为yarn-tez
3. vim hadoop-env.sh
   添加tez jar到HADOOP_CLASSPATH
   

   tez_home=/home/ochadoop/app/tez-0.4.0-incubating
   if [ -d ${tez_home} ]; then
     for f in ${tez_home}/*.jar; do
       if [ "$HADOOP_CLASSPATH" ]; then
         export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
       else
         export HADOOP_CLASSPATH=$f
       fi
     done
     for f in ${tez_home}/lib/*.jar; do
       export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
     done
   fi
   

本步配置结束可通过命令验证tez安装：

$hadoop jar ${HADOOP_HOME}/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar sleep -mt 1 -rt 1 -m 1 -r 1

3.修改hive配置

3.1添加tez jar包到HIVE_AUX_JARS_PATH

export HIVE_HOME=/home/ochadoop/app/apache-hive-0.13.0-bin
export HIVE_AUX_JARS_PATH=${HIVE_HOME}/hcatalog/share/hcatalog/hive-hcatalog-core-0.13.0.jar
tez_home=/home/ochadoop/app/tez-0.4.0-incubating
if [ -d ${tez_home} ]; then
  for f in ${tez_home}/*.jar; do
    export HIVE_AUX_JARS_PATH=$HIVE_AUX_JARS_PATH:$f
  done
  for f in ${tez_home}/lib/*.jar; do
    export HIVE_AUX_JARS_PATH=$HIVE_AUX_JARS_PATH:$f
  done
fi

3.2上传hive-exec-*.jar到hdfs

$hdfs dfs -put ${HIVE_HOME}/lib/hive-exec-0.13.0.jar /user/hive/

3.3修改hive-site.xml

hive.execution.engine修改为tez

4.验证

set hive.use.tez.natively=true;
set hive.enable.mrr=true;

$hive
hive> use yangzq;
hive> select count(1) from yangzq.act_201301;

参考资源：
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.1/bk_installing_manually_book/content/rpm-chap-tez_configure_tez.html
https://issues.apache.org/jira/browse/HIVE-6098

---

Updated by 杨 志强 , 更新于 7 天 之前