hive + tez

最新推荐文章于 2024-08-01 08:58:44 发布

bigdataf

最新推荐文章于 2024-08-01 08:58:44 发布

阅读量1.4k

点赞数 1

分类专栏： hive 文章标签： hive tez

本文链接：https://blog.csdn.net/bigdataf/article/details/78912855

版权

hive 专栏收录该内容

7 篇文章 0 订阅

订阅专栏

安装配置

1.下载：

wget  http://mirrors.hust.edu.cn/apache/tez/0.8.5/apache-tez-0.8.5-bin.tar.gz
tar -zxf apache-tez-0.8.5-bin.tar.gz
cd apache-tez-0.8.5-bin/conf
vim tez-site.xml
增加一下配置
<configuration>

<property>
<name>tez.lib.uris</name>
<value>${fs.defaultFS}/hive/tez/tez.tar.gz</value>
</property>

<property>
<name>tez.use.cluster.hadoop-libs</name>
<value>true</value>
</property>

<property>
<name>tez.runtime.compress</name>
<value>true</value>
</property>

<property>
<name>tez.runtime.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

</configuration>

2.注意这儿的配置需要将 tez.tar.gz 上传的对应的hdfs上


hdfs  dfs -put  ./share/tez.tar.gz  /hive/tez

3.配置hive-env.sh 添加一下内容

export TEZ_HOME=/home/hadoop/apache-tez-0.8.5-bin
export TEZ_CONF_DIR=/home/hadoop/apache-tez-0.8.5-bin/conf
export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${TEZ_HOME}/*:${TEZ_HOME}/lib/*

4.测试tez 是否成功进入hive
set hive.execution.engine=tez;

Total jobs = 1
Launching Job 1 out of 1


Status: Running (Executing on YARN cluster with App id application_1513414830046_12162)

--------------------------------------------------------------------------------
        VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
--------------------------------------------------------------------------------
Map 1 ..........   SUCCEEDED     97         97        0        0       0       0
Reducer 2 ......   SUCCEEDED      1          1        0        0       0       0
--------------------------------------------------------------------------------
VERTICES: 02/02  [==========================>>] 100%  ELAPSED TIME: 8.51 s     
--------------------------------------------------------------------------------
OK

测试ok

5.但再次 set hive.execution.engine=mr; 时报如下错误

Query ID = hadoop_20171227150449_5c0acd22-4dc6-421d-8251-ab501bb193bd
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
java.lang.NoSuchFieldError: DEFAULT_MR_AM_ADMIN_USER_ENV
        at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:473)
        at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:285)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:536)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
        at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
        at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:431)
        at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. DEFAULT_MR_AM_ADMIN_USER_ENV

此错误是tez 中hadoop包与hadoop 版本包不一致导致，删除原来的hadoop-mapreduce包，解决如下

 cp ./share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.4.jar  /home/hadoop/apache-tez-0.8.5-bin/lib/
cp ./share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.4.jar  /home/hadoop/apache-tez-0.8.5-bin/lib/