Hive on Tez

最新推荐文章于 2024-05-19 15:04:31 发布
最新推荐文章于 2024-05-19 15:04:31 发布
阅读量329 收藏
点赞数
文章标签： 大数据 git 运维
原文链接：https://my.oschina.net/guol/blog/915416
版权

为什么80%的码农都做不了架构师？>>>   hot3.png

介绍

        Tez目标是用来构建复杂的有向无环图数据处理程序。Tez项目是构建在YARN之上的。Tez的设计上有两点优势：

            1    用户体验

                    使用API来自定义数据流

                    灵活的Input-Processor-Output运行模式

                    与计算的数据类型无关

                    简单的部署流程

            2    计算性能

                    性能高于MapReduce

                    资源管理更加优化

                    运行时配置预加载

                    物理数据流动态运行

 

MR区别

        基于MR的Job：

        在YARN中，每个作业的AM会先向RM申请资源（Container），申请到资源之后开始运行作业，作业处理完成后释放资源，期间没有资源重新利用的环节。这样会使作业大大的延迟。基于MR的Hive/Pig的DAG数据流处理过程”，可以看出图中的每一节点都是把结果写到一个中间存储（HDFS/S3）中，下个节点从中间存储读取数据，再来继续接下来的计算。可见中间存储的读写性能对整个DAG的性能影响是很大的。

 

        基于Tez的Job：

        使用Tez后，yarn的作业不是先提交给RM了，而是提交给AMPS。AMPS在启动后，会预先创建若干个AM，作为AM资源池，当作业被提交到AMPS的时候，AMPS会把该作业直接提交到AM上，这样就避免每个作业都创建独立的AM，大大的提高了效率。AM缓冲池中的每个AM在启动时都会预先创建若干个container，以此来减少因创建container所话费的时间。每个任务运行完之后，AM不会立马释放Container，而是将它分配给其它未执行的任务。

 

编译

        基础环境：

            操作系统：CentOS release 6.7

            Hadoop：2.6.0+cdh5.10.0 (Tez是Hortonworks贡献的，CDH要使用Tez，你懂得)

        软件依赖：

            java：jdk1.7.0_67

            mvn：maven-3.3.9

            node：node-v6.10.3

            tez：tez-0.7.1

            protobuf：protobuf-2.5.0

        其他编译依赖

yum -y install gcc-c++ openssl-devel glibc
yum -y install autoconf automake libtool
yum -y install git unzip

        下载相关软件

#编译主目录
cd /opt/programs/

#下载pb
wget 'https://github-production-release-asset-2e65be.s3.amazonaws.com/23357588/09f5cfca-d24e-11e4-9840-20d894b9ee09?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20170606%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20170606T001340Z&X-Amz-Expires=300&X-Amz-Signature=fd9ea5018702a129ef6fa9bc0830b72370a0ad11adc46f38d81328f95fd9145b&X-Amz-SignedHeaders=host&actor_id=12913767&response-content-disposition=attachment%3B%20filename%3Dprotobuf-2.5.0.tar.gz&response-content-type=application%2Foctet-stream'

#下载node
wget https://nodejs.org/dist/v6.10.3/node-v6.10.3-linux-x64.tar.xz

#下载tez
wget 'http://mirrors.hust.edu.cn/apache/tez/0.7.1/apache-tez-0.7.1-src.tar.gz'

        修改bash_profile

export JAVA_HOME=/opt/programs/jdk1.7.0_67
PATH=$PATH:/opt/programs/jdk1.7.0_67/bin:/opt/programs/apache-maven-3.3.9/bin:/opt/programs/node-v6.10.3-linux-x64/bin
export PATH

        修改tez主pom.xml

            tez编译主目录：/opt/programs/apache-tez-0.7.1-src，修改pom.xml，在<profiles>...</profiles>中添加CDH的mvn依赖。

<profile>
        <id>cdh5.10.0</id>
        <activation>
            <activeByDefault>true</activeByDefault>
        </activation>
        <properties>
            <hadoop.version>2.6.0-cdh5.10.0</hadoop.version>
        </properties>
        <pluginRepositories>
            <pluginRepository>
                <id>cloudera</id>
                <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
            </pluginRepository>
            <pluginRepository>
                <id>nexus public</id>
                <url>http://central.maven.org/maven2/</url>
            </pluginRepository>
        </pluginRepositories>

        <repositories>
            <repository>
                <id>nexus public</id>
                <url>http://central.maven.org/maven2/</url>
            </repository>
            <repository>
                <id>cloudera</id>
                <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
            </repository>
        </repositories>    
</profile>

        开始编译

mvn -Pcdh5.10.0  clean package -Dtar -DskipTests=true -Dmaven.javadoc.skip=true

            tez-mapreduce编译错误

[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /opt/programs/apache-tez-0.7.1-src/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/hadoop/mapreduce/JobContextImpl.java:[57,8] org.apache.tez.mapreduce.hadoop.mapreduce.JobContextImpl is not abstract and does not override abstract method userClassesTakesPrecedence() in org.apache.hadoop.mapreduce.JobContext
[INFO] 1 error
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] tez ................................................ SUCCESS [  0.779 s]
[INFO] tez-api ............................................ SUCCESS [  7.778 s]
[INFO] tez-common ......................................... SUCCESS [  0.394 s]
[INFO] tez-runtime-internals .............................. SUCCESS [  0.629 s]
[INFO] tez-runtime-library ................................ SUCCESS [  1.644 s]
[INFO] tez-mapreduce ...................................... FAILURE [  0.484 s]
[INFO] tez-examples ....................................... SKIPPED

                改正：修改apache-tez-0.7.1-src/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/hadoop/mapreduce/JobContextImpl.java文件：在471行追加下面代码

465   @Override
466   public Progressable getProgressible() {
467     return progress;
468   }
469  
470   @Override
471   public boolean userClassesTakesPrecedence() {
472     return getJobConf().getBoolean(MRJobConfig.MAPREDUCE_JOB_USER_CLASSPATH_FIRST, false);
473   }

            tez-ui编译错误

[INFO] --- exec-maven-plugin:1.3.2:exec (Bower install) @ tez-ui ---
bower ESUDO         Cannot be run with sudo

Additional error details:
Since bower is a user command, there is no need to execute it with superuser permissions.
If you're having permission errors when using bower without sudo, please spend a few minutes learning more about how your system should work and make any necessary repairs.

http://www.joyent.com/blog/installing-node-and-npm
https://gist.github.com/isaacs/579814

You can however run a command with sudo using --allow-root option
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] tez ................................................ SUCCESS [  0.750 s]
[INFO] tez-api ............................................ SUCCESS [  7.433 s]
[INFO] tez-common ......................................... SUCCESS [  0.377 s]
[INFO] tez-runtime-internals .............................. SUCCESS [  0.625 s]
[INFO] tez-runtime-library ................................ SUCCESS [  1.575 s]
[INFO] tez-mapreduce ...................................... SUCCESS [  0.764 s]
[INFO] tez-examples ....................................... SUCCESS [  0.238 s]
[INFO] tez-dag ............................................ SUCCESS [  2.887 s]
[INFO] tez-tests .......................................... SUCCESS [  0.642 s]
[INFO] tez-ui ............................................. FAILURE [01:07 min]
[INFO] tez-plugins ........................................ SKIPPED
[INFO] tez-yarn-timeline-history .......................... SKIPPED
[INFO] tez-yarn-timeline-history-with-acls ................ SKIPPED
[INFO] tez-history-parser ................................. SKIPPED
[INFO] tez-tools .......................................... SKIPPED
[INFO] tez-perf-analyzer .................................. SKIPPED
[INFO] tez-job-analyzer ................................... SKIPPED
[INFO] tez-dist ........................................... SKIPPED
[INFO] Tez ................................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:23 min
[INFO] Finished at: 2017-06-06T15:30:48+08:00
[INFO] Final Memory: 64M/368M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.2:exec (Bower install) on project tez-ui: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]

                改正：修改apache-tez-0.7.1-src/tez-ui/pom.xml文件，在<artifactId>exec-maven-plugin</artifactId>下面的arguments tag追加<argument>--allow-root</argument>，如下部分：

<plugin>
  <artifactId>exec-maven-plugin</artifactId>
  <groupId>org.codehaus.mojo</groupId>
  <executions>
    <execution>
      <id>Bower install</id>
      <phase>generate-sources</phase>
      <goals>
        <goal>exec</goal>
      </goals>
      <configuration>
        <workingDirectory>${webappDir}</workingDirectory>
        <executable>${node.executable}</executable>
        <arguments>
          <argument>node_modules/bower/bin/bower</argument>
          <argument>install</argument>
          <argument>--remove-unnecessary-resolutions=false</argument>
          <argument>--allow-root</argument>
        </arguments>
      </configuration>
    </execution>

        编译成功

[INFO] --- build-helper-maven-plugin:1.8:maven-version (maven-version) @ tez-docs ---
[INFO] 
[INFO] --- maven-jar-plugin:2.4:test-jar (default) @ tez-docs ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: /opt/programs/apache-tez-0.7.1-src/docs/target/tez-docs-0.7.1-tests.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] tez ................................................ SUCCESS [  0.821 s]
[INFO] tez-api ............................................ SUCCESS [  7.644 s]
[INFO] tez-common ......................................... SUCCESS [  0.388 s]
[INFO] tez-runtime-internals .............................. SUCCESS [  0.578 s]
[INFO] tez-runtime-library ................................ SUCCESS [  1.443 s]
[INFO] tez-mapreduce ...................................... SUCCESS [  0.766 s]
[INFO] tez-examples ....................................... SUCCESS [  0.237 s]
[INFO] tez-dag ............................................ SUCCESS [  2.694 s]
[INFO] tez-tests .......................................... SUCCESS [  0.609 s]
[INFO] tez-ui ............................................. SUCCESS [18:01 min]
[INFO] tez-plugins ........................................ SUCCESS [  0.032 s]
[INFO] tez-yarn-timeline-history .......................... SUCCESS [  0.470 s]
[INFO] tez-yarn-timeline-history-with-acls ................ SUCCESS [  0.307 s]
[INFO] tez-history-parser ................................. SUCCESS [01:02 min]
[INFO] tez-tools .......................................... SUCCESS [  0.016 s]
[INFO] tez-perf-analyzer .................................. SUCCESS [  0.015 s]
[INFO] tez-job-analyzer ................................... SUCCESS [  1.461 s]
[INFO] tez-dist ........................................... SUCCESS [ 20.955 s]
[INFO] Tez ................................................ SUCCESS [  0.026 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 19:42 min
[INFO] Finished at: 2017-06-06T09:21:29+08:00
[INFO] Final Memory: 56M/462M
[INFO] ------------------------------------------------------------------------

       生成的文件

                在apache-tez-0.7.1-src/tez-dist/target目录下

安装

        创建tez部署目录

mkdir /opt/programs/tez_0.7.1/

        拷贝tez-dist/target/tez-0.7.1.tar.gz到部署目录并解压

cp /opt/programs/apache-tez-0.7.1-src/tez-dist/target/tez-0.7.1.tar.gz  /opt/programs/tez_0.7.1/
cd /opt/programs/tez_0.7.1/
tar zxvf tez-0.7.1.tar.gz

        拷贝lzo包到解压后的lib目录

cp /usr/lib/hadoop/lib/hadoop-lzo.jar /opt/programs/tez_0.7.1/lib/
cp /usr/lib/hadoop/lib/hadoop-lzo-0.4.15-cdh5.5.1.jar /opt/programs/tez_0.7.1/lib/

        在/etc/hadoop/conf目录创建tez-site.xml配置文件(hadoop1-6)

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
 <property>
   <name>tez.lib.uris</name>
   <value>hdfs://dev-dalu:8020/user/tez/tez-0.7.1.tar.gz</value>
 </property>

 <property>
   <name>tez.history.logging.service.class</name>
   <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
 </property>

 <property>
   <name>tez.tez-ui.history-url.base</name>
   <value>http://172.31.217.155:8080/tez-ui/</value>
 </property>

 <property>
   <name>tez.use.cluster.hadoop-libs</name>
   <value>false</value>
 </property>
</configuration>

        在/etc/hadoop/conf目录创建hadoop-env.sh文件(hadoop1-6)

export TEZ_HOME=/opt/programs/tez_0.7.1
export TEZ_CONF_DIR=/etc/hadoop/conf

for jar in `ls $TEZ_HOME |grep jar`; do
    export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$TEZ_HOME/$jar
done

for jar in `ls $TEZ_HOME/lib`; do
    export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$TEZ_HOME/lib/$jar
done

        在hadoop5上启动timeline-service服务

yarn timelineserver

        调整yarn-site.xml配置文件(hadoop1-6)

<?xml version="1.0"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
    <!-- RM Manager Configd -->
    <property>
        <name>yarn.resourcemanager.connect.retry-interval.ms</name>
        <value>2000</value>
    </property>

    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
        <value>true</value>
    </property>

    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>dalu-rm-cluster</value>
    </property>

    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>hadoop1,hadoop6</value>
    </property>

    <property>
        <name>yarn.resourcemanager.ha.id</name>
        <value>hadoop1</value>
    </property>


    <!--scheduler capacity -->
    <property>
        <name>yarn.resourcemanager.scheduler.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
    </property>

    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    </property>

    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>hadoop2:2181,hadoop3:2181,hadoop4:2181</value>
    </property>

    <property>
        <name>yarn.resourcemanager.zk.state-store.address</name>
        <value>hadoop2:2181,hadoop3:2181,hadoop4:2181</value>
    </property>

    <property>
        <name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>
        <value>5000</value>
    </property>

    <!-- RM1 Configs-->
    <property>
        <name>yarn.resourcemanager.address.hadoop1</name>
        <value>hadoop1:23140</value>
    </property>

    <property>
        <name>yarn.resourcemanager.scheduler.address.hadoop1</name>
        <value>hadoop1:23130</value>
    </property>

    <property>
        <name>yarn.resourcemanager.webapp.https.address.hadoop1</name>
        <value>hadoop1:23189</value>
    </property>

    <property>
        <name>yarn.resourcemanager.webapp.address.hadoop1</name>
        <value>hadoop1:23188</value>
    </property>

    <property>
        <name>yarn.resourcemanager.resource-tracker.address.hadoop1</name>
        <value>hadoop1:23125</value>
    </property>

    <property>
        <name>yarn.resourcemanager.admin.address.hadoop1</name>
        <value>hadoop1:23141</value>
    </property>


    <!-- RM2 Configs -->
    <property>
        <name>yarn.resourcemanager.address.hadoop6</name>
        <value>hadoop6:23140</value>
    </property>

    <property>
        <name>yarn.resourcemanager.scheduler.address.hadoop6</name>
        <value>hadoop6:23130</value>
    </property>

    <property>
        <name>yarn.resourcemanager.webapp.https.address.hadoop6</name>
        <value>hadoop6:23189</value>
    </property>

    <property>
        <name>yarn.resourcemanager.webapp.address.hadoop6</name>
        <value>hadoop6:23188</value>
    </property>

    <property>
        <name>yarn.resourcemanager.resource-tracker.address.hadoop6</name>
        <value>hadoop6:23125</value>
    </property>

    <property>
        <name>yarn.resourcemanager.admin.address.hadoop6</name>
        <value>hadoop6:23141</value>
    </property>

    <!-- Node Manager Configs -->
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>2048</value>
    </property>

    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>2</value>
    </property>

    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>86400</value>
    </property>

    <property>
        <name>yarn.log-aggregation.retain-check-interval-seconds</name>
        <value>8640</value>
    </property>

    <property>
        <name>yarn.nodemanager.localizer.address</name>
        <value>0.0.0.0:23344</value>
    </property>

    <property>
        <name>yarn.nodemanager.webapp.address</name>
        <value>0.0.0.0:23999</value>
    </property>
    
    <property>
        <name>yarn.web-proxy.address</name>
        <value>0.0.0.0:8080</value>
    </property>

    <property>
        <name>mapreduce.shuffle.port</name>
        <value>23080</value>
    </property>

    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle,llama_nm_plugin</value>
    </property>

    <property>
        <name>yarn.nodemanager.aux-services.llama_nm_plugin.class</name>
        <value>com.cloudera.llama.nm.LlamaNMAuxiliaryService</value>
    </property>

    <property>
        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>

    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>

    <property>
        <name>yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds</name>
        <value>7200</value>
    </property>

    <property>
        <name>yarn.log.server.url</name>
        <value>http://hadoop5:19888/jobhistory/logs/</value>
    </property>

    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>file:///opt/hadoop/yarn/dn</value>
    </property>

    <property>
        <name>yarn.nodemanager.log-dirs</name>
        <value>file:///opt/hadoop/yarn/logs</value>
    </property>


    <property>
        <name>yarn.nodemanager.remote-app-log-dir</name>
        <value>hdfs://dev-dalu:8020/var/log/hadoop-yarn/apps</value>
    </property>

    <property>
        <name>yarn.web-proxy.address</name>
        <value>hadoop5:41202</value>
    </property>

    <property>
        <name>yarn.resourcemanager.delegation.key.update-interval</name>
        <value>31536000000</value>
    </property>

    <property>
        <name>yarn.resourcemanager.delegation.token.max-lifetime</name>
        <value>31536000000</value>
    </property>

    <property>
        <name>yarn.resourcemanager.delegation.token.renew-interval</name>
        <value>31536000000</value>
    </property>

    <property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>false</value>
    </property>

    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>1024</value>
    </property>

    <property>
        <name>yarn.scheduler.minimum-allocation-vcores</name>
        <value>1</value>
    </property>


    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>4096</value>
    </property>

    <property>
        <name>yarn.application.classpath</name>
        <value>
                $HADOOP_CONF_DIR,
                $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
                $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
                $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
                $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
        </value>
    </property>

    <!-- tez -->
    <property>
        <name>yarn.timeline-service.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>yarn.timeline-service.hostname</name>
        <value>172.31.217.156</value>
    </property>

    <property>
        <name>yarn.timeline-service.http-cross-origin.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>yarn.timeline-service.address</name>
        <value>${yarn.timeline-service.hostname}:10200</value>
    </property>

    <property>
        <name>yarn.timeline-service.webapp.address</name>
        <value>${yarn.timeline-service.hostname}:8188</value>
    </property>

    <property>
        <name>yarn.timeline-service.webapp.https.address</name>
        <value>${yarn.timeline-service.hostname}:8190</value>
    </property>

    <property>
        <name>yarn.timeline-service.handler-thread-count</name>
        <value>10</value>
    </property>

    <property>
        <name>yarn.timeline-service.http-cross-origin.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>yarn.timeline-service.http-cross-origin.allowed-origins</name>
        <value>*</value>
    </property>

    <property>
        <name>yarn.timeline-service.http-cross-origin.allowed-methods</name>
        <value>GET,POST,HEAD,OPTIONS</value>
    </property>

    <property>
        <name>yarn.timeline-service.http-cross-origin.allowed-headers</name>
        <value>X-Requested-With,Content-Type,Accept,Origin,Access-Control-Allow-Origin</value>
    </property>

    <property>
        <name>yarn.timeline-service.http-cross-origin.max-age</name>
        <value>1800</value>
    </property>

    <property>
        <name>yarn.resourcemanager.webapp.cross-origin.enabled</name>
        <value>true</value>
    </property>

</configuration>

        打包/opt/programs/tez_0.7.1/目录下文件为tez-0.7.1.tar.gz

cd /opt/programs/tez_0.7.1
tar czvf tez-0.7.1.tar.gz ./*

        hadoop上创建tez目录

hadoop fs -mkdir /user/tez

        上传/opt/programs/tez_0.7.1/tez-0.7.1.tar.gz到hadoop的/user/tez目录

hadoop fs -put /opt/programs/tez_0.7.1/tez-0.7.1.tar.gz /user/tez/

        拷贝tez-ui-0.7.1.war到tomcat的webapp目录，并解压

#拷贝
cp /opt/programs/apache-tez-0.7.1-src/tez-dist/target/tez-0.7.1/tez-ui-0.7.1.war /opt/programs/tomcat_7.0.68/webapps/

#解压tez-ui-0.7.1.war到tez-ui
cd /opt/programs/tomcat_7.0.68/webapps
jar -xvf tez-ui-0.7.1.war -C tez-ui

        修改tomcat_7.0.68/webapps/tez-ui/scripts/configs.js

timelineBaseUrl: 'http://172.31.217.156:8188',
RMWebUrl: 'http://172.31.217.151:21388',

        部署/opt/programs/tez_0.7.1/tez-0.7.1.tar.gz到整个集群(hadoop1-6)

        重启RM/Job-history/NM

实验word count

        在hadoop上创建相关输出目录

hadoop fs -mkdir /tmp/output

        上传计算文件

hadoop fs -put /opt/programs/apache-tez-0.7.1-src/LICENSE.txt /tmp/

        开始计算

hadoop jar /opt/programs/apache-tez-0.7.1-src/tez-dist/target/tez-0.7.1/tez-examples-0.7.1.jar orderedwordcount /tmp/LICENSE.txt /tmp/output

        计算过程

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/programs/tez_0.7.1/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/06/06 20:14:03 INFO client.TezClient: Tez Client Version: [ component=tez-api, version=0.7.1, revision=${buildNumber}, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=2017-06-06T07:34:50Z ]
17/06/06 20:14:03 INFO impl.TimelineClientImpl: Timeline service address: http://172.31.217.156:8188/ws/v1/timeline/
17/06/06 20:14:04 INFO client.TezClient: Using org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager to manage Timeline ACLs
17/06/06 20:14:04 INFO impl.TimelineClientImpl: Timeline service address: http://172.31.217.156:8188/ws/v1/timeline/
17/06/06 20:14:05 INFO examples.OrderedWordCount: Running OrderedWordCount
17/06/06 20:14:05 INFO client.TezClient: Submitting DAG application with id: application_1496746956128_0005
17/06/06 20:14:05 INFO client.TezClientUtils: Using tez.lib.uris value from configuration: hdfs://dev-dalu:8020/user/tez/tez-0.7.1.tar.gz
17/06/06 20:14:05 INFO client.TezClient: Tez system stage directory hdfs://dev-dalu:8020/tmp/root/tez/staging/.tez/application_1496746956128_0005 doesn't exist and is created
17/06/06 20:14:05 INFO acls.ATSHistoryACLPolicyManager: Created Timeline Domain for History ACLs, domainId=Tez_ATS_application_1496746956128_0005
17/06/06 20:14:05 INFO client.TezClient: Submitting DAG to YARN, applicationId=application_1496746956128_0005, dagName=OrderedWordCount, callerContext={ context=TezExamples, callerType=null, callerId=null }
17/06/06 20:14:05 INFO impl.YarnClientImpl: Submitted application application_1496746956128_0005
17/06/06 20:14:05 INFO client.TezClient: The url to track the Tez AM: http://hadoop5:41202/proxy/application_1496746956128_0005/
17/06/06 20:14:10 INFO impl.TimelineClientImpl: Timeline service address: http://172.31.217.156:8188/ws/v1/timeline/
17/06/06 20:14:10 INFO client.DAGClientImpl: DAG initialized: CurrentState=Running
17/06/06 20:14:10 INFO client.DAGClientImpl: DAG: State: RUNNING Progress: 0% TotalTasks: 3 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
17/06/06 20:14:10 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Tokenizer Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
17/06/06 20:14:10 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
17/06/06 20:14:10 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
17/06/06 20:14:13 INFO client.DAGClientImpl: DAG: State: RUNNING Progress: 33.33% TotalTasks: 3 Succeeded: 1 Running: 1 Failed: 0 Killed: 0
17/06/06 20:14:13 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Tokenizer Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
17/06/06 20:14:13 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 1 Failed: 0 Killed: 0
17/06/06 20:14:13 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
17/06/06 20:14:14 INFO client.DAGClientImpl: DAG: State: SUCCEEDED Progress: 100% TotalTasks: 3 Succeeded: 3 Running: 0 Failed: 0 Killed: 0
17/06/06 20:14:14 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Tokenizer Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
17/06/06 20:14:14 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Summation Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
17/06/06 20:14:14 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Sorter Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
17/06/06 20:14:14 INFO client.DAGClientImpl: DAG completed. FinalState=SUCCEEDED

       

        查看结果

hadoop fs -ls /tmp/output
hadoop fs -cat /tmp/output/part-v002-o000-r-00000

        WEB显示

        Tez-ui

转载于:https://my.oschina.net/guol/blog/915416

weixin_34211761
关注
Hive On Tez
weixin_43920040的博客
03-17 438
组件版本介绍 Hadoop：2.7.2 Hive：2.3.6 Tez:0.9.2 部署tez 下载tez-0.9.2； https://mirror.bit.edu.cn/apache/tez/0.9.2/ 解压并修改名称（强迫症没办法） tar -zxvf apache-tez-0.9.2-bin.tar.gz –C ../app mv apache-tez-0.9.2-bin/ tez...
Hive使用Tez引擎的好处
Laoddaaa的博客
08-16 357
Hive的三种引擎
hive on tez
weixin_42073629的博客
11-01 826
Tez说明 将xyz替换为您正在使用的tez发行版号。例如0.5.0。对于Tez版本0.8.3和更高版本，Tez需要Apache Hadoop版本为2.6.0或更高版本。对于Tez版本0.9.0及更高版本，Tez需要Apache Hadoop版本为2.7.0或更高版本。 关于版本 1.Hadoop 2.7.0+（我的是2.7.1） 2.Tez还需要Protocol Buffers 2.5.0，包括protoc编译器。(如果你下载的是apache-tez-0.9.0-src.tar.gz包则需要编译，需
Hive on Tez切换及性能对比
weixin_38751513的博客
01-13 418
tez
CentOS 6.5 Maven 编译 Apache Tez 0.8.3 踩坑/报错解决记录
ayw0484的博客
06-14 560
　　最近准备学习使用Tez，因此从官网下载了最新的Tez 0.8.3源码，按照安装教程编译使用。平时使用的集群环境是离线的，本打算这一次也进行离线编译，无奈一编译就开始报缺少jar包的错，即使手动下载jar包复制到目录、尝试使用Maven的包安装命令，还是无法消除错误。只得尝试在联网环境下先进行编译（环境与离线集群环境相同），再看编译得到的jar包是否可以在Hadoop离线集群上正常使用...
hive on tez hive运行在tez之上 安装测试
duguyiren3476的专栏
05-26 1269
hive on tez详细配置和运行测试 tez hadoop hive hdfs yarn 环境: hadoop-2.5.2 hive-0.14 tez-0.5.3 hive on tez 的方式有两种安装配置方式: 在hadoop中配置 在hive中配置 比较: 第二种方式:当已经有了稳定的hadoop集群,而不想动这个集群时,可以考虑采用第二种方式配置...
hive on tez 常见报错问题收集
01-26
这里我们将深入探讨五个在Hive on Tez中常见的报错问题及其解决方案。 1. 错误一：Failing because I am unlikely to write too。 这个问题是由于Hive中的一个已知bug，具体问题可以在Apache JIRA的HIVE-16398中...
Hive on Tez 性能优化
最新发布
m0_46237405的博客
05-19 1197
Hive on Tez 性能优化 优化Hive on Tez查询永远不能以一种万能的方法来完成。查询的性能取决于数据的大小、文件类型、查询设计和查询模式。在性能测试期间，要评估和验证配置参数和任何SQL修改。建议在工作负载的性能测试期间一次进行一项更改，并且最好在生产环境中使用它们之前评估调整更改在您的开发和测试环境中的影响。
Hive on Tez配置过程
m0_45847655的博客
08-25 805
此时的hive已经配置完成，可以使用mapreduce引擎进行操作，前提是集群的hdfs,yarn开启，并且启动hive前，要先启动metastore和hiveserver2这两个服务。也可以直接把log4j-slf4j-impl-2.10.0.jar 删了，我就是这么搞的，也没有问题。以上命令启动后会在后台运行，启动之后jps所在的shell命令行，会显示两个runjar,即为启动成功。因为MySQL的密码策略，需要设置一个比较复杂的密码（此处先随便设置一个复杂的密码）是在hadoop的目录下创建。
Hive on Tez map阶段task划分源码分析（map task个数）
lijian222491的博客
04-10 1053
Hive on Tez中map task的划分逻辑在Tez源码中，总体实现逻辑如下：（1）Tez源码中实现map task划分的逻辑为TezSplitGrouper类；具体实现方法为getGroupedSplits；（2）Tez源码中对应该部分的单元测试类为TestGroupedSplits.java（3）选择单元测试中testRepeatableSplits进行单元测试；如下图：（4）该部分可以自由造数据，例如有多少个文件目录，filesplit目录、副本路径位置、文件的大小、机架等等；
hive on spark VS SparkSQL VS hive on tez
wtq1993的博客
09-04 8173
hive on spark VS SparkSQL VS hive on tez 前一篇已经弄好了SparkSQL，SparkSQL也有thriftserver服务，这里说说为啥还选择搞hive-on-spark： SparkSQL-Thriftserver所有结果全部内存，快是快，但是不能满足查询大量数据的需求。如果查询几千万的数据，SparkSQL是搞不定的。而hive-o
Hive-On-Tez性能测试
JavaMan_chen的专栏
07-21 7899
Table of Contents Hive-On-Tez测试 MRR计算模型测试MPJ计算模型测试 Hive-On-Tez测试 在MRR和MPJ计算模型的处理上，TEZ能够提升的性能较为明显，具体测试如下： MRR计算模型测试 测试表格 1.users(id,name,password): 数据总量1千万条
Command execution failed. 错误: 找不到或无法加载主类
热门推荐
wangyue033的博客
03-23 1万+
项目场景： maven项目中启动main方法报错：Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (default-cli) on project rtm-storm: Command execution failed. 错误: 找不到或无法加载主类 com.xxxx.xxxx.storm.bootstrap.TopologyBootstrap 解决方案： maven 设置中，Runner选项里面去掉Delega
hive集成tez
sunlang的博客
06-12 788
hive1集成引擎Tez 3.2.1 安装包准备 1）下载tez的依赖包：http://tez.apache.org 2）拷贝apache-tez-0.9.1-bin.tar.gz到hadoop213的/root/soft目录 [root@hadoop213 soft]$ ls apache-tez-0.9.1-bin.tar.gz 3）将apache-tez-0.9.1-bin.tar.gz上传到HDFS的/tez目录下。 [root@hadoop213 conf]$ hadoop fs -mkdir .
Shuffle failed with too many fetch failures and insufficient progress!
maomao_dreamhigh的博客
10-11 2036
在hive中跑数时候报错 Caused by: java.io.IOException: Reducer_2: Shuffle failed with too many fetch failures and insufficient progress!failureCounts=1, pendingInputs=1, fetcherHealthy=false, reducerProgressedEnough=true, reducerStalled=true at org.apache.tez.runt
Hive的执行引擎Tez、spark出现的问题
cuichunchi的博客
09-15 1767
Hive on Tez中，切换成mr引擎出现以下问题： hadoop版本为2.7.3 tez版本为0.9.1 因为tez安装目录中应该有hadoop-mapreduce-client-core、hadoop-mapreduce-client-common等jar，hadoop中也有hadoop-mapreduce-client-core、hadoop-mapreduce-client-common等jar，确认他们的版本是否相同，如果不同，会有这个问题； 解决：将tez低版本的jar替换成2.7
Hive on Spark
weixin_42073629的博客
10-29 1508
简介 本文主要记录如何安装配置Hive on Spark，在执行以下步骤之前，请先确保已经安装Hadoop集群，Hive，MySQL，JDK，Scala，具体安装步骤不再赘述。 背景 Hive默认使用MapReduce作为执行引擎，即Hive on mr。实际上，Hive还可以使用Tez和Spark作为其执行引擎，分别为Hive on Tez和Hive on Spark。由于MapReduce中间计算均需要写入磁盘，而Spark是放在内存中，所以总体来讲Spark比MapReduce快很多。因此，Hi
Hive on tez 运行报错集锦
cuichunchi的博客
10-26 2015
执行 hive：直接报java.lang.ClassNotFoundException: org.apache.hadoop.mapred.MRVersion CDH集群安装组件都ok，在Hue界面中使用Hive的HQL是都是ok的 但是直接在liunx中，打开客户端命令： 执行 hive：直接报java.lang.ClassNotFoundException: org.apache.hadoop.mapred.MRVersion 解决方案： 因为CDH的hive-site.xml配置文件中没
部署hive on tez
05-26
部署Hive on Tez需要按照以下步骤进行： 1. 安装Tez：在Hadoop集群中安装Tez，可以通过以下命令进行安装： ``` sudo apt-get install tez ``` 2. 配置Hive：在Hive的配置文件hive-site.xml中设置以下属性： ```...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值