zeus作业平台在hadoop2.x中的安装部署

最新推荐文章于 2020-09-10 17:08:13 发布

黑猫guang

最新推荐文章于 2020-09-10 17:08:13 发布

阅读量3.1k

点赞数 1

分类专栏：大数据文章标签： hadoop

本文链接：https://blog.csdn.net/u013623958/article/details/69063249

版权

大数据专栏收录该内容

2 篇文章 0 订阅

订阅专栏

前期工作：

1、hadoop 2.7.3安装。
2、hive 2.1.1安装。
3、java 1.7安装。
4、mysql安装。
5、maven 3.3.9安装。
6、tomcat 7.0安装。
7、安装dos2unix。（命令:yum install -y dos2unix）

zeus安装配置

1、下载zeus

下载zeus：https://github.com/michael8335/zeus2

执行命令解压zeus

unzip zeus2-master.zip
mv zeus2-master /opt/zeus2

2、修改zeus配置

#环境配置，用来标记环境，可以配合Environment使用
zeus.env=test-env
zeus.loggingRoot=/usr/zeuslog   
zeus.loggingLevel=warn
#任务执行文件夹，每次任务都会在此文件夹下新建一个临时文件夹
zeus.localdata.dir=/usr/zeus/run_job_dir
#此处必须是hdfs路径，所有的上传附件都会存放在下面路径上
zeus.hdfsLibPath=/opt/hadoop-2.7.3/bin/hdfs
#环境配置，如果是在一个集群中，需要配置城同一个名称
zeus.schedule.group=test-env
#Scheduler 与 Worker 通信的端口
zeus.connect.port=9887
#hadoop jobtracker地址，填写ip+port 或者 domain+port
zeus.jobtracker=127.0.0.1:50030

#zk配置非必选，可以为空，此处的zk是用来通知任务的成功失败事件的
zeus.zookeeper.host=

拷贝antx.properties 到用户home目录下

3、配置highcharts

vi /opt/zeus2/web/pom.xml

修改路径

<local.highcharts>/opt/zeus2/web/libs/highcharts-1.4.0.jar</local.highcharts>

4、配置zeus数据库连接

vi /opt/zeus2/web/src/main/resources/persistence.xml

修改配置文件

<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"> 
        <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
        <property name="url" value="jdbc:mysql://127.0.0.1:3306/zeus2?createDatabaseIfNotExist=true&amp;useUnicode=true&amp;characterEncoding=utf-8&amp;autoReconnect=true&amp;autoReconnectForPools=true"/>
        <property name="username" value="root"/>
        <property name="password" value="1234"/>
        <property name="maxActive" value="20" />
        <property name="initialSize" value="3" />
        <property name="maxWait" value="60000" />
        <property name="maxIdle" value="15" />
        <property name="minIdle" value="5" />
        <property name="removeAbandoned" value="true" />
        <property name="removeAbandonedTimeout" value="180" />
        <property name="validationQuery" value="SELECT 1" />
        <property name="testOnBorrow" value="true" />
    </bean>

5、修改google-protobuf-java版本

由于安装的hive 2.1.1版本使用的是protobuf-java 2.5。这与zeus源码的版本不一致，需要重新生成报文文件。

修改 zeus-schedule的pom.xml文件

    <dependency> 
        <groupId>com.google.protobuf</groupId>
        <artifactId>protobuf-java</artifactId>
        <version>2.5.0</version> 
    </dependency>

重新下载下载protoc.exe （点击下载），这里我下载的是windos的protoc.exe,所以需要在windows下进行编译。将protoc.exe覆盖“\zeus2-master\schedule\src\main\java”下的protoc.exe。运行CMD,cd到protoc.exe的目录下。运行

protoc.exe  --java_out=. SocketProtocol.proto

将生成的Protocol.java 覆盖到/opt/zeus2/schedule/src/main/java/com/taobao/zeus/socket/protocol。

6、更新pom文件的jar

由于zeus中使用的一些jar和hadoop2.7以及hive2.1.1存在不兼容的问题，所以需要修改pom.xml文件下的一些jar包的版本。
这里贴一下zeus-schedule下的pom文件

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>com.taobao.zeus</groupId>
        <artifactId>zeus</artifactId>
        <version>1.0.0-SNAPSHOT</version>
    </parent>
    <artifactId>zeus-schedule</artifactId>
    <version>2.0.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <name>调度系统内核</name>
    <description>调度系统</description>
    <dependencies>              
        <dependency>
            <groupId>com.taobao.zeus</groupId>
            <artifactId>zeus-client</artifactId>
            <version>2.0.1-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>commons-dbcp</groupId>
            <artifactId>commons-dbcp</artifactId>
            <version>1.4</version>
        </dependency>
        <dependency>
            <groupId>org.jboss.netty</groupId>
            <artifactId>netty</artifactId>
            <version>3.2.4.Final</version>
        </dependency>
        <dependency> 
            <groupId>com.google.protobuf</groupId>
            <artifactId>protobuf-java</artifactId>
            <version>2.5.0</version> 
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.2.0</version>
            <exclusions>
                <exclusion>
                    <groupId>org.eclipse.jdt</groupId>
                    <artifactId>core</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.velocity</groupId>
            <artifactId>velocity</artifactId>
            <version>1.5</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-metastore</artifactId>
            <version>2.1.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-exec</artifactId>
            <version>2.1.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-common</artifactId>
            <version>2.1.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.thrift</groupId>
            <artifactId>libthrift</artifactId>
            <version>0.9.0</version>
        </dependency>
        <dependency>
          <groupId>org.datanucleus</groupId>
          <artifactId>datanucleus-rdbms</artifactId>
          <version>4.1.0-release</version>
        </dependency>


        <dependency>
            <groupId>org.datanucleus</groupId>
            <artifactId>datanucleus-api-jdo</artifactId>
            <version>4.1.0-release</version>
        </dependency>

        <dependency>
          <groupId>org.datanucleus</groupId>
          <artifactId>datanucleus-core</artifactId>
          <version>4.1.0-release</version>
        </dependency>

        <dependency>
          <groupId>javax.jdo</groupId>
          <artifactId>jdo2-api</artifactId>
          <version>2.3-eb</version>
        </dependency>

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring</artifactId>
        </dependency>
        <dependency>
            <groupId>commons-beanutils</groupId>
            <artifactId>commons-beanutils</artifactId>
            <version>1.8.0</version>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.10</version>
        </dependency>
        <!-- -->
        <dependency>
            <groupId>net.sf.json-lib</groupId>
            <artifactId>json-lib</artifactId>
            <version>2.2</version>
            <classifier>jdk15</classifier>
        </dependency> 

        <dependency>
            <groupId>net.sf.ezmorph</groupId>
            <artifactId>ezmorph</artifactId>
            <version>1.0.6</version>
        </dependency>
        <dependency>
            <groupId>org.hibernate</groupId>
            <artifactId>hibernate</artifactId>
            <version>3.2.7.ga</version>
            <exclusions>
                <exclusion>
                    <groupId>cglib</groupId>
                    <artifactId>cglib</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.hibernate</groupId>
            <artifactId>hibernate-annotations</artifactId>
            <version>3.4.0.GA</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.hibernate</groupId>
            <artifactId>ejb3-persistence</artifactId>
            <version>1.0.2.GA</version>
        </dependency>
        <dependency>
            <groupId>org.jboss.javaee</groupId>
            <artifactId>jboss-transaction-api</artifactId>
            <version>1.0.1.GA</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-io</artifactId>
            <version>1.3.2</version>
            <type>jar</type>
        </dependency>
        <dependency>
            <groupId>joda-time</groupId>
            <artifactId>joda-time</artifactId>
        </dependency>
        <dependency>
            <groupId>commons-lang</groupId>
            <artifactId>commons-lang</artifactId>
            <version>2.4</version>
        </dependency>
        <dependency>
            <groupId>commons-fileupload</groupId>
            <artifactId>commons-fileupload</artifactId>
            <version>1.2.2</version>
            <type>jar</type>
        </dependency>
        <dependency>
            <groupId>quartz</groupId>
            <artifactId>quartz</artifactId>
        </dependency>
        <dependency>
            <groupId>net.sf.jopt-simple</groupId>
            <artifactId>jopt-simple</artifactId>
            <version>3.2</version>
            <type>jar</type>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>commons-collections</groupId>
            <artifactId>commons-collections</artifactId>
            <version>3.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
            <version>3.3.6</version>
        </dependency>
        <dependency>
            <groupId>org.unitils</groupId>
            <artifactId>unitils-spring</artifactId>
            <version>3.3</version>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-nop</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.4</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>jcl-over-slf4j</artifactId>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
        </dependency>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>19.0</version>
            <type>jar</type>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>commons-httpclient</groupId>
            <artifactId>commons-httpclient</artifactId>
            <version>3.1</version>
        </dependency>
    </dependencies>
</project>

ok到此为止zeus的相关修改已经完成了。

7、打包部署
进入zeus根目录顺序执行以下命令：

mvn clean:clean
mvn package -Dmaven.test.skip=true
cd deploy
mvn assembly:assembly

将构建好的/deploy/target/zeus.war拷贝到tomcat的webapps下，运行tomcat。
在浏览器中执行输入“http://10.111.66.11:8080/zeus”即可正常访问。

这里写图片描述

问题整理：
1、在运行hive命令时候报错，解决办法：
修改zeus-mastar下的 ProcessJob.java 文件

        if(jobType!=null&&jobType.equals("HiveJob")){

            //Configuration hiveC=ConfUtil.getDefaultCoreSite();
            Configuration hiveC=ConfUtil.getDefaultHiveSite();
            for(String key:hive.keySet()){
                hiveC.set(key, hive.get(key));
            }
            try {
                File xml=new File(dir.getAbsolutePath()+File.separator+"hive-site.xml");
                if(xml.exists()){
                    xml.delete();
                }
                xml.createNewFile();
                hiveC.writeXml(new FileOutputStream(xml));
            } catch (Exception e) {
                log.error("create file hive-site.xml error",e);
            }
        }