oozie-5.0.0安装部署

最新推荐文章于 2020-02-10 13:02:43 发布

chiyong9824

最新推荐文章于 2020-02-10 13:02:43 发布

阅读量969

点赞数

文章标签：大数据数据库 java

原文链接：https://my.oschina.net/u/3825598/blog/1865904

版权

1.下载oozie5.0

1.1 安装基础环境

oozie5.0.0的编译安装环境需求如下：

System Requirements:
Unix box (tested on Mac OS X and Linux)
Java JDK 1.8+
Maven 3.0.1+
Hadoop 2.6.0+
Pig 0.10.1+
JDK commands (java, javac) must be in the command path.
The Maven command (mvn) must be in the command path.

1.2 下载oozie

官网下载, 选择一个国内镜像下载文件oozie-5.0.0.tar.gz 即可

2.编译oozie

2.1 解压文件

将oozie的安装包解压到/usr/local目录下： /usr/local/oozie-5.0.0
目录结构如下:

[root@cm-server-1 oozie-5.0.0]# ll
total 352
drwxr-xr-x  2 502 wheel   4096 Jul 25 16:23 bin
drwxr-xr-x  2 502 wheel   4096 Jul 25 16:23 builds
drwxr-xr-x  3 502 wheel   4096 Jul 25 16:23 client
drwxr-xr-x  3 502 wheel   4096 Jul 25 16:23 core
drwxr-xr-x  3 502 wheel   4096 Jul 25 16:23 distro
drwxr-xr-x  3 502 wheel   4096 Jul 25 16:23 docs
drwxr-xr-x  3 502 wheel   4096 Jul 25 16:23 examples
-rw-r--r--  1 502 wheel   1330 Mar 27 20:22 findbugs-filter.xml
-rw-r--r--  1 502 wheel  37664 Mar 27 20:22 LICENSE.txt
drwxr-xr-x  3 502 wheel   4096 Jul 25 16:23 minitest
-rw-r--r--  1 502 wheel    450 Mar 27 20:22 NOTICE.txt
-rw-r--r--  1 502 wheel 105126 Mar 27 20:22 pom.xml
-rw-r--r--  1 502 wheel   3084 Mar 27 20:22 README.txt
-rw-r--r--  1 502 wheel 137736 Mar 27 20:22 release-log.txt
drwxr-xr-x  3 502 wheel   4096 Jul 25 16:23 server
drwxr-xr-x 12 502 wheel   4096 Jul 25 16:23 sharelib
-rw-r--r--  1 502 wheel   2653 Mar 27 20:22 source-headers.txt
drwxr-xr-x  3 502 wheel   4096 Mar 27 20:22 src
drwxr-xr-x  3 502 wheel   4096 Jul 25 16:23 tools
drwxr-xr-x  3 502 wheel   4096 Jul 25 16:23 webapp
drwxr-xr-x  3 502 wheel   4096 Jul 25 16:23 zookeeper-security-tests
[root@cm-server-1 oozie-5.0.0]#

2.2 编译代码

bin/mkdistro.sh -Dhadoop.version=2.6.0 -Dhbase.version=1.2.0 -Dspark.version=2.1.0 -DskipTests

（本来要添加hive的版本为1.1.0的，可以版本太低了吧编译不通过于是给去掉了，默认hive应该是1.2.0的了）

2.3 获取打包版本

编译完成之后在获取打包文件 oozie-5.0.0/distro/target/oozie-5.0.0-distro.tar.gz

3. 安装oozie server

3.1 环境准备

(1)将文件移动到/user/local 目录

[root@cm-server-1 local]# cp oozie-5.0.0/distro/target/oozie-5.0.0-distro.tar.gz /user/local

(2)删除/user/local 目录下的oozie-5.0.0文件，因为解压oozie-5.0.0-distro.tar.gz文件解压后的文件名也是oozie-5.0.0，所以为防止文件冲突，将/user/local下的oozie-5.0.0文件删除

[root@cm-server-1 local]# rm -rf oozie-5.0.0

（3）将ExtJS 2.2的zip包添加到文件夹libext中（如果不需要要oozie的web功能可以不添加）可以直接下载后复制到libext文件夹中，亦可以用如下wget命令下载

[root@cm-server-1 local]# wget  http://archive.cloudera.com/gplextras/misc/ext-2.2.zip

（4）将hadoop对应版本下的lib目录下的文件复制libext
下载对于版本的hadoop，本文是2.6.0

[root@cm-server-1 local]# wget https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz

3.2 为oozie程序配置hadoop集群的代理用户

在 Hadoop的core-site.xml中添加如下两项配置
官网给的参考

 <!-- OOZIE -->
  <property>
    <name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name>
    <value>[OOZIE_SERVER_HOSTNAME]</value>
  </property>
  <property>
    <name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name>
    <value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value>
  </property>

其中，[OOZIE_SERVER_USER]需要改为后面启动oozie的用户,如设置为oozie 我们可以配置为如下：

<property>
    <name>hadoop.proxyuser.oozie.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.oozie.groups</name>
    <value>*</value>
  </property>

不重启hadoop集群，而使配置生效

hdfs dfsadmin -refreshSuperUserGroupsConfiguration
yarn rmadmin -refreshSuperUserGroupsConfiguration

注：因为我们部署的系统中已经配置了这两项且[OOZIE_SERVER_USER]设置为oozie了，因此设置代理用户这一步就省了了

3.3 解压包

首先修改hadoop以及oozie压缩包的权限（这个好像有点多余）

[root@cm-server-1 local]# chmod 777 hadoop-2.6.0.tar.gz 
[root@cm-server-1 local]# chmod 777 oozie-5.0.0-distro.tar.gz

(1) 解压oozie-5.0.0-distro.tar.gz到/user/local 目录下

[root@cm-server-1 local]# tar zxvf oozie-5.0.0-distro.tar.gz

解压后的目录结构如下：

[root@cm-server-1 local]# ll oozie-5.0.0
total 335732
drwxr-xr-x 2 root root       4096 Jul 25 17:25 bin
drwxr-xr-x 4  502 wheel      4096 Jul 25 17:25 conf
-rwsrwsrwt 1 root root    8807259 Jul 25 16:31 docs.zip
drwxr-xr-x 4 root root       4096 Jul 25 17:25 embedded-oozie-server
drwxr-xr-x 2 root root       4096 Jul 25 17:25 libtools
-rw-r--r-- 1  502 wheel     37664 Mar 27 20:22 LICENSE.txt
-rw-r--r-- 1  502 wheel       450 Mar 27 20:22 NOTICE.txt
-rwsrwsrwt 1 root root    9477951 Jul 25 16:30 oozie-client-5.0.0.tar.gz
drwxr-xr-x 2 root root       4096 Jul 25 17:25 oozie-core
-rwsrwsrwt 1 root root      47569 Jul 25 16:31 oozie-examples.tar.gz
-rwsrwsrwt 1 root root  325239739 Jul 25 16:31 oozie-sharelib-5.0.0.tar.gz
-rw-r--r-- 1  502 wheel      3084 Mar 27 20:22 README.txt
-rw-r--r-- 1  502 wheel    137736 Mar 27 20:22 release-log.txt
[root@cm-server-1 local]#

（2）解压hadoop到/user/local 目录下解压hadoop，官网有这么一句
Expand the Hadoop distribution tar.gz (as the Oozie Unix user).
应该是解压hadoop的用户与运行oozie服务的用户想同吧，因为我们规划用oozie这个用户来部署运行oozie服务，所以处解压也用oozie用户吧(我还是用的root，因为我的oozie没有写local目录的权限，直接用root也没什么可以安装)

[root@cm-server-1 local]# tar zxvf hadoop-2.6.0.tar.gz

3.4 添加libext

官网
Create a libext/ directory in the directory where Oozie was expanded.

If using the ExtJS library copy the ZIP file to the libext/ directory. If hadoop and hcatalog libraries are not already included in the war, add the corresponding libraries to libext/ directory.

按照官网的说明进行如下操作
修改oozie-5.0.0文件夹权限（oozie用户操作需要些权限，embedded-oozie-server文件夹主要是启动时写入pid文件）

[root@cm-server-1 local]# chmod 777 oozie-5.0.0
[root@cm-server-1 local]# chmod 777 oozie-5.0.0/embedded-oozie-server/

在解压的文件夹下面添加libext文件夹

[root@cm-server-1 local]$ mkdir oozie-5.0.0/libext

(1) 添加ext2.2

[root@cm-server-1 local]# cp ext-2.2.zip oozie-5.0.0/libext/

(2)添加hadoop lib下的包

[root@cm-server-1 local]# cp hadoop-2.6.0/share/hadoop/*/*.jar oozie-5.0.0/libext/
[root@cm-server-1 local]# cp hadoop-2.6.0/share/hadoop/*/lib/*.jar oozie-5.0.0/libext/

（3）添加对于数据库的驱动
下载对应的驱动到libext文件夹，本文是postgresql，自带的不用下载了

3.5配置参数

(1)之后再在oozie-site.xml中添加如下代码

<property>
        <name>oozie.service.JPAService.jdbc.driver</name>
        <value>org.postgresql.Driver</value>
        <description>
            JDBC driver class.
        </description>
    </property>

    <property>
        <name>oozie.service.JPAService.jdbc.url</name>
        <value>jdbc:postgresql://*.*.*.*:5432/oozie?createDatabaseIfNotExist=true</value>
        <description>
            JDBC URL.
        </description>
    </property>

    <property>
        <name>oozie.service.JPAService.jdbc.username</name>
        <value>admin</value>
        <description>
            DB user name.
        </description>
    </property>

    <property>
        <name>oozie.service.JPAService.jdbc.password</name>
        <value>123456</value>
        <description>
            DB user password.
        </description>
    </property>

   <property>
        <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
        <value>*=/etc/hadoop/conf</value>
        <description>
            Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of
            the Hadoop service (JobTracker, YARN, HDFS). The wildcard '*' configuration is
            used when there is no exact match for an authority. The HADOOP_CONF_DIR contains
            the relevant Hadoop *-site.xml files. If the path is relative is looked within
            the Oozie configuration directory; though the path can be absolute (i.e. to point
            to Hadoop client conf/ directories in the local filesystem.
        </description>
    </property>


    <property>
        <name>oozie.service.HadoopAccessorService.action.configurations</name>
        <value>*=/etc/hadoop/conf</value>
        <description>
            Comma separated AUTHORITY=ACTION_CONF_DIR, where AUTHORITY is the HOST:PORT of
            the Hadoop MapReduce service (JobTracker, YARN). The wildcard '*' configuration is
            used when there is no exact match for an authority. The ACTION_CONF_DIR may contain
            ACTION.xml files where ACTION is the action type ('java', 'map-reduce', 'pig',
            'hive', 'sqoop', etc.). If the ACTION.xml file exists, its properties will be used
            as defaults properties for the action. If the path is relative is looked within
            the Oozie configuration directory; though the path can be absolute (i.e. to point
            to Hadoop client conf/ directories in the local filesystem.
        </description>
    </property>

    <property>
        <name>oozie.service.SparkConfigurationService.spark.configurations</name>
        <value>*=/etc/spark2/conf</value>
        <description>
            Comma separated AUTHORITY=SPARK_CONF_DIR, where AUTHORITY is the HOST:PORT of
            the ResourceManager of a YARN cluster. The wildcard '*' configuration is
            used when there is no exact match for an authority. The SPARK_CONF_DIR contains
            the relevant spark-defaults.conf properties file. If the path is relative is looked within
            the Oozie configuration directory; though the path can be absolute.  This is only used
            when the Spark master is set to either "yarn-client" or "yarn-cluster".
        </description>
    </property>

(2)运行ooziedb.sh命令行工具创建数据库

[root@cm-server-1 oozie-5.0.0]# bin/ooziedb.sh create -sqlfile oozie.sql -run

运行完命令之后会在数据库中生成表，并在本目录下生成oozie.sql文件

3.5 上传jar到hdfs

上传sharelib到hdfs中，这里上传的jar包都是用户程序通过oozie运行时用到的。而libext中添加的hadoopjar是oozie系统本身用到的，oozie相当于hadoop的一个调用客户端

[oozie@cm-server-1 oozie-5.0.0]$ sudo -u oozie bin/oozie-setup.sh sharelib create -fs hdfs://cm-server-1.pm.com:8020 -locallib oozie-sharelib-5.0.0.tar.gz

执行完之后再hafs中产生如下/ user/ oozie/ share/ lib/ lib_20180726163837 目录

4 运行

4.1 环境准备

[root@cm-server-1 oozie-5.0.0]# sudo -u oozie bin/oozie-setup.sh

运行成功之后会出现INFO: Oozie is ready to be started的信息
这一步会做一些运行前准备工作，将libext中的文件copy到jetty的webapp/WEB-INF/lib下的lib目录中

注意：运行成功之后webapp/WEB-INF/lib下的jar会有冲突，需要将从hadoop添加的包手动删除

4.2 启动

 sudo -u oozie bin/oozied.sh start

4.3 测试

bin/oozie admin -oozie http://localhost:11000/oozie -status

输出System mode: NORMAL,表示系统正常

参考文献

http://oozie.apache.org/docs/5.0.0/DG_QuickStart.html

https://www.cnblogs.com/juncaoit/p/6122187.html

转载于:https://my.oschina.net/u/3825598/blog/1865904

chiyong9824

关注

0
点赞
踩
2

收藏

觉得还不错? 一键收藏
0
评论
oozie-5.0.0安装部署

1.下载oozie5.01.1 安装基础环境oozie5.0.0的编译安装环境需求如下：System Requirements:Unix box (tested on Mac OS X and Linux)Java JDK 1.8+Maven 3.0.1+Hadoop 2.6....
复制链接

扫一扫