oozie学习笔记

最新推荐文章于 2022-03-09 20:54:18 发布

weixin_34174105

最新推荐文章于 2022-03-09 20:54:18 发布

阅读量102

点赞数

文章标签：大数据数据库 java

##########################################################################################################
##########################################################################################################

Oozie安装部署：
1、安装需求 System Requirements

Unix (tested in Linux and Mac OS X)
Java 1.6+
Hadoop
Apache Hadoop (tested with 1.0.0 & 0.23.1)
ExtJS library (optional, to enable Oozie webconsole)
ExtJS 2.2
The Java 1.6+ bin directory should be in the command path.

2、上传文件并且解压安装包：
[hadoop@db01 softwares]$ pwd
/opt/softwares
[hadoop@db01 softwares]$ tar -zxvf oozie-4.0.0-cdh5.3.6.tar.gz -C /opt/cdh-5.3.6/

3、在hadoop的core-site.xml配置文件中添加如下内容,重启hadoop集群：

<property>
    <name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name>
    <value>[OOZIE_SERVER_HOSTNAME]</value>
</property>
<property>
    <name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name>
    <value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value>
</property>

--------------------------------------------------------

      <property>
        <name>hadoop.proxyuser.hadoop.hosts</name>
        <value>db01</value>
    </property>
    <property>
        <name>hadoop.proxyuser.hadoop.groups</name>
        <value>*</value>
    </property>

4、在oozie安装目录解压hadooplibs jar包：

[hadoop@db01 oozie-4.0.0]$ tar -zxvf oozie-hadooplibs-4.0.0-cdh5.3.6.tar.gz

5、创建libext目录

[hadoop@db01 oozie-4.0.0]$ pwd
/opt/cdh-5.3.6/oozie-4.0.0
[hadoop@db01 oozie-4.0.0]$ mkdir libext/

6、cp4步骤解压目录下相应hadooplibs下jar包到5步骤创建libext目录下：
[hadoop@db01 oozie-4.0.0]$ cp -r oozie-4.0.0-cdh5.3.6/hadooplibs/hadooplib-2.5.0-cdh5.3.6.oozie-4.0.0-cdh5.3.6/* libext/

7、If using the ExtJS library copy the ZIP file to the libext/ directory.
[hadoop@db01 oozie-4.0.0]$ cp /opt/softwares/ext-2.2.zip libext/

8、打包操作

[hadoop@db01 oozie-4.0.0]$ bin/oozie-setup.sh prepare-war

9、启动hadoop服务

略

10、
[hadoop@db01 oozie-4.0.0]$ cp /opt/cdh-5.3.6/hadoop-2.5.0/etc/hadoop/core-site.xml /opt/cdh-5.3.6/hadoop-2.5.0/etc/hadoop/hdfs-site.xml /opt/cdh-5.3.6/oozie-4.0.0/conf/

11、

----------------------------
[hadoop@db01 oozie-4.0.0]$ bin/oozie-setup.sh sharelib create -fs hdfs://db01:8020 -locallib oozie-sharelib-4.0.0-cdh5.3.6-yarn.tar.gz
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cdh-5.3.6/oozie-4.0.0/libtools/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cdh-5.3.6/oozie-4.0.0/libtools/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cdh-5.3.6/oozie-4.0.0/libext/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
the destination path for sharelib is: /user/hadoop/share/lib/lib_20170324165042
---------------------------------------------------------------------------------------

12、创建数据库
bin/ooziedb.sh create -sqlfile oozie.sql -run DB Connection

13、启动oozie
[hadoop@db01 oozie-4.0.0]$ bin/oozied.sh start

14、oozie-site.xml配置hdoop conf目录：
<property>
        <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
        <value>*=/opt/cdh-5.3.6/hadoop-2.5.0/etc/hadoop</value>
        <description>
            Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of
            the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is
            used when there is no exact match for an authority. The HADOOP_CONF_DIR contains
            the relevant Hadoop *-site.xml files. If the path is relative is looked within
            the Oozie configuration directory; though the path can be absolute (i.e. to point
            to Hadoop client conf/ directories in the local filesystem.
        </description>
    </property>

15、重新启动oozie
[hadoop@db01 oozie-4.0.0]$ bin/oozied.sh stop
[hadoop@db01 oozie-4.0.0]$ bin/oozied.sh start

http://db01:11000 登录console

[hadoop@db01 oozie-4.0.0]$ bin/oozie admin -oozie http://localhost:11000/oozie -status
System mode: NORMAL

16、oozie使用mysql数据库存储元数据
1)修改配置文件：

     <property>
        <name>oozie.service.JPAService.jdbc.driver</name>
        <value>com.mysql.jdbc.Driver</value>
        <description>
            JDBC driver class.
        </description>
    </property>

    <property>
        <name>oozie.service.JPAService.jdbc.url</name>
        <value>jdbc:mysql://db01:3306/oozie</value>
        <description>
            JDBC URL.
        </description>
    </property>

    <property>
        <name>oozie.service.JPAService.jdbc.username</name>
        <value>root</value>
        <description>
            DB user name.
        </description>
    </property>

    <property>
        <name>oozie.service.JPAService.jdbc.password</name>
        <value>mysql</value>
        <description>
            DB user password.

            IMPORTANT: if password is emtpy leave a 1 space string, the service trims the value,
                       if empty Configuration assumes it is NULL.
        </description>
    </property>

2）cp mysql 驱动到libext/下：
cp /opt/cdh-5.3.6/hive-0.13.1/lib/mysql-connector-java-5.1.27-bin.jar /opt/cdh-5.3.6/oozie-4.0.0/libext/
3）穿件数据库：
bin/ooziedb.sh create -sqlfile oozie.sql -run DB Connection
4）打包上传hdfs
bin/oozie-setup.sh prepare-war
bin/oozie-setup.sh sharelib create -fs hdfs://db01:8020 -locallib oozie-sharelib-4.0.0-cdh5.3.6-yarn.tar.gz
4）重启即可
[hadoop@db01 oozie-4.0.0]$ bin/oozied.sh stop
[hadoop@db01 oozie-4.0.0]$ bin/oozied.sh start
********************************************************************************************
Examples:

bin/oozie job -oozie http://chavin.king:11000/oozie -config examples/apps/map-reduce/job.properties -run

insert overwrite directory '/user/hadoop/hive/output'
select empno,ename,mgr,job,sal,comm,deptno from chavin.emp;

bin/sqoop import --connect jdbc:mysql://db01:3306/chavin --username root --password mysql --table emp --target-dir ${nameNode}/${oozieDataRoot}/${outputDir} --num-mappers 1 --as-parquetfile

regsvr32.exe C:\Program Files (x86)\IDM Computer Solutions\UltraEdit\wodFtpDLX.dll

import --connect jdbc:mysql://db01:3306/chavin
--username root --password mysql --table emp
--target-dir ${nameNode}/${oozieDataRoot}/${outputDir}
--num-mappers 1 --fields-terminated-by "/t"

export --connect jdbc:mysql://chavin.king:3306/chavin --username root --password mysql --table emp --num-mappers 1 --fields-terminated-by "/t" --export-dir /user/hadoop/oozie/datas/bi-select-emp/output

<file>db.hsqldb.properties#db.hsqldb.properties</file>
<file>db.hsqldb.script#db.hsqldb.script</file>

bin/sqoop export \
--connect jdbc:mysql://db01:3306/chavin \
--username root \
--password mysql \
--table emp01 \
--export-dir /user/hadoop/sqoop/import/emp

create table chavin.emp02
(
EMPNO    int,
ENAME    string,
JOB      string,
) row format delimited fields terminated by '\t';

create table chavin.emp01
(
EMPNO    int,
ENAME    string,
JOB      string
) row format delimited fields terminated by '\t';

#########################################################################################################