黑猴子的家：Oozie Coordinator 周期性调度任务

Coordinator周期性调度任务

1、配置Linux时区以及时间服务器

定时调度，时间一定要同步
https://www.jianshu.com/p/930bb81b3eb1

2、UTC时间

通过修改oozie-site.xml和oozie-console.js文件，显示成能看懂的时间

 

3、配置oozie-site.xml文件

<property>
    <name>oozie.processing.timezone</name>
    <value>GMT+0800</value>
    <description>
        Oozie server timezone. Valid values are UTC and GMT(+/-)####,
        for example 'GMT+0530' would be India
        timezone. All dates parsed and genered dates by Oozie
        Coordinator/Bundle will be done in the specified
        timezone. The default value of 'UTC' should not be changed
        under normal circumtances. If for any reason
        is changed, note that GMT(+/-)#### timezones do not observe DST changes.
    </description>
</property>

解释：修改时区为东八区区时
尖叫提示：该属性去oozie-default.xml中找到即可

4、修改js框架中的关于时间设置的代码

[victor@hadoop102 oozie-4.0.0-cdh5.3.6]$ vim \
oozie-server/webapps/oozie/oozie-console.js

function getTimeZone() {
    Ext.state.Manager.setProvider(new Ext.state.CookieProvider());
    return Ext.state.Manager.get("TimezoneId","GMT+0800");
}

5、重启oozie服务，并重启浏览器

[victor@hadoop102 oozie-4.0.0-cdh5.3.6]$ bin/oozied.sh stop
[victor@hadoop102 oozie-4.0.0-cdh5.3.6]$ bin/oozied.sh start

很重要的操作，把浏览器关了，重新开启，因为浏览器吧oozie-console.js代码缓存了

6、拷贝官方模板配置定时任务

[victor@hadoop102 oozie-4.0.0-cdh5.3.6]$ cp -r examples/apps/cron/ oozie-apps/

7、修改模板job.properties和coordinator.xml以及workflow.xml

job.properties

nameNode=hdfs://hadoop102:9000
jobTracker=hadoop103:8032
queueName=default
examplesRoot=oozie-apps

oozie.coord.application.path=${nameNode}/user/${user.name}/${examplesRoot}/cron
#start：必须设置为未来时间，否则任务失败
start=2018-10-08T17:00+0800
end=2018-10-08T17:00+0800

#hdfs://hadoop102:9000/user/victor/oozie-apps/cron
workflowAppUri=${nameNode}/user/${user.name}/${examplesRoot}/cron

EXEC1=p1.sh
EXEC2=p2.sh

coordinator.xml
最小5分钟

<coordinator-app name="cron-coord" frequency="${coord:minutes(5)}"
    start="${start}" end="${end}" timezone="GMT+0800" xmlns="uri:oozie:coordinator:0.2">
    <action>
        <workflow>
            <app-path>${workflowAppUri}</app-path>
            <configuration>
                <property>
                    <name>jobTracker</name>
                    <value>${jobTracker}</value>
                </property>
                <property>
                    <name>nameNode</name>
                    <value>${nameNode}</value>
                </property>
                <property>
                    <name>queueName</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
        </workflow>
    </action>
</coordinator-app>

frequency ：频率
start：开始时间
end：结束时间
在这个时间段里面，按照频率的设定，周期性调度job任务

workflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.5" name="one-op-wf">
    <start to="p1-shell-node" />
    <action name="p1-shell-node">
        <shell xmlns="uri:oozie:shell-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <exec>${EXEC1}</exec>
            <file>/user/victor/oozie-apps/shell/${EXEC1}#${EXEC1}</file>
            <!-- <argument>my_output=Hello Oozie</argument> -->
            <capture-output />
        </shell>
        <ok to="p2-shell-node" />
        <error to="fail" />
    </action>

    <action name="p2-shell-node">
        <shell xmlns="uri:oozie:shell-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <exec>${EXEC2}</exec>
            <file>/user/victor/oozie-apps/shell/${EXEC2}#${EXEC2}</file>
            <!-- <argument>my_output=Hello Oozie</argument> -->
            <capture-output />
        </shell>
        <ok to="end" />
        <error to="fail" />
    </action>
    <decision name="check-output">
        <switch>
            <case to="end">
                ${wf:actionData('shell-node')['my_output'] eq 'Hello Oozie'}
            </case>
            <default to="fail-output" />
        </switch>
    </decision>
    <kill name="fail">
        <message>Shell action failed, error
            message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <kill name="fail-output">
        <message>Incorrect output, expected [Hello Oozie] but was
            [${wf:actionData('shell-node')['my_output']}]</message>
    </kill>
    <end name="end" />
</workflow-app>

8、上传配置

[victor@hadoop102 oozie-4.0.0-cdh5.3.6]$ /opt/module/cdh\
/hadoop-2.5.0-cdh5.3.6/bin/hdfs dfs -put \
oozie-apps/cron/ /user/admin/oozie-apps

9、启动任务

[victor@hadoop102 oozie-4.0.0-cdh5.3.6]$ bin/oozie job \
-oozie http://hadoop102:11000/oozie \
-config oozie-apps/cron/job.properties \
-run

尖叫提示：oozie允许的最小执行任务的频率是5分钟

10、应该注意的几个坑

（1）start：必须设置为未来时间，否则任务失败
（2）晚上学习，注意0点跳时问题
（3）frequency 最小执行频率 是5分钟