Oozie 集成 Shell

sun_xo

已于 2022-07-13 16:05:22 修改

阅读量164

点赞数

文章标签： oozie hadoop

于 2022-06-29 18:38:13 首次发布

本文链接：https://blog.csdn.net/sun_xo/article/details/125527175

版权

1) prepare shell case
$ cd ~/work/oozie-5.2.1
$ tree oozie/apps/shell

oozie/apps/shell
├── hello.sh
├── job.properties
└── workflow.xml

$ cat oozie/apps/shell/hello.sh

#!/bin/sh

echo "my_output=Hello Oozie"

$ cat oozie/apps/shell/job.properties

nameNode=hdfs://localhost:9000
resourceManager=localhost:8032
queueName=default
oozieRoot=user/${user.name}/oozie

oozie.wf.application.path=${nameNode}/${oozieRoot}/apps/shell
EXEC=hello.sh

$ cat oozie/apps/shell/workflow.xml

<workflow-app xmlns="uri:oozie:workflow:1.0" name="shell-wf">
    <start to="shell-node"/>
    <action name="shell-node">
        <shell xmlns="uri:oozie:shell-action:1.0">
            <resource-manager>${resourceManager}</resource-manager>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <exec>${EXEC}</exec>
            <file>${nameNode}/${oozieRoot}/apps/shell/${EXEC}#${EXEC}</file>
            <capture-output/>
        </shell>
        <ok to="check-output"/>
        <error to="fail"/>
    </action>
    <decision name="check-output">
        <switch>
            <case to="end">
                ${wf:actionData('shell-node')['my_output'] eq 'Hello Oozie'}
            </case>
            <default to="fail-output"/>
        </switch>
    </decision>
    <kill name="fail">
        <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <kill name="fail-output">
        <message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

2) upload to hdfs
$ hdfs dfs -put oozie/apps/shell oozie/apps/

3) run and check
$ bin/oozie job -config oozie/apps/shell/job.properties -run
job: 0000004-220629164930563-oozie-sun_-W
$ bin/oozie job -info 0000004-220629164930563-oozie-sun_-W

Job ID : 0000004-220629164930563-oozie-sun_-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : shell-wf
App Path      : hdfs://localhost:9000/user/sun_xo/oozie/apps/shell
Status        : SUCCEEDED
Run           : 0
User          : sun_xo
Group         : -
Created       : 2022-06-29 09:22 GMT
Started       : 2022-06-29 09:22 GMT
Last Modified : 2022-06-29 09:22 GMT
Ended         : 2022-06-29 09:22 GMT
CoordAction ID: -

Actions
------------------------------------------------------------------------------------------------------------------------------------
ID                                                                            Status    Ext ID                 Ext Status Err Code  
------------------------------------------------------------------------------------------------------------------------------------
0000004-220629164930563-oozie-sun_-W@:start:                                  OK        -                      OK         -         
------------------------------------------------------------------------------------------------------------------------------------
0000004-220629164930563-oozie-sun_-W@shell-node                               OK        application_1656492227290_0007SUCCEEDED  -         
------------------------------------------------------------------------------------------------------------------------------------
0000004-220629164930563-oozie-sun_-W@check-output                             OK        -                      end        -         
------------------------------------------------------------------------------------------------------------------------------------
0000004-220629164930563-oozie-sun_-W@end                                      OK        -                      OK         -         
------------------------------------------------------------------------------------------------------------------------------------

Note:
Shell action’s stdout and stderr output are redirected to the Oozie Launcher map-reduce job task STDOUT that runs the shell command
The output can be found under hdfs://localhost:9000/tmp/sun_xo/logs/application_1656492227290_0007/
application_1656492227290_0007 is ext_id of shell-node