Oozie的使用

最新推荐文章于 2024-08-07 10:17:45 发布

weixin_40652340

最新推荐文章于 2024-08-07 10:17:45 发布

阅读量865

点赞数

分类专栏：大数据

本文链接：https://blog.csdn.net/weixin_40652340/article/details/78889312

版权

大数据专栏收录该内容

37 篇文章 0 订阅

订阅专栏

一：Oozie运行examples。

1、解压示例包
$ tar -zxf oozie-examples.tar.gz -C ./
2、$ bin/hdfs dfs -put /opt/cdh5/oozie-4.0.0-cdh5.3.6/examples/
3、编写job.properties

#      http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

nameNode=hdfs://bigdata-01.yushu.com:8020
jobTracker=bigdata-01.yushu.com:8032
queueName=default
examplesRoot=examples

oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce/workflow.xml
outputDir=map-reduce

${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce/workflow.xml 对应的路径
hdfs://bigdata-senior01.ibeifeng.com:8020/user/beifeng/examples/apps/map-reduce/workflow.xml
4、运行 (job.properties 运行的是linux本地的job文件报错的话看 logs/oozie.log的日志文件查看)
bin/oozie job -oozie http://bigdata-01.yushu.com:11000/oozie -config examples/apps/map-reduce/job.properties -run
5、修改oozie-site.xml
<property>
<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
<value>*=/opt/cdh5/hadoop-2.5.0-cdh5.3.6/etc/hadoop/</value>
</property>
-》重启oozie bin/oozied.sh stop bin/oozied.sh start
6、oozie机制
默认一个workflow会先启动一个MR-》对WF进行封装，初始化工作

二：oozie自定义MapReduce WorkFlow
1、拷贝官方提供的example的示例
2、将Java代码打包放在lib目录下
3、修改job.properties

#

nameNode=hdfs://bigdata-01.yushu.com:8020
jobTracker=bigdata-01.yushu.com:8032
queueName=default
examplesRoot=user/ds/application/apps/mr-wf

oozie.wf.application.path=${nameNode}/${examplesRoot}/workflow.xml
inputDir=input
outputDir=output-oozie

4、修改 workflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.2" name="map-reduce-wf">
    <start to="mr-node"/>
    <action name="mr-node">
        <map-reduce>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/${outputDir}"/>
            </prepare>
            <configuration>
                <property>
                    <name>mapreduce.job.queuename</name>
                    <value>${queueName}</value>
                </property>
				
		<property>
                    <name>mapred.mapper.new-api</name>
                    <value>true</value>
                </property>
				
		<property>
                    <name>mapred.reducer.new-api</name>
                    <value>true</value>
                </property>
				
				
                <property>
                    <name>mapreduce.job.map.class</name>
                    <value>com.bigdata.mapreduce.WCMapReduce$WCMapper</value>
                </property>
                <property>
                    <name>mapreduce.job.reduce.class</name>
                    <value>com.bigdata.mapreduce.WCMapReduce$WCReducer</value>
                </property>
				
		<property>
                    <name>mapreduce.map.output.key.class</name>
                    <value>org.apache.hadoop.io.Text</value>
                </property>
		<property>
                    <name>mapreduce.map.output.value.class</name>
                    <value>org.apache.hadoop.io.IntWritable</value>
                </property>
				
		<property>
                    <name>mapreduce.job.output.key.class</name>
                    <value>org.apache.hadoop.io.Text</value>
                </property>
		<property>
                    <name>mapreduce.job.output.value.class</name>
                    <value>org.apache.hadoop.io.IntWritable</value>
                </property>
				
                <property>
                    <name>mapreduce.input.fileinputformat.inputdir</name>
                    <value>${nameNode}/${inputDir}</value>
                </property>
                <property>
                    <name>mapreduce.output.fileoutputformat.outputdir</name>
                    <value>${nameNode}/${outputDir}</value>
                </property>
            </configuration>
        </map-reduce>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

5、放jar包
6、上传目录到HDFS
7、运行
export OOZIE_URL="http://bigdata-01.yushu.com:11000/oozie"
bin/oozie job -config apps/app/mrwf/job.properties -run

三、oozie自定义Shell WorkFlow

job.properties配置文件内容：

#

nameNode=hdfs://bigdata-01.yushu.com:8020
jobTracker=bigdata-01.yushu.com:8032
queueName=default
examplesRoot=user/ds/application/apps/shell-wf

oozie.wf.application.path=${nameNode}/${examplesRoot}/workflow.xml
EXEC=oozie.sh

workflow.xml文件内容：

<workflow-app xmlns="uri:oozie:workflow:0.4" name="shell-wf">
    <start to="shell-node"/>
    <action name="shell-node">
        <shell xmlns="uri:oozie:shell-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <exec>${EXEC}</exec>
            <file>${EXEC}#${EXEC}</file> <!--Copy the executable to compute node's current working directory -->
        </shell>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>



oozie.sh文件内容：

#!/bin/bash
HIVE_HOME=/opt/cdh5/hive-0.13.1-cdh5.3.6
$HIVE_HOME/bin/hive -e "show databases" >>/opt/datas/oozie-hive.txt



最后执行命令：bin/oozie job -config application/apps/shell-wf/job.properties -run








四、oozie修改时区



1、时间调度

-》coordinate

-》基于时间的调度

-》基于数据可用性的调度

-》起始时间

-》结束时间

-》时间频率

2、时区

3、oozie支持从UTC和GMT

4、将oozie的默认时区指定为GMT

5、命令：date -R

-》如果命令没有输出+0800的显示

-》删除时区文件/etc/localtime 

-》删除后重新生成，创建一个软连接ln -s

/usr/share/zoneinfo/Asia/shanghai

6、修改oozie时区

修改oozie-site.xml

<property>

        <name>oozie.processing.timezone</name>

        <value>GMT+0800</value>      

    </property>

-》重启oozie



-》修改oozie-server/webapps/oozie/oozie-console.js代码

function getTimeZone() {

    Ext.state.Manager.setProvider(new Ext.state.CookieProvider());

    return Ext.state.Manager.get("TimezoneId","GMT+0800");

}








五、oozie Coordinator讲解

-》修改oozie-site.xml

<property>

        <name>oozie.service.coord.check.maximum.frequency</name>

        <value>false</value>

</property>



* * * * *


job.properties文件内容：



nameNode=hdfs://bigdata-01.yushu.com:8020
jobTracker=bigdata-01.yushu.com:8032
queueName=default
examplesRoot=user/ds/application/apps/cron-wf

oozie.coord.application.path=${nameNode}/${examplesRoot}/coordinator.xml
start=2017-06-26T19:50+0800
end=2017-06-26T19:58+0800
workflowAppUri=${nameNode}/${examplesRoot}/workflow.xml



coordinator.xml文件内容：frequency="${coord:minutes(2)}  表示每隔两分钟执行

<coordinator-app name="cron-coord" frequency="${coord:minutes(2)}" start="${start}" end="${end}" timezone="GMT+0800"
                 xmlns="uri:oozie:coordinator:0.2">
        <action>
        <workflow>
            <app-path>${workflowAppUri}</app-path>
            <configuration>
                <property>
                    <name>jobTracker</name>
                    <value>${jobTracker}</value>
                </property>
                <property>
                    <name>nameNode</name>
                    <value>${nameNode}</value>
                </property>
                <property>
                    <name>queueName</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
        </workflow>
    </action>
</coordinator-app>

workflow.xml文件内容：
<workflow-app xmlns="uri:oozie:workflow:0.5" name="one-op-wf">
    <start to="action1"/>
    <action name="action1">
        <fs/>
    <ok to="end"/>
    <error to="end"/>
    </action>
    <end name="end"/>
</workflow-app>






参考文献：http://blog.csdn.net/nsrainbow/article/details/43746111
参考文献2：https://www.zybuluo.com/tsing1226/note/263232
参考文献3：http://shiyanjun.cn/archives/684.html oozie

Coordinator Job