黑猴子的家：Oozie 调度 MapReduce 任务

使用Oozie调度MapReduce任务

1、找到一个可以运行的mapreduce任务的jar包

这里用官网的hadoop-mapreduce-examples-2.5.0-cdh5.3.6.jar包（可以用官方的，也可以是自己写的）

2、拷贝官方模板到oozie-apps

[victor@hadoop102 oozie-4.0.0-cdh5.3.6]$ cp -r examples/apps/map-reduce/ oozie-apps/

3、测试一下wordcount在yarn中的运行

[victor@hadoop102 hadoop]$ bin/yarn jar \
share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.6.jar \
wordcount /input/ /output/

4、配置map-reduce任务的job.properties以及workflow.xml

job.properties

[victor@hadoop102 oozie-4.0.0-cdh5.3.6]$ vim oozie-apps/map-reduce/job.properties
nameNode=hdfs://hadoop102:9000
jobTracker=hadoop103:8032
queueName=default
examplesRoot=oozie-apps
#hdfs://hadoop102:9000/user/victor/oozie-apps/map-reduce/workflow.xml
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/map-reduce/workflow.xml
outputDir=map-reduce

workflow.xml

[victor@hadoop102 oozie-4.0.0-cdh5.3.6]$ vim oozie-apps/map-reduce/workflow.xml
<workflow-app xmlns="uri:oozie:workflow:0.2" name="map-reduce-wf">
    <start to="mr-node" />
    <action name="mr-node">
        <map-reduce>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/output/" />
            </prepare>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
                <!-- 配置调度MR任务时，使用新的API -->
                <property>
                    <name>mapred.mapper.new-api</name>
                    <value>true</value>
                </property>

                <property>
                    <name>mapred.reducer.new-api</name>
                    <value>true</value>
                </property>

                <!-- 指定Job Key输出类型 -->
                <property>
                    <name>mapreduce.job.output.key.class</name>
                    <value>org.apache.hadoop.io.Text</value>
                </property>

                <!-- 指定Job Value输出类型 -->
                <property>
                    <name>mapreduce.job.output.value.class</name>
                    <value>org.apache.hadoop.io.IntWritable</value>
                </property>

                <!-- 指定输入路径 -->
                <property>
                    <name>mapred.input.dir</name>
                    <value>/input/</value>
                </property>

                <!-- 指定输出路径 -->
                <property>
                    <name>mapred.output.dir</name>
                    <value>/output/</value>
                </property>

                <!-- 指定Map类 -->
                <property>
                    <name>mapreduce.job.map.class</name>
                    <value>org.apache.hadoop.examples.WordCount$TokenizerMapper</value>
                </property>

                <!-- 指定Reduce类 -->
                <property>
                    <name>mapreduce.job.reduce.class</name>
                    <value>org.apache.hadoop.examples.WordCount$IntSumReducer</value>
                </property>

                <property>
                    <name>mapreduce.tasktracker.map.tasks.maximum</name>
                    <value>2</value>
                </property>
                <!-- 
                <property>
                    <name>mapred.map.tasks</name>
                    <value>1</value>
                </property>
                -->
            </configuration>
        </map-reduce>
        <ok to="end" />
        <error to="fail" />
    </action>
    <kill name="fail">
        <message>Map/Reduce failed, error
            message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end" />
</workflow-app>

使用的是官方jar包，里面可能有多个job, 逻辑是 wordcount /input/ /output/ -> String[] args -> if(args[0].equals("wordcount")){.....} ,oozie没有这样的逻辑，但是可以在oozie中只需指定map.class 和 reduce.class 它自己封装成一个job

5、拷贝待执行的jar包到map-reduce的lib目录下

[victor@hadoop102 oozie-4.0.0-cdh5.3.6]$ cp -a \
/opt/module/hadoop\
/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.6.jar \
oozie-apps/map-reduce/lib

6、上传配置好的app文件夹到HDFS

[victor@hadoop102 oozie-4.0.0-cdh5.3.6]$ /opt/modules/cdh/hadoop-2.5.0-cdh5.3.6\
/bin/hdfs dfs -put oozie-apps/map-reduce/ /user/victor/oozie-apps

7、执行任务

[victor@hadoop102 oozie-4.0.0-cdh5.3.6]$ bin/oozie job \
-oozie http://hadoop102:11000/oozie \
-config oozie-apps/map-reduce/job.properties \
-run

8、寻找workflow.xml配置的key和value

尤其不知道到底是哪个类时，建议都提前跑一边jar包，因为类有的时候会变，跑出来的才是正确的