oozie hive-action，MapReduce-action示例

1、对MapReduce的action的示例

我们的环境所有都是基于cdh5.3.6搭建的，具体的oozie的安装及配置请参考：

oozie配置：https://www.cnblogs.com/TiestoRay/p/6077865.html

这里有一个要注意的地方，在我们写自己的wordcount的workflow的时候，我们应该可以看到，xml文件里的api是旧的，而我们的运行的MapReduce却是新的api,所以我们要把官房给的模板做一下改动，然后当成自己的模板，对于要修改的参数，要根据自己的功能而定，自己进行参数的设定，但都是要新的api，下面是wordcount的自己的workflow模板

<workflow-app xmlns="uri:oozie:workflow:0.5" name="mr-wordcount-wf">
    <start to="mr-node-wordcount"/>
    <action name="mr-node-wordcount">
        <map-reduce>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/${oozieAppsData}/${outputDir}"/>
            </prepare>
            <configuration>
				<property>
                    <name>mapred.mapper.new-api</name>
                    <value>true</value>
                </property>
				<property>
                    <name>mapred.reducer.new-api</name>
                    <value>true</value>
                </property>
				<property>
                    <name>mapreduce.job.queuename</name>
                    <value>${queueName}</value>
                </property>
                <property>
                    <name>mapreduce.job.map.class</name>
                    <value>make.hadoop.com.mapreduce.Mymapreduce$MyMapper</value>
                </property>
                <property>
                    <name>mapreduce.job.reduce.class</name>
                    <value>make.hadoop.com.mapreduce.Mymapreduce$MyReduce</value>
                </property>
				<property>
                    <name>mapreduce.map.output.key.class</name>
                    <value>org.apache.hadoop.io.Text</value>
                </property><property>
                    <name>mapreduce.map.output.value.class</name>
                    <value>org.apache.hadoop.io.IntWritable</value>
                </property><property>
                    <name>mapreduce.job.output.key.class</name>
                    <value>org.apache.hadoop.io.Text</value>
                </property><property>
                    <name>mapreduce.job.output.value.class</name>
                    <value>org.apache.hadoop.io.IntWritable</value>
                </property>
                <property>
                    <name>mapreduce.input.fileinputformat.inputdir</name>
                    <value>/${oozieAppsData}/${inputDir}</value>
                </property>
                <property>
                    <name>mapreduce.output.fileoutputformat.outputdir</name>
                    <value>/${oozieAppsData}/${outputDir}</value>
                </property>
            </configuration>
        </map-reduce>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

job.properties

nameNode=hdfs://make.hadoop.com:8020
jobTracker=make.hadoop.com:8032
queueName=default
oozieAppsRoot=user/make/oozie-apps         总目录
oozieAppsData=user/make/oozie-apps/data     数据目录

oozie.wf.application.path=${nameNode}/${oozieAppsRoot}/mr-wordcount-wf/workflow.xml
inputDir=mr-wordcount-wf/input         输入目录
outputDir=mr-wordcount-wf/output       输出目录

对于各个目录的设置，要给定一个格式，以后的测试action，都按照这个样的格式进行存放和结果输出

这里本地的各个文件，最好确定没有问题了，然后全部put到hdfs的总目录下，这里用的是我们hadoop阶段自己编写的wordcount程序，自行上传

这是我们UI界面的显示

可以看到，也失败了很多次，其实都是配置出了问题，要么单词写错等等，所以在运行前，一定要仔细核对自己的配置到底是不是正确的！！！，不然就跟我一样，如果出错了，去logs/oozie.log查看日志，自行改错

这里就是我们运行成功的结果mapreduce-result

2、hive的action的一些简单配置

这个需要把hive-site.xml文件放到工作目录，也就是你workflow所在的目录，以及lib目录需要一个mysql的连接的jar包，这样大概最基本的就有了，下面使我们配置文件

workflow 

<workflow-app xmlns="uri:oozie:workflow:0.5" name="hive-my-wf">
    <start to="mr-node-hive-my"/>

    <action name="mr-node-hive-my">
        <hive xmlns="uri:oozie:hive-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/${oozieAppsData}/${outputDir}"/>
            </prepare>
			<job-xml>${nameNode}/${oozieAppsRoot}/hive-my-wf/hive-site.xml</job-xml>     这个job=xml的位置在configuration前面，里面是hive配置文件的绝对路径
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <script>emp.sql</script>
            <param>OUTPUT=${nameNode}/${oozieAppsData}/${outputDir}</param>
        </hive>
        <ok to="end"/>
        <error to="fail"/>
    </action>

    <kill name="fail">
        <message>Hive failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

job.properties

nameNode=hdfs://make.hadoop.com:8020
jobTracker=make.hadoop.com:8032
queueName=default
oozieAppsRoot=user/make/oozie-apps
oozieAppsData=user/make/oozie-apps/data
oozie.use.system.libpath=true

oozie.wf.application.path=${nameNode}/${oozieAppsRoot}/hive-my-wf/workflow.xml
outputDir=hive-my-wf/output

emp.sql   这个就是我们这个的具体任务

insert overwrite directory 
'${OUTPUT}'     --这个为我们在workflow.xml里面的配置好的
select count(1) from default.emp;

然后我们执行这个job    bin/oozie job -config   xxx/xxx/job.properties    -oozie  可以通过环境变量导入

export OOZIE_URL=http://make.hadoop.com:11000/oozie/

最后到我们的输出目录，查看我们的结果，存在的

hive-result

3、sqoop-action示例

workflow.xml

<?xml version="1.0" encoding="UTF-8"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<workflow-app xmlns="uri:oozie:workflow:0.5" name="sqoop-export-wf">
    <start to="sqoop-node-wf"/>

    <action name="sqoop-node-wf">
        <sqoop xmlns="uri:oozie:sqoop-action:0.3">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/${oozieAppsData}/${outputDir}"/>
            </prepare>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <command>import --options-file ${EXEC}</command>
			<file>${nameNode}/${oozieAppsRoot}/sqoop-export-wf/${EXEC}#${EXEC}</file> 
        </sqoop>
        <ok to="end"/>
        <error to="fail"/>
    </action>

    <kill name="fail">
        <message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

这里注意，file这个一定要写，不然不报错，所unable read xxx.sql  不能找到所指定的文件

job.properties

nameNode=hdfs://make.hadoop.com:8020
jobTracker=make.hadoop.com:8032
queueName=default
oozieAppsRoot=user/make/oozie-apps
oozieAppsData=user/make/oozie-apps/data
oozie.use.system.libpath=true

oozie.wf.application.path=${nameNode}/${oozieAppsRoot}/sqoop-export-wf/workflow.xml
outputDir=sqoop-export-wf/output
EXEC=sqoop-export.sql

具体的sql的内容自己编写进行测试，这样就可以成功了，我也是改了很多次的，哭瞎