一:Oozie运行examples。
1、解压示例包
$ tar -zxf oozie-examples.tar.gz -C ./
2、$ bin/hdfs dfs -put /opt/cdh5/oozie-4.0.0-cdh5.3.6/examples/
3、编写job.properties
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
nameNode=hdfs://bigdata-01.yushu.com:8020
jobTracker=bigdata-01.yushu.com:8032
queueName=default
examplesRoot=examples
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce/workflow.xml
outputDir=map-reduce
${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce/workflow.xml 对应的路径
hdfs://bigdata-senior01.ibeifeng.com:8020/user/beifeng/examples/apps/map-reduce/workflow.xml
4、运行 (job.properties 运行的是linux本地的job文件 报错的话看 logs/oozie.log的日志文件查看)
bin/oozie job -oozie http://bigdata-01.yushu.com:11000/oozie -config examples/apps/map-reduce/job.properties -run
5、修改oozie-site.xml
<property>
<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
<value>*=/opt/cdh5/hadoop-2.5.0-cdh5.3.6/etc/hadoop/</value>
</property>
-》重启oozie bin/oozied.sh stop bin/oozied.sh start
6、oozie机制
默认一个workflow会先启动一个MR-》对WF进行封装,初始化工作
二:oozie自定义MapReduce WorkFlow
1、拷贝官方提供的example的示例
2、将Java代码打包 放在lib目录下
3、修改job.properties
#
nameNode=hdfs://bigdata-01.yushu.com:8020
jobTracker=bigdata-01.yushu.com:8032
queueName=default
examplesRoot=user/ds/application/apps/mr-wf
oozie.wf.application.path=${nameNode}/${examplesRoot}/workflow.xml
inputDir=input
outputDir=output-oozie
4、修改 workflow.xml
<workflow-app xmlns="uri:oozie:workflow:0.2" name="map-reduce-wf">
<start to="mr-node"/>
<action name="mr-node">
<map-reduce>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/${outputDir}"/>
</prepare>
<configuration>
<property>
<name>mapreduce.job.queuename</name>
<value>${queueName}</value>
</property>
<property>
<name>mapred.mapper.new-api</name>
<value>true</value>
</property>
<property>
<name>mapred.reducer.new-api</name>
<value>true</value>
</property>
<property>
<name>mapreduce.job.map.class</name>
<value>com.bigdata.mapreduce.WCMapReduce$WCMapper</value>
</property>
<property>
<name>mapreduce.job.reduce.class</name>
<value>com.bigdata.mapreduce.WCMapReduce$WCReducer</value>
</property>
<property>
<name>mapreduce.map.output.key.class</name>
<value>org.apache.hadoop.io.Text</value>
</property>
<property>
<name>mapreduce.map.output.value.class</name>
<value>org.apache.hadoop.io.IntWritable</value>
</property>
<property>
<name>mapreduce.job.output.key.class</name>
<value>org.apache.hadoop.io.Text</value>
</property>
<property>
<name>mapreduce.job.output.value.class</name>
<value>org.apache.hadoop.io.IntWritable</value>
</property>
<property>
<name>mapreduce.input.fileinputformat.inputdir</name>
<value>${nameNode}/${inputDir}</value>
</property>
<property>
<name>mapreduce.output.fileoutputformat.outputdir</name>
<value>${nameNode}/${outputDir}</value>
</property>
</configuration>
</map-reduce>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
5、放jar包
6、上传目录到HDFS
7、运行
export OOZIE_URL="http://bigdata-01.yushu.com:11000/oozie"
bin/oozie job -config apps/app/mrwf/job.properties -run
三、oozie自定义Shell WorkFlow
job.properties配置文件内容:
#
nameNode=hdfs://bigdata-01.yushu.com:8020
jobTracker=bigdata-01.yushu.com:8032
queueName=default
examplesRoot=user/ds/application/apps/shell-wf
oozie.wf.application.path=${nameNode}/${examplesRoot}/workflow.xml
EXEC=oozie.sh
workflow.xml文件内容:
<workflow-app xmlns="uri:oozie:workflow:0.4" name="shell-wf"> <start to="shell-node"/> <action name="shell-node"> <shell xmlns="uri:oozie:shell-action:0.2"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <exec>${EXEC}</exec> <file>${EXEC}#${EXEC}</file> <!--Copy the executable to compute node's current working directory --> </shell> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/> </workflow-app>
oozie.sh文件内容:
#!/bin/bash HIVE_HOME=/opt/cdh5/hive-0.13.1-cdh5.3.6 $HIVE_HOME/bin/hive -e "show databases" >>/opt/datas/oozie-hive.txt
最后执行命令:bin/oozie job -config application/apps/shell-wf/job.properties -run
四、oozie修改时区
1、时间调度
-》coordinate
-》基于时间的调度
-》基于数据可用性的调度
-》起始时间
-》结束时间
-》时间频率
2、时区
3、oozie支持从UTC和GMT
4、将oozie的默认时区指定为GMT
5、命令:date -R
-》如果命令没有输出+0800的显示
-》删除时区文件/etc/localtime
-》删除后重新生成,创建一个软连接ln -s
/usr/share/zoneinfo/Asia/shanghai
6、修改oozie时区
修改oozie-site.xml
<property>
<name>oozie.processing.timezone</name>
<value>GMT+0800</value>
</property>
-》重启oozie
-》修改oozie-server/webapps/oozie/oozie-console.js代码
function getTimeZone() {
Ext.state.Manager.setProvider(new Ext.state.CookieProvider());
return Ext.state.Manager.get("TimezoneId","GMT+0800");
}
五、oozie Coordinator讲解
-》修改oozie-site.xml
<property>
<name>oozie.service.coord.check.maximum.frequency</name>
<value>false</value>
</property>
* * * * *
job.properties文件内容:
nameNode=hdfs://bigdata-01.yushu.com:8020 jobTracker=bigdata-01.yushu.com:8032 queueName=default examplesRoot=user/ds/application/apps/cron-wf oozie.coord.application.path=${nameNode}/${examplesRoot}/coordinator.xml start=2017-06-26T19:50+0800 end=2017-06-26T19:58+0800 workflowAppUri=${nameNode}/${examplesRoot}/workflow.xml
coordinator.xml文件内容:
frequency="${coord:minutes(2)} 表示每隔两分钟执行
<coordinator-app name="cron-coord" frequency="${coord:minutes(2)}" start="${start}" end="${end}" timezone="GMT+0800" xmlns="uri:oozie:coordinator:0.2"> <action> <workflow> <app-path>${workflowAppUri}</app-path> <configuration> <property> <name>jobTracker</name> <value>${jobTracker}</value> </property> <property> <name>nameNode</name> <value>${nameNode}</value> </property> <property> <name>queueName</name> <value>${queueName}</value> </property> </configuration> </workflow> </action> </coordinator-app>
workflow.xml文件内容:
<workflow-app xmlns="uri:oozie:workflow:0.5" name="one-op-wf"> <start to="action1"/> <action name="action1"> <fs/> <ok to="end"/> <error to="end"/> </action> <end name="end"/> </workflow-app>
参考文献:http://blog.csdn.net/nsrainbow/article/details/43746111
参考文献2:https://www.zybuluo.com/tsing1226/note/263232
参考文献3:http://shiyanjun.cn/archives/684.html oozie Coordinator Job