- 目标:Coordinator周期性调度任务
- 检查系统当前时区:
命令:date -R
注意:如果显示的时区不是+0800,删除localtime文件夹后,再关联一个正确时区的链接过去
注意:检测ntp是否安装,命令:rpm -qa | grep ntp,使用 root 用户制定计划任务,周期性同步集群时间
- 修改/usr/local/module/oozie-4.0.0-cdh5.3.6/conf/ oozie-site.xml 文件
- 修改 js 框架中的关于时间设置的代码
- 重启 oozie 服务,并重启浏览器(一定要注意清除缓存)
- 效果
- 拷贝官方模板配置定时任务
- 修改coordinator.xml 文件
<!-- 修改频率为5分钟,这是最小值:frequency="${coord:minutes(5) timezone="GMT+0800" --> <coordinator-app name="cron-coord" frequency="${coord:minutes(5)}" start="${start}" end="${end}" timezone="GMT+0800" xmlns="uri:oozie:coordinator:0.2"> <action> <workflow> <app-path>${workflowAppUri}</app-path> <configuration> <property> <name>jobTracker</name> <value>${jobTracker}</value> </property> <property> <name>nameNode</name> <value>${nameNode}</value> </property> <property> <name>queueName</name> <value>${queueName}</value> </property> </configuration> </workflow> </action> </coordinator-app>
- 修改workflow.xml 文件
<workflow-app xmlns="uri:oozie:workflow:0.4" name="shell-wf"> <start to="p1-shell-node"/> <action name="p1-shell-node"> <shell xmlns="uri:oozie:shell-action:0.2"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <!--要执行的脚本--> <exec>p1.sh</exec> <!--脚本路径,注意:这个是HDFS上路径--> <file>/user/kgf/oozie-apps/cron/p1.sh</file> <capture-output/> </shell> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <kill name="fail-output"> <message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message> </kill> <end name="end"/> </workflow-app>
编辑p1.sh脚本
- 编辑job.properties文件
nameNode=hdfs://hadoop111:8020 jobTracker=hadoop112:8032 queueName=default examplesRoot=oozie-apps oozie.coord.application.path=${nameNode}/user/${user.name}/${examplesRoot}/cron #注意:start必须设置为未来时间,否则任务失败 start=2019-11-10T13:28+0800 end=2019-11-10T13:33+0800 workflowAppUri=${nameNode}/user/${user.name}/${examplesRoot}/cron
- 提交任务
- 效果
a:出错
2019-11-10 12:15:00,579 INFO ActionStartXCommand:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@:start:] Start action [0000001-191110121143517-oozie-kgf-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2019-11-10 12:15:00,579 INFO ActionStartXCommand:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@:start:] [***0000001-191110121143517-oozie-kgf-W@:start:***]Action status=DONE 2019-11-10 12:15:00,580 INFO ActionStartXCommand:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@:start:] [***0000001-191110121143517-oozie-kgf-W@:start:***]Action updated in DB! 2019-11-10 12:15:02,358 INFO ActionStartXCommand:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] Start action [0000001-191110121143517-oozie-kgf-W@p1-shell-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2019-11-10 12:15:04,840 INFO ShellActionExecutor:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] checking action, hadoop job ID [job_1573438272511_0003] status [RUNNING] 2019-11-10 12:15:04,841 INFO ActionStartXCommand:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] [***0000001-191110121143517-oozie-kgf-W@p1-shell-node***]Action status=RUNNING 2019-11-10 12:15:04,842 INFO ActionStartXCommand:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] [***0000001-191110121143517-oozie-kgf-W@p1-shell-node***]Action updated in DB! 2019-11-10 12:15:31,915 INFO CallbackServlet:541 - SERVER[hadoop111] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] callback for action [0000001-191110121143517-oozie-kgf-W@p1-shell-node] 2019-11-10 12:15:32,063 WARN ShellActionExecutor:544 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] Exception in check(). Message[JA017: Unknown hadoop job [job_1573438272511_0003] associated with action [0000001-191110121143517-oozie-kgf-W@p1-shell-node]. Failing this action!] org.apache.oozie.action.ActionExecutorException: JA017: Unknown hadoop job [job_1573438272511_0003] associated with action [0000001-191110121143517-oozie-kgf-W@p1-shell-node]. Failing this action! at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1210) at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:181) at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:55) at org.apache.oozie.command.XCommand.call(XCommand.java:281) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-11-10 12:15:32,066 WARN ActionCheckXCommand:544 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] Exception while executing check(). Error Code [JA017], Message[JA017: Unknown hadoop job [job_1573438272511_0003] associated with action [0000001-191110121143517-oozie-kgf-W@p1-shell-node]. Failing this action!] org.apache.oozie.action.ActionExecutorException: JA017: Unknown hadoop job [job_1573438272511_0003] associated with action [0000001-191110121143517-oozie-kgf-W@p1-shell-node]. Failing this action! at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1210) at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:181) at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:55) at org.apache.oozie.command.XCommand.call(XCommand.java:281) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-11-10 12:15:32,067 WARN ActionCheckXCommand:544 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] Failing Job due to failed action [p1-shell-node] 2019-11-10 12:15:32,068 WARN LiteWorkflowInstance:544 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[0000001-191110121143517-oozie-kgf-W@p1-shell-node] Workflow Failed. Failing node [p1-shell-node] 2019-11-10 12:15:33,740 INFO KillXCommand:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[-] STARTED WorkflowKillXCommand for jobId=0000001-191110121143517-oozie-kgf-W 2019-11-10 12:15:35,218 INFO KillXCommand:541 - SERVER[hadoop111] USER[kgf] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-191110121143517-oozie-kgf-W] ACTION[-] ENDED WorkflowKillXCommand for jobId=0000001-191110121143517-oozie-kgf-W
b:解决方案:
⑴在配置历史服务器时,/usr/local/module/hadoop-cdh5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop/mapred-site.xml要有
如下配置
⑵在/usr/local/module/oozie-4.0.0-cdh5.3.6/conf/oozie-site.xml要如下配置
-
最后效果
Oozie定时任务/循环任务
最新推荐文章于 2023-12-06 20:26:18 发布