oozie 常用标签 -- coordinator.xml

统计前设置检查标志文件至关重要,要是不检查就有可能没有准备好,就统计了,这样的就好出现结果无法解释,重新统计就好了


<coordinator-app name="gwk-med-stat-coord" frequency="${coord:days(1)}" start="${dayStart}" end="${dayEnd}" timezone="Asia/Shanghai"

                 xmlns="uri:oozie:coordinator:0.2">
        <controls>
         <timeout>3600</timeout>   每次调度的超时时间,-1 永不超时,0不符合条件立即超时推出。
        <concurrency> 20</concurrency>  可以同时并行的 任务数。
        <!--execution>${execution_order}</execution>
        <throttle>${materialization_throttle}</throttle-->
    </controls>      
    <datasets>
        <dataset name=" dataset-cjf-impr" frequency="${coord:days(1)}" initial-instance="${dayDataStart}" timezone="Asia/Shanghai">

            <uri-template>${nameNode}/adx-data/log/cjfImprLog/yyyymmdd=${YEAR}${MONTH}${DAY}/</uri-template> 数据文件目录,默认检查目录是 _SUCCESS

              主要是用来检查数据文件有没有准备好了。

        </dataset>
        <dataset name="dataset-cjf-click" frequency="${coord:days(1)}" initial-instance="${dayDataStart}" timezone="Asia/Shanghai">
            <uri-template>${nameNode}/adx-data/log/cjfClickLog/yyyymmdd=${YEAR}${MONTH}${DAY}/</uri-template>
        </dataset>
        <dataset name="dataset-cjf-cps_med" frequency="${coord:days(1)}" initial-instance="${dayDataStart}" timezone="Asia/Shanghai">
            <uri-template>${nameNode}/user/hive/warehouse/yiqifa_db.db/redo_date/create_date=${YEAR}${MONTH}${DAY}/</uri-template>
        </dataset>
  </datasets>
     <input-events>
          <data-in name="in-cjf-impr" dataset="dataset-cjf-impr">      数据输入到计算程序时,检查指定数据目录上的标志文件,有没有好,不好就等待。
              <instance>${coord:current(-1)}</instance>
         </data-in>

         <data-in name="in-cjf-click" dataset="dataset-cjf-click">
              <instance>${coord:current(-1)}</instance>
         </data-in>
        <data-in name="in-cjf-cps_med" dataset="dataset-cjf-cps_med">
              <instance>${coord:current(-1)}</instance>
         </data-in>
     </input-events>    
    <action>
        <workflow>
            <app-path>${workflowDayGwkMedAppUri}</app-path>
            <configuration>
                <property>
                    <name>yyyymmdd</name>
                    <value>${coord:formatTime(coord:dateOffset(coord:nominalTime(),-1,'DAY'), 'yyyyMMdd')}</value>     设置动态参数
                </property>

                <property>
                    <name>yyyymmdd30</name>
                    <value>${coord:formatTime(coord:dateOffset(coord:nominalTime(),-15,'DAY'), 'yyyyMMdd')}</value>
                </property>
            </configuration>
        </workflow>
    </action>
</coordinator-app>
  • 0
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Stdoutput Warning: /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Stdoutput Please set $ACCUMULO_HOME to the root of your Accumulo installation. Exit code of the Shell command 1 <<< Invocation of Shell command completed <<< java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:410) at org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:55) at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:223) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217) at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141) Caused by: org.apache.oozie.action.hadoop.LauncherMainException at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:76) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:104) at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:63) ... 16 more Failing Oozie Launcher, Main Class [org.apache.oozie.action.hadoop.ShellMain], exit code [1] Oozie Launcher, uploading action data to HDFS sequence file: hdfs://nameservice0/user/admin/oozie-oozi/0000118-230724100647793-oozie-oozi-W/shell-af10--shell/action-data.seq Stopping AM Callback notification attempts left 0 Callback notification trying http://nn.hdfs.offline:11000/oozie/callback?id=0000118-230724100647793-oozie-oozi-W@shell-af10&status=FAILED Callback notification to http://nn.hdfs.offline:11000/oozie/callback?id=0000118-230724100647793-oozie-oozi-W@shell-af10&status=FAILED succeeded Callback notification succeeded
最新发布
07-25

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值