Oozie 4.3.0 更新日志

说明

完整更新日志：release log
注意：更新日志里面只有 jira 编号和标题，详细信息需要到 jira 网站上依次查找。

Oozie 4.3.0 更新日志

1. OOZIE-2613 hive 版本从 0.13.1 升级到 1.2.0
2. OOZIE-2676 构建 oozie 时默认 profile 从 hadoop-1 变为 hadoop-2
3. OOZIE-1978 修复 ForkJoin 过多时，验证 Forkjoin 是否有效速度过慢的问题
   (引入了 新的bug（OOZIE-3345）：Multiple “ok to” transitions to the same node are not allowed)

     A1--->A2--->A3    A7--->A8--->A9 
     |           |     |            |
S--> F1          J1-->F2            J2--->E
     |           |     |            |
     A4--->A5--->A6    A10-->A11-->A12
例如，第一个递归选择路径 F1-> A1-> A2-> A3-> J1，选择 F2-> A7-> A8-> A9-> J2；然后返回检查 F2-> A10-> A11-> A12-> J2。
然后，返回到F1，选择第二条路径：F1-> A4-> A5-> A6。但是，它再次重新走了F2-> J2的两条路径。

例如，现在有10个ForkJoin对，每对有4个分支，则总共有4^10条路径。

4. OOZIE-2037 禁用SSL3，支持TLSv1.1，TLSv1.2
5. OOZIE-2552 ActiveMQ 版本从 5.8.0 升级至 5.10.2
6. OOZIE-2571 添加 spark.scala.binary.version 属性以支持Scala 2.11
7. OOZIE-2488 OpenJPA 版本从 2.2.2 升级到 2.4.1
8. OOZIE-2648 子作业不应该发送回调给 Oozie Server，而是由Launcher Job完成（MapReduce Action 除外）
9. OOZIE-2584 在 TestMemoryLocks 类中消除Thread.sleep() 调用，用CountDownLatch替换
10. OOZIE-2632 提供数据库导入导出功能，便于数据库迁移（oozie-setup.sh）
11. OOZIE-2243 修复 kill 只杀死 launcher job 而未杀死由它启动的子作业
12. OOZIE-2649 修复子工作流不能覆盖父工作流定义的配置属性问题
13. OOZIE-1173 null值校验应统一用参数检查形式

/**
 *
if (conf == null) {
	throw new IllegalArgumentException("conf cannot be null");
}
if (scriptFile == null) {
	throw new IllegalArgumentException("scriptFile cannot be null");
}
*/
OozieClient.notNull(conf, "conf");
OozieClient.notNull(scriptFile, "scriptFile");

14. OOZIE-2657 去除接口冗余修饰符

 public interface CoordInputDependency {
 
    // public static final String INTERNAL_VERSION_ID = "V=1";
    String INTERNAL_VERSION_ID = "V=1";
 
    // public void addInputInstanceList(String inputEventName, List<CoordInputInstance> inputInstanceList);
    void addInputInstanceList(String inputEventName, List<CoordInputInstance> inputInstanceList);

15. OOZIE-2517 coord 和 bundles 支持 startCreatedTime 和 endCreatedTime 过滤条件

curl "http://host:11000/oozie/v2/jobs?filter=startCreatedTime%3D2016-04-28T00%3A00Z&jobtype=bundle"

curl "http://host:11000/oozie/v2/jobs?filter=startCreatedTime%3D2016-04-28T00%3A00Z&jobtype=coordinator"

curl "http://host:11000/oozie/v2/jobs?filter=startCreatedTime%3D2016-04-28T00%3A00Z&jobtype=wf"

16. OOZIE-2036 构建 Oozie 时 JDK 版本最低要求为 1.7
17. OOZIE-2602 pig 版本从 0.12.1 升级至 0.16.0
18. OOZIE-2574 支持 Mysql replication 模式 (abhishekbafna via rkanter)

jdbc:mysql:replication://master_host:port,slave_host:port/database

19. OOZIE-2440 workflow 重试策略支持周期型（periodic ）与指数型（exponential）

全局配置参数（oozie.site.xml）：oozie.service.LiteWorkflowStoreService.user.retry.policy
工作流配置：<action name="a" retry-max="2" retry-interval="1" retry-policy=" exponential"> 

20. OOZIE-2507 通过 Oozie 中的JMX beans 暴露监控信息（Zabbix）

<property>
    <name>oozie.jmx_monitoring.enable</name>
    <value>false</value>
    <description>
        If the oozie functional metrics needs to be exposed via JMX interface, 
        set it to true.
    </description>
</property>

21. OOZIE-2603 给线程池（SchedulerService 与 CallableQueueService）中的线程赋予有意义的名字

// executor = new ThreadPoolExecutor(threads, threads, 10, TimeUnit.SECONDS, (BlockingQueue) queue);
executor = new ThreadPoolExecutor(threads, threads, 10, TimeUnit.SECONDS, (BlockingQueue) queue, new NamedThreadFactory("CallableQueue"));

public class NamedThreadFactory implements ThreadFactory {
    private final AtomicInteger counter = new AtomicInteger();
    private final String threadPrefix;

    public NamedThreadFactory(String threadPrefix) {
        this.threadPrefix = ParamChecker.notEmpty(threadPrefix, "threadPrefix");
    }

    @Override
    public Thread newThread(Runnable r) {
        Thread t = new Thread(r);
        t.setName(threadPrefix + "-" + counter.getAndIncrement());
        return t;
    }
}

22. OOZIE-2362 修复 BulkJPAExecutor 中存在的 SQL 注入问题
23. OOZIE-2567 修复 HCat 连接未关闭的问题
24. OOZIE-2541 修复未关闭 hive 连接可能导致的内存泄露问题
25. OOZIE-2551 增加转时间戳函数 coord:epochTime(String timestamp, String millis)

示例：timeStamp 为 2009-01-01T00:00Z
	millis 为 false, 返回 ‘1230768000’
	millis 为 ‘true’，返回 ‘1230768000000’

26. OOZIE-2542 添加选项以禁用 OpenJPA BrokerImpl finalization，减少GC时间

<property>
    <name>oozie.service.JPAService.openjpa.BrokerImpl</name>
    <value>non-finalizing</value>
    <description>
      	The default OpenJPAEntityManager implementation automatically closes 
        itself during instance finalization.
      	This guards against accidental resource leaks that may occur if a 
        developer fails to explicitly close EntityManagers when finished with 
        them, but it also incurs a scalability bottleneck, since the JVM must
      	perform synchronization during instance creation, and since the finalizer 
        thread will have more instances to monitor.
      	To avoid this overhead, set the openjpa.BrokerImpl configuration property
        to non-finalizing.
      	To use default implementation set it to empty space.
   </description>
</property>

27. OOZIE-2546 OozieDBCLI中在finally块中关闭资源
28. OOZIE-2476 修复一个fork 分支中的操作因短暂错误而失败（但在重试几次后成功），可能永远不会汇聚到join的问题
29. OOZIE-2475 修复杀死 action 后未删除 action 目录的问题
30. OOZIE-2467 修复因GC时间过长 Oozie 杀死自己的情况

<property>   
    <name>oozie.zookeeper.connection.timeout</name>
    <value>180</value>
    <description>
   		 Default ZK connection timeout (in sec).
    </description>
</property>
<property>
    <name>oozie.zookeeper.session.timeout</name>
    <value>300</value>
    <description>
        Default ZK session timeout (in sec). If connection is lost even 
        after retry, then Oozie server will shutdown
        itself if oozie.zookeeper.server.shutdown.ontimeout is true.
    </description>
</property>
<property>
    <name>oozie.zookeeper.max.retries</name>
    <value>10</value>
    <description>
        Maximum number of times to retry.
    </description>
</property>

31. OOZIE-2312 修复未清理 audit 与 error 日志的问题
32. OOZIE-2431 支持使用 hive.metastore.uris 作为 hcat.metastore.uri 的次要选项
33. OOZIE-1976 coordinator 输入 datasets 之前只支持 and 逻辑，现在支持 and、or、combine 逻辑

<!-- 若dataset A与B可用，或者dataset C与D可用，action则会启动；注意检查的顺序由xml中定义的顺序决定 -->
<input-logic>
    <or>
        <and name="AorB">
              <data-in dataset="A"/>
              <data-in dataset="B"/>
        </and>
        <and name="CorD">
              <data-in dataset="C"/>
              <data-in dataset="D"/>
        </and>
    </or>
</input-logic>

<!-- 首先从A中检查，缺少的instance将从B中补充 -->
<input-logic>
       <combine name="AorB">
            <data-in dataset="A"/>
            <data-in dataset="B"/>
       </combine>
</input-logic>

<!-- dataset A的instance数最少为5，达到5后会等待10分钟，期间新增的instance都会被加进来；或者dataset B的instance数最少为5，达到5后会等待10分钟，期间新增的instance都会被加进来-->
<input-logic>
    <or name="AorB" min="5" wait="10">
        <data-in dataset="A"/>
        <data-in dataset="B"/>
    </or>
</input-logic>

34. OOZIE-2444 在 bundle xml 中添加 enabled 标记，以支持是否启用某个 coordinator
35. OOZIE-2380 解决 hive action 中调用 “show tables" 或者 "show databases“ 报错的问题

22257 [uber-SubtaskRunner] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: IllegalArgumentException java.net.URISyntaxException: Relative path in absolute URI: file:./tmp/mapr/b79b1ef5-c8a4-4da9-a307-b4f9045c7bb9/hive_2015-10-01_11-30$
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:./tmp/mapr/b79b1ef5-c8a4-4da9-a307-b4f9045c7bb9/hive_2015-10-01_11-30-46_978_7012892802177357645-1
at org.apache.hadoop.fs.Path.initialize(Path.java:217)
at org.apache.hadoop.fs.Path.<init>(Path.java:208)
at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:284)
at org.apache.hadoop.hive.ql.Context.getLocalScratchDir(Context.java:319)
at org.apache.hadoop.hive.ql.Context.getLocalTmpPath(Context.java:414)
...      

36. OOZIE-2402 修复在大规模集群上通过 oozie-setup.sh 创建 sharelib 过慢的问题

sharelib create -fs FS_URI [-locallib SHARED_LIBRARY] [-concurrency CONCURRENCY]

private void concurrentCopyFromLocal(final FileSystem fs, int threadPoolSize,
        						File srcFile, final Path dstPath) throws IOException {
    List<Future<Void>> futures = Collections.emptyList();
    ExecutorService threadPool = Executors.newFixedThreadPool(threadPoolSize);
    try {
        futures = copyFolderRecursively(fs, threadPool, srcFile, dstPath);
        System.out.println("Running " + futures.size() + " copy tasks on " + threadPoolSize + " threads");
    } finally {
        try {
            threadPool.shutdown();
        } finally {
            checkCopyResults(futures);
        }
    }
}

37. OOZIE-2185 oozie cli 支持在 oozie-client-env.sh 文件中定义变量
38. OOZIE-2397 解决 LAST_ONLY 与 NONE 无法正确处理处于 READY 状态的 action 的问题
39. OOZIE-2168 将 action 名字长度限制从 50 个字符增加到128
40. OOZIE-2251 暴露 instrumental 监控信息给某些第三方图形化监控工具（graphite 或者 ganglia）

<property>
    <name>oozie.external_monitoring.enable</name>
    <value>false</value>
    <description>
        If the oozie functional metrics needs to be exposed to the metrics-server 
        backend, set it to true
        If set to true, the following properties has to be specified : 
        oozie.metrics.server.name, oozie.metrics.host, oozie.metrics.prefix, 	
        oozie.metrics.report.interval.sec, oozie.metrics.port
    </description>
</property>

<property>
    <name>oozie.external_monitoring.type</name>
    <value>graphite</value>
    <description>
        The name of the server to which we want to send the metrics, 
        would be graphite or ganglia.
    </description>
</property>

<property>
    <name>oozie.external_monitoring.address</name>
    <value>http://localhost:2020</value>
</property>

<property>
    <name>oozie.external_monitoring.metricPrefix</name>
    <value>oozie</value>
</property>

<property>
    <name>oozie.external_monitoring.reporterIntervalSecs</name>
    <value>60</value>
</property>

41. OOZIE-2377 修复 Hive2 Action 将 oozie.hive2.* properties 传递给 Beeline 的问题
42. OOZIE-2345 fork 中的 action 由串行提交改为并行提交
43. OOZIE-2356 增加属性决定是否跳过获取 credentials

<property>
    <name>oozie.credentials.skip</name>
    <value>false</value>
    <description>
        This determines if Oozie should skip getting credentials from the 
        credential providers.  
        This can be overwritten at a job-level or action-level.
    </description>
</property>

44. OOZIE-2347 移除不必要的对象创建（new Configuration()、new jobConf()）
45. OOZIE-2322 修复 Oozie Web UI 启用 Kerberos 后与 Internet Explorer 10/11 兼容问题及 curl 无法正常工作问题
46. OOZIE-2332 Hive 和 Hive 2 Action 支持在 workflow.xml 文件中内嵌查询（<query>标签替换<script>标签）
47. OOZIE-2325 修复当用户覆盖 oozie.launcher.mapreduce.map.env 变量时 Shell action 运行失败问题
48. OOZIE-2324 修复在 kill 节点发生语法错误时导致工作流卡住的问题
49. OOZIE-2308 增加函数 bundle:conf()
50. OOZIE-1837 修复因时钟偏移导致作业失败的问题

long endTime = System.currentTimeMillis();
if (startTime > endTime) {
    System.out.println("WARNING: Clock skew between the Oozie server host and this host detected.  Please fix this.  " +
            "Attempting to work around...");
    // We don't know which one is wrong (relative to the RM), so to be safe, let's assume they're both wrong and add an
    // offset in both directions
    long diff = 2 * (startTime - endTime);
    startTime = startTime - diff;
    endTime = endTime + diff;
}

51. OOZIE-2187 支持在 Oozie 配置文件中指定默认 JobTracker 和 NameNode

<property>
    <name>oozie.actions.default.name-node</name>
    <value> </value>
    <description>
        The default value to use for the <name-node> element in applicable action types. 
        This value will be used when neither the action itself nor the global section
        specifies a <name-node>.  
        As expected, it should be of the form "hdfs://HOST:PORT".
    </description>
</property>

<property>
    <name>oozie.actions.default.job-tracker</name>
    <value> </value>
    <description>
        The default value to use for the <job-tracker> element in applicable action
        types.  
        This value will be used when neither the action itself nor the global section 
        specifies a <job-tracker>.  
        As expected, it should be of the form "HOST:PORT".
    </description>
</property>

52. OOZIE-2159 将 ‘oozie validate’ 命令移至服务端，支持本地文件与 HDFS 文件

oozie validate hdfs://localhost:8020/user/test/myApp/bundle.xml
oozie validate /home/test/myApp/coordinator.xml

53. OOZIE-2271 Tomcat 版本从 6.0.43 升级到 6.0.44
54. OOZIE-2178 修复 Java 8 不允许 self-closing element 的问题