HIVE/OOZIE ERROR记录

问题1

报错日志：日志上的报错为
return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
遇见这个报错就表示需要进yarn看log了
yarn日志：yarn日志报错为
Output column number expected to be 0 when isRepeating

查询hive的官方jira，有对此报错的临时解决方案：

set hive.vectorized.execution.enabled = false;

注意，设置会导致hive有一定的性能下降；

问题2

java oozie API创建coordinate调度任务时失败，报错内容如下：（此问题未解决）

ERROR org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet](175) - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is E0307 : E0307: Runtime error [App directory [null] does not exist and app definition cannot be created because of missing config value [oozie.jobs.api.generated.xml]]] with root cause
org.apache.oozie.client.OozieClientException: E0307: Runtime error [App directory [null] does not exist and app definition cannot be created because of missing config value [oozie.jobs.api.generated.xml]]
        at org.apache.oozie.client.OozieClient.handleError(OozieClient.java:618)
        at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:719)
        at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:672)
        at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:565)
        at org.apache.oozie.client.OozieClient.submit(OozieClient.java:734)
        at com.dingxinwen.bi.bigdata.service.impl.OozieServiceImpl.testCoordinator(OozieServiceImpl.java:681)
        at com.dingxinwen.bi.bigdata.controller.OozieController.testCoordinator(OozieController.java:105)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:791)

外网查询到的资料Issue Navigator - ASF JIRA

The exception is created when trying to create the workflow.xml. At this point no workflow.xml exist on the defined path and no generated value is defined for the workflow.xml to be created, so int the uploaded patch I only extended the error message to the following one:

Error: E0307 : E0307: Runtime error [hdfs://localhost:9000/user/test/examples10/apps/java-main/workflow.xml does not exist and it cannot be created because of missing config value oozie.jobs.api.generated.xml]

大致意思是说，再提交coordinate定时任务前并没有创建工作流。我仔细检查了下代码好像确实是这样。

public String testCoordinator() {
        log.info("OOZIE: Test Submit Coordinator Start");

        FileSystem hdfsFs = hadoopHDFSService.getFileSystem();
        OozieClient oozie = new OozieClient(oozieConfig.getUrl());

        Path homeDirectory = hdfsFs.getHomeDirectory();
        Path appPath = new Path(hdfsFs.getHomeDirectory(), "testApp");
        hdfsFs.mkdirs(new Path(appPath, "lib"));
        Path workflow = new Path(appPath, "workflow.xml");
        Path coordinator = new Path(appPath, "coordinator.xml");

        //write workflow.xml
        String wfApp =
                "<workflow-app xmlns='uri:oozie:workflow:0.1' name='test-wf'>" +
                        "    <start to='end'/>" +
                        "    <action name='my-shell-action'>\n" +
                        "       <shell xmlns='uri:oozie:shell-action:0.1'>\n" +
                        "           <job-tracker>${jobTracker}</job-tracker>\n" +
                        "           <name-node>${nameNode}</name-node>\n" +
                        "           <exec>/warehouse/dd/oozie/workspace/workspace-test_coorder/shell/test_coorder-shell.sh</exec>\n" +
                        "           <file>/warehouse/dd/oozie/workspace/workspace-test_coorder/shell/test_coorder-shell.sh#test_coorder-shell.sh</file>\n" +
                        "       </shell>" +
                        "    <ok to='end'/>\n" +
                        "    <error to='fail'/>\n" +
                        "    </action>\n" +
                        "    <kill name='fail'>\n" +
                        "        <message>执行脚本失败</message>\n" +
                        "    </kill>\n" +
                        "    <end name='end'/>" +
                        "</workflow-app>";

        String coordApp =
                "<coordinator-app timezone='UTC' end='2016-07-26T02:26Z' start='2016-07-26T01:26Z' frequency='${coord:hours(1)}' name='test-coordinator' xmlns='uri:oozie:coordinator:0.4'>" +
                        "    <action>" +
                        "        <workflow>" +
                        "            <app-path>" + workflow.toString() + "</app-path>" +
                        "        </workflow>" +
                        "    </action>" +
                        "</coordinator-app>";

        Writer writer = new OutputStreamWriter(hdfsFs.create(workflow));
        writer.write(wfApp);
        writer.close();

        Writer coordWriter = new OutputStreamWriter(hdfsFs.create(coordinator));
        coordWriter.write(coordApp);
        coordWriter.close();

        //write job.properties
        Properties conf = oozie.createConfiguration();
        generComoonProperties(homeDirectory, conf, TaskTypeEnum.COORDINATOR.name());
        conf.setProperty(OozieClient.COORDINATOR_APP_PATH, coordinator.toString());
        conf.setProperty(OozieClient.USER_NAME, UserGroupInformation.getCurrentUser().getUserName());

        //submit and check
        final String jobId = oozie.submit(conf);
        CoordinatorJob coord  = oozie.getCoordJobInfo(jobId);
        assertNotNull(coord);
        log.info("OOZIE: Coordinator: {}", coord.toString());

        return coord.toString();
    }

得到的解决方案是PreCommit-OOZIE-Build started，但是怎么做呢？

待研究