简单上手版使用oozie提交spark程序(不带定时)

1. 做这个实验之前你的服务器上最好装了cdh集群，以及添加必要的组件，如hadoop，oozie，spark，yarn等。必须都启动成功。版本如下：
   hadoop 3.0.0-cdh6.3.2
   zookeeper 3.4.5-cdh6.3.2
   oozie 5.1.0-cdh6.3.2
   spark 2.4.0-cdh6.3.2
2. 需要准备一个关于spark的demo架包、job.propertie、workflow.xml。创建一个文件夹将三个放入其中，然后将文件夹put到HDFS指定的路径下面。(*注意每次在linux下修改完后，记得put到HDFS中 *)
3. 准备一个job.propertie和workflow.xml，内容如下
   3.1job.propertie

#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#HDFS文件系统的端口
nameNode=hdfs://emg101.emapgo.com:8020 

#resourceManager的端口,也有人写jobTracker
resourceManager=emg104.emapgo.com:8032 

#需要有，是否使用系统lib路径
oozie.use.system.libpath=true 

#需要有，workflow.xml在hdfs上的路径   
oozie.wf.application.path=${nameNode}/user/${user.name}/examples/apps/spark/workflow.xml

#workflow.xml中使用
queueName=default

#下面是spark的属性参数，具体参照自己的workflow.xml
jobmaster=yarn-cluster
jobmode=cluster
jobname=oozie_missing
jarclass=com.emg.etp.analysis.missing.MissingRoad
#jar包所在HDFS的绝对路径
jarpath=${nameNode}/user/root/examples/apps/spark/missing_road.jar
#运行spark任务的参数，有的可能需要assembly相关的jar包，我的不需要
sparkopts=--driver-memory 1G --executor-memory 2G --executor-cores 2 --num-executors 20
#jar包所需要的参数
input1=a.txt

3.2workflow.xml

<!--
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<workflow-app xmlns="uri:oozie:workflow:1.0" name="spark-wf">
    <start to="spark-node"/>
    <action name="spark-node">
        <spark xmlns="uri:oozie:spark-action:1.0">
            <resource-manager>${resourceManager}</resource-manager>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>queueName</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <master>${jobmaster}</master>
            <mode>${jobmode}</mode>
            <jar>${jarpath}</jar>
            <spark-opts>${sparkopts}</spark-opts>
            <arg>${input1}</arg>
        </spark>
        <!--<ok to="check-output"/>-->
        <ok to="end"/>
        <error to="fail"/>
    </action>

    <kill name="fail">
        <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <kill name="fail-output">
        <message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

启动命令(注意 这里的job.properties是linux下的路径)
bin/oozie job -oozie http://xxxx:11000/oozie -config oozie-apps/job.properties -run

kill命令，jobid可以到oozie自带的web界面查看
bin/oozie job -oozie http://xxxx:11000/oozie -kill 0000000-170724202624644-oozie-oozi-C

8088集群上看所运行的任务，可以看到先是启动Oozie Launcher，然后再调的spark任务。（oozieUI报错打不开，只能看这里了）


参考：
https://blog.csdn.net/Abysscarry/article/details/81784179
https://blog.csdn.net/pt707621764/article/details/52437567