oozie spark on yarn

废话不多说，现在开始正文，本文将阐述如何将一个Spark程序通过oozie提交到Hadoop的Yarn上运行。
coordinator.properties这个配置文件主要是一些oozie的配置，包括oozie地址 queue名称等等。
oozie.coord.application.path app_dir这里是HDFS路径，因为大家都知道，oozie运行之前要将 workflow整个目录上传到HDFS目录中。

Coordinator

oozie.coord.application.path=/user/mxb/workflows/mxbtest
app_dir=/user/mxb/workflows/mxbtest
oozie.use.system.libpath=true
oozie.user=mxb

Workflow

job_tracker=10.13.13.21:8032

job_tracker=mycluster
name_node=hdfs://mycluster
queue_name=offline
thrift_uri=thrift://10.50.12.11:9083
oozie_server=http://10.50.12.11:11000/oozie

ACL

acl_view_job=*
acl_modify_job=*

spark_localtion=/user/mxb/share/software/spark-1.6.0-bin-hadoop2.6.tar.gz

start_time=2017-03-20T20:56+0800
end_time=2020-05-10T01:00+0800

param

timezone=GMT+08:00
concurrency=3
parallelism=128
user_name=mxb
PID=3

//coordinator中配置运行频率，要运行的workflow
coordinator.xml

!/usr/bin/env bash

./spark-home/bin/spark-submit \
–conf spark.app.name=ViewCount \
–conf spark.rpc.askTimeout=300 \
–conf spark.task.maxFailures=100 \
–conf spark.yarn.executor.memoryOverhead=1024 \
–conf spark.yarn.max.executor.failures=200 \
–conf spark.memory.storageFraction=0.6 \
–class com.liebao.spark.ViewCount \
–master yarn \
–deploy-mode cluster \
–driver-memory 1g \
–executor-memory 1g \
–executor-cores 2 \
–num-executors 20 \
–queue offline \
news-2.0-shaded.jar “$@”
mxbtest目录结构，lib中是编写好的spark程序jar包以及start.sh

.
最终运行结果如下：