Spark2与Oozie整合

背景:

纯粹个人瞎摸索,相关资料太少了,估计大多数公司还没使用spark2吧?

1.在oozie共享目录创建spark2目录

可直接在页面上创建

2.将集群中spark2的jar复制到spark2目录下

hadoop fs -put / /opt/cloudera/parcels/SPARK2- 2.2 . 0 .cloudera2- 1 .cdh5. 12.0 .p0. 232957 /lib/spark2/jars /user/oozie/share/lib/lib_20180523180130/spark2 

3.将原oozie共享目录中spark的两个与oozie share相关的jar也拷贝到spark2目录下

hadoop fs -cp /user/oozie/share/lib/lib_20180523180130/spark/oozie-sharelib-spark- 4.1 . 0 -cdh5. 14.0 .jar  /user/oozie/share/lib/lib_20180523180130/spark2 

hadoop fs -cp /user/oozie/share/lib/lib_20180523180130/spark/oozie-sharelib-spark.jar /user/oozie/share/lib/lib_20180523180130/spark2

其目录结构如下:

4.创建oozie的spark工作流

进入oozie计划工作页面

上图中的Jar/py名称框里 只需选中spark-examples_2.11-2.2.0.cloudera2.jar 即可

选中/user/hue/oozie/workspaces/hue-oozie-1527557010.76/lib/spark-examples_2.11-2.2.0.cloudera2.jar 点击选择此文件夹

选完之后自己检查一下看看是不是全选了

图中的例子是官方的例子

hadoop fs -put /opt/cloudera/parcels/SPARK2/lib/spark2/examples/jars/spark-examples_2.11-2.2.0.cloudera2.jar /user/hue/oozie/workspaces/hue-oozie-1527557010.76/lib

Main class :org.apache.spark.examples.SparkPi

点击上图的齿轮设置属性

重点来了:

点击上图中的 齿轮

设置spark2 : name=oozie.action.sharelib.for.spark 

value=spark2

填写完之后保存提交即可

注意:应该注意一些权限问题

这些可能会用上

sudo -u hdfs hadoop fs -chown -R oozie:oozie /user/oozie/share/lib/lib_20180528183927/spark2

sudo -u hdfs hadoop fs -chown -R hue:hue /user/hue/oozie/workspaces

sudo -u hdfs hadoop fs -chmod 777 /user/hue/oozie/workspaces

参考:https://blog.csdn.net/qq_24908345/article/details/80017660

https://cloud.tencent.com/developer/article/1078092