工作流开源系统--OOZIE之安装spark2

最新推荐文章于 2024-05-08 09:38:31 发布
最新推荐文章于 2024-05-08 09:38:31 发布
阅读量1.4k 收藏 1
点赞数 1
分类专栏： bigdata workflow 文章标签： ooxie
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq1226317595/article/details/80816524
版权

引言

使用Hue可以方便的通过界面制定Oozie的工作流，支持Hive、Pig、Spark、Java、Sqoop、MapReduce、Shell等等。Spark？当让可以，但是自带是spark1的，那能不能支持Spark2的呢？接下来本文章就主要讲述如何使用Hue创建Spark1和Spark2的Oozie工作流。

安装spark2-lib到oozie

环境说明
spark2 ：
CDH内
jar包在：111节点的 /opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/jars
oozie：
CDH内，
主地址：192.168.1.100:11000/oozie
lib目录是：HDFS:/user/oozie/share/lib/lib_20180606144037/
操作系统：centos
步骤说明
1>添加lib到oozie-lib下

#先进入一个能够管理员的账号，这里使用import账号（为什么叫这个账号，不重要）
#建立本地jars目录
mkdir sparkyyh
#拷贝spark
cp /opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/jars/* sparkyyh
#把oozie的相关jar放进sparkyyh，否则，会出现 oozie-spark的MainClass找不到
cp /opt/cloudera/parcels/CDH/lib/oozie/oozie-sharelib-yarn/lib/spark/oozie-sharelib-spark*.jar sparkyyh
#传到hgfs的lib上（现在知道为啥import了吧，因为这是一个管理员账户，可以sudo）
sudo -u hdfs hadoop fs -mkdir /user/oozie/share/lib/lib_20180606144037/sparkyyh
sudo -u hdfs hadoop fs -put ./sparkyyh/*.jar /user/oozie/share/lib/lib_20180606144037/sparkyyh
#记得最后要保持一致
sudo -u hdfs hadoop fs -chown -R oozie:oozie /user/oozie/share/lib/
sudo -u hdfs hadoop fs –chmod -R 775 /user/oozie/share/lib/lib_20180606144037/sparkyyh

效果图如下：
这里写图片描述

[import@slave113 yyh]$ oozie admin -oozie http://192.168.1.100:11000/oozie -shareliblist spark2
[Available ShareLib]
spark2
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/JavaEWAH-0.3.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/RoaringBitmap-0.5.11.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/ST4-4.0.4.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/antlr-2.7.7.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/antlr-runtime-3.4.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/antlr4-runtime-4.5.3.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/aopalliance-repackaged-2.4.0-b34.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/apache-log4j-extras-1.2.17.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/arpack_combined_all-0.1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/avro-1.7.6-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/avro-ipc-1.7.6-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/avro-mapred-1.7.6-cdh5.7.0-hadoop2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/bonecp-0.7.1.RELEASE.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/breeze-macros_2.11-0.12.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/breeze_2.11-0.12.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/calcite-avatica-1.2.0-incubating.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/calcite-core-1.2.0-incubating.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/calcite-linq4j-1.2.0-incubating.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/chill-java-0.8.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/chill_2.11-0.8.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-cli-1.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-codec-1.10.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-collections-3.2.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-compiler-3.0.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-compress-1.4.1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-crypto-1.0.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-dbcp-1.4.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-httpclient-3.1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-io-2.4.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-lang-2.6.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-lang3-3.5.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-logging-1.1.3.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-math3-3.4.1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-net-2.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-pool-1.5.4.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/compress-lzf-1.0.3.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/core-1.1.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/datanucleus-api-jdo-3.2.6.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/datanucleus-core-3.2.10.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/datanucleus-rdbms-3.2.9.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/derby-10.11.1.1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/eigenbase-properties-1.1.5.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/fastutil-6.3.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/flume-ng-configuration-1.6.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/flume-ng-core-1.6.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/flume-ng-sdk-1.6.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/gson-2.2.4.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hive-metastore-1.1.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hive-serde-1.1.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hive-shims-0.23-1.1.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hive-shims-1.1.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hive-shims-common-1.1.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hive-shims-scheduler-1.1.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hk2-api-2.4.0-b34.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hk2-locator-2.4.0-b34.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hk2-utils-2.4.0-b34.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/httpcore-4.4.4.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/ivy-2.4.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jackson-annotations-2.6.5.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jackson-core-2.6.5.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jackson-databind-2.6.5.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jackson-jaxrs-1.8.8.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jackson-module-paranamer-2.6.5.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jackson-module-scala_2.11-2.6.5.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jackson-xc-1.8.8.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/janino-3.0.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/javassist-3.18.1-GA.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/javax.annotation-api-1.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/javax.inject-2.4.0-b34.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/javax.servlet-api-3.1.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/javax.ws.rs-api-2.0.1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/javolution-5.5.1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jcl-over-slf4j-1.7.5.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jdo-api-3.0.1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jersey-client-2.22.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jersey-common-2.22.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jersey-container-servlet-2.22.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jersey-container-servlet-core-2.22.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jersey-guava-2.22.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jersey-media-jaxb-2.22.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jersey-server-2.22.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jetty-6.1.26.cloudera.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jetty-util-6.1.26.cloudera.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jline-2.12.1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/joda-time-2.9.3.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jodd-core-3.5.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/json-20090211.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/json4s-ast_2.11-3.2.11.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/json4s-core_2.11-3.2.11.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/json4s-jackson_2.11-3.2.11.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jta-1.1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jtransforms-2.4.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jul-to-slf4j-1.7.5.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/kryo-shaded-3.0.3.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/libfb303-0.9.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/libthrift-0.9.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/logredactor-1.0.3.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/lz4-1.3.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/metrics-core-3.1.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/metrics-graphite-3.1.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/metrics-json-3.1.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/metrics-jvm-3.1.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/mina-core-2.0.0-M5.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/minlog-1.3.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/netty-3.8.0.Final.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/netty-all-4.0.42.Final.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/objenesis-2.1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/opencsv-2.3.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/oro-2.0.8.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/osgi-resource-locator-1.0.1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/paranamer-2.3.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/parquet-avro-1.5.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/parquet-column-1.5.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/parquet-common-1.5.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/parquet-encoding-1.5.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/parquet-format-2.1.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/parquet-hadoop-1.5.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/parquet-hadoop-bundle-1.5.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/parquet-jackson-1.5.0-cdh5.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/pmml-model-1.2.15.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/pmml-schema-1.2.15.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/py4j-0.10.4.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/pyrolite-4.13.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/scala-compiler-2.11.8.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/scala-library-2.11.8.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/scala-parser-combinators_2.11-1.0.4.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/scala-reflect-2.11.8.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/scala-xml_2.11-1.0.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/scalap-2.11.8.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/shapeless_2.11-2.0.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/snappy-0.2.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-catalyst_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-core_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-graphx_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-hive-exec_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-hive_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-launcher_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-mllib-local_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-mllib_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-network-common_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-network-shuffle_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-repl_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-sketch_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-sql_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-streaming-flume-sink_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-streaming-flume_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-streaming_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-tags_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-unsafe_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-yarn_2.11-2.1.0.cloudera1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spire-macros_2.11-0.7.4.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spire_2.11-0.7.4.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/stax-api-1.0.1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/stream-2.7.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/stringtemplate-3.2.1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/univocity-parsers-2.2.1.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/validation-api-1.1.0.Final.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/xbean-asm5-shaded-4.4.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/xz-1.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/oozie-sharelib-spark-4.1.0-cdh5.9.0.jar
    hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/oozie-sharelib-spark.jar

2>添加此lib到oozie系统中（更新Oozie的share-lib）

#添加
oozie admin -oozie http://192.168.1.100:11000/oozie -sharelibupdate
#确认
oozie admin -oozie http://192.168.1.100:11000/oozie -shareliblist

应当返回
这里写图片描述

测试oozie中的spark2–local

这里写图片描述
从下一张图可以看出，要添加sparkyyh属性
这里写图片描述
通过以下的参数可以看出，sparkyyh已经成功添加
这里写图片描述
这是8088运行
这里写图片描述
运行结果如下，是分析一个hdfs上文件的WC（代码多low，不重要，因为要的是能操作hdfs）
这里写图片描述

测试oozie中的spark2–cluster(代码中无local)

首先这个过程是踩了俩天的坑，才踩出来的
直接提交client或者cluster，将会报错

2018-06-27 09:10:18,710 WARN org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[manager100.weiboyi.com] USER[hive] GROUP[-] TOKEN[] APP[yyhTestWorkflow-spark] JOB[0000402-180621160350771-oozie-oozi-W] ACTION[0000402-180621160350771-oozie-oozi-W@spark-5908] Launcher exception: Library directory '/data3/yarn/nm/usercache/hive/appcache/application_1528447799658_20866/container_1528447799658_20866_01_000070/./assembly/target/scala-2.11/jars' does not exist; make sure Spark is built.
java.lang.IllegalStateException: Library directory '/data3/yarn/nm/usercache/hive/appcache/application_1528447799658_20866/container_1528447799658_20866_01_000070/./assembly/target/scala-2.11/jars' does not exist; make sure Spark is built.
    at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
    at org.apache.spark.launcher.CommandBuilderUtils.findJarsDir(CommandBuilderUtils.java:347)
    at org.apache.spark.launcher.YarnCommandBuilderUtils$.findJarsDir(YarnCommandBuilderUtils.scala:38)
    at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:526)
    at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:814)
    at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:169)
    at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
    at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
    at yyh.WC$.main(WC.scala:22)
    at yyh.WC.main(WC.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:256)
    at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:207)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:49)
    at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:52)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:231)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

意思就是你没有指定某些参数，而系统在找不到路径的情况下，会找默认的路径./assembly/target/scala-2.11/jars，但是路径中没有而报错

这里写图片描述

这说明参数错了么？不是，这是oozie4.10版本和spark2的bug：https://issues.apache.org/jira/browse/OOZIE-2606
这里写图片描述
根据这个issue的思路，这里选用给出jars路径的方式（这里走了一个弯路，他提示包重复：
，因为当时我上传的tmp路径在nm的共享之中，所以，传了两遍，所以报错，删掉上传tmp就行了）
给出jars的路径采用：

--num-executors 3 --driver-memory 1g --executor-memory 1g --conf spark.yarn.jars=hdfs://master110.weiboyi.com:8020/user/hive/yyhtest/RemoteShellExecutor.jar

其中RemoteShellExecutor.jar是没有用的jar包，只是为了设置spark.yarn.jars路径：是不是很机智啊！！！！因为放有用的冲突，而且这个加载感觉有顺序
这里写图片描述
这里写图片描述
/
下面放运行情况：
测试
这里写图片描述
运行成功率很高
这里写图片描述
调度
这里写图片描述
rm8088资源管理
这里写图片描述
邮箱通知
这里写图片描述

袁一白
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 4
    评论
专栏目录
使用oozie实现spark定时调度和spark依赖
zhangfan1994的博客
08-07 9896
oozie是针对hadoop的任务调度框架，因为spark on yarn的模式是基于hadoop的任务调度模块yarn来做的，所以spark on yarn模式可以利用oozie的定时模块和任务依赖调度模块，实现按时按步骤的执行spark的job，下面就简单介绍一下刚使用oozie时容易忽略和比较重要的地方。 最简单来讲，一个简单依赖关系的ooziejob，需要一个提交到hdfs上的xml用来解析
Oozie Spark on YARN requirement failed
fansy1990的专栏
12-24 4470
软件环境：CDH：5.7.3；Oozie：4.1.0-CDH5.7.3 ； Spark：1.6.0-cdh5.7.3-hadoop2.6.0-cdh5.7.3 ； Hadoop：hadoop2.6.0-cdh5.7.3（HDFS 采用HA方式）；问题描述：在使用CDH5.7.3版本的时候，发起一个Oozie工作流，该工作流使用Spark On YARN的方式提交一个Spark程序，但是在Oozie
oozie+spark2结合
03-01
CDH环境中oozie+spark2集成的详细步骤，让spark2也支持工作流。
oozie调用spark
cihongmo6452的博客
09-23 321
oozie单独调用spark的过程中，也是问题多多。 首先，你的spark程序肯定是要能单独跑过的。另外spark运行任务的几种方式的区别。 spark运行模式这篇博客有讲。 然后看我的一些配置吧，测试中碰到各种问题，关于这方面详细讲解的资料不多，只能一点点摸索尝试。 我...
推荐使用：高效多线程Java FFT库 - JTransforms
最新发布
gitblog_00050的博客
05-08 559
推荐使用：高效多线程Java FFT库 - JTransforms 项目地址:https://gitcode.com/wendykierp/JTransforms 1、项目介绍 JTransforms是一个创新的开源项目，它是首个纯Java编写的多线程快速傅里叶变换（FFT）库。该项目不仅提供了快速傅里叶变换，还支持离散余弦变换（DCT）、离散正弦变换（DST）和离散哈特利变换（DHT）。JTra...
Hue-oozie运行spark
12-17
本文档是入门文档，详细描述使用Hue编辑workflows提交spark程序。
工作流oozie文档
12-12
Oozie是一款由Apache基金会维护的开源工作流调度管理系统，主要用于管理和编排Hadoop生态系统中的各种任务。通过对不同类型的任务进行编排和调度，Oozie帮助开发者轻松地实现复杂的业务逻辑流程。 - **目标**：学习...
Hadoop+Spark+Hive+HBase+Oozie+Kafka+Flume+Flink+ES+Redash等详细安装部署
06-22
Oozie是Hadoop的工作流调度器，用于管理Hadoop作业（包括MapReduce、Pig、Hive、Sqoop等）和Spark作业的调度。配置Oozie时，需要设置Hadoop和Spark的相关路径，并创建Oozie数据库。 Kafka是一个分布式流处理平台，...
Jigsaw:用于创建 Oozie 工作流的基于拼图的 UI
06-25
Oozie是一个开源的工作流调度系统，它能够管理和协调Hadoop生态系统中的各种任务。Oozie是Apache软件基金会的一个顶级项目，旨在简化大数据处理过程中的作业管理和调度。它支持Hadoop MapReduce、Pig、Hive、Sqoop等...
hue-4.1大数据监控利器
10-19
- **Oozie工作流可视化**：用户可以创建、编辑和监控Oozie工作流，流程图式的展示使得工作流管理更加直观。 - **更好的安全性**：Hue 4.1增强了与Kerberos的集成，提供了更强大的身份验证和授权机制，确保数据的...
hue-4.0大数据监控利器
10-19
4. **工作流管理**：Oozie工作流设计器是Hue的重要组成部分，4.0版本中，它提供了更流畅的工作流创建和编辑体验，支持拖拽式操作，使得复杂的工作流程设计变得简单。 5. **集群监控**：Hue 4.0集成了对Hadoop集群的...
spark机器学习二 pipeline工作流
24koby
10-09 200
import org.apache.spark.ml.classification.LogisticRegression import org.apache.spark.ml.linalg.Vectors import org.apache.spark.ml.linalg.Vector import org.apache.spark.ml.param.ParamMap import org.ap...
大数据篇：oozie与spark2整合进行资源调度
yycc
08-24 1659
前言： Oozie是Hadoop平台上开源的工作流调度引擎，通过hue集成的oozie管理界面，可以清晰地发布/查看/管理相关调度任务，基于此可以完成整个BI中间表，结果表的存储与计算。 整体流程如下： 文章目录1.oozie资源调度1.1 oozie概览1.2 oozie与spark2整合1.2.1 oozie添加spark2依赖库1.2.2 打包与提交spark2 jar, 创建spark...
大数据 | Spark机器学习工作流开发指南
大数据Spark教程
12-25 279
Spark.ml是在Spark 1.2开始引入的一个包，它旨在提供一套统一的高级API，帮助用户创建和优化实用的机器学习工作流，它在原来的MLlib的基础上进行了大量的改进和优化，让Spark生态更见坚不可摧，本文就来详细介绍一下Spark机器学习工作流的基本概念和用法。 —▼— 我建了一个QQ学习交流群，旨在“分享、讨论、学习、资源分享、就业机会、互联网内推、共同进步！”，感兴趣的可以加...
Spark实战(十一)Spark2.x集成Hive
钟晚语夕的专栏
04-03 1070
一、Hive配置 （一）、简介    一般的公司都会有自己的数据仓库，而大多数都选择的Hive数据仓库，总所周知hive默认使用MapReduce来进行数据操作，MapReduce在计算过程中会涉及数量巨大的网络传输，这需要耗费大量的时间，在计算过程中会反复读写磁盘，极其耗时，因此使用spark来计算可以提高效率，但是之前有很多数据都是通过hive来操作，好在spark可以无缝集成hive，使用h...
Oozie 提交Spark On Yarn示例
热门推荐
pt707621764的专栏
09-05 1万+
大家好，我是crazy_老中医，我写程序就像老中医一样，全屏感觉和经验，但是有用！ 废话不多说，现在开始正文，本文将阐述如何将一个Spark程序通过oozie提交到hadoop的Yarn上运行。 准备工作 集群规划   hdp-master hdp-slave1 hdp-slave2 hadoop NameNod
Spark2与Oozie整合
Gedeon
04-20 8365
项目背景 公司部门成立初期，现在业务是由Shell脚本编写然后定时进行运行。由于现在公司一般都是搭建的CDH，内置有Oozie。且Oozie操作简单，功能强大，并且有很好的图形化界面所以还是想要搞一下。在搞的过程中出现了一堆问题，就是spark2与Oozie的整合现在不是很成熟，网上资料也很少，基本是淌着过去的。。。 运行环境 CDH:CDH-5.13.2-1.cdh5.13.2. Ja...
[oozie基础]-- 部署spark、hadoop、hive、shell和java程序举例
欢迎来到我的博客，一起探索代码里的世界！
07-09 2682
一、oozie执行shell脚本(执行mr任务,实现合并增量数据) 参考：http://gethue.com/use-the-shell-action-in-oozie/ 1、点击创建、拖动到上面   2、添加命令:bash，当然也可以是linux的其他可执行的命令     3、添加参数：注意---》参数名称是shell脚本的全称(run-mr-compact.sh)    ...
CDH部署Hive、Spark、oozie、hue、sqoop等
菜鸟的博客
07-18 1121
hive spark cdh
oozie教程-实现你自己的hadoop/spark作业工作流
03-16
Oozie是一个开源的工作流协调器，可以用于管理和调度Hadoop和Spark作业。通过Oozie，用户可以定义和运行复杂的作业工作流，包括多个Hadoop和Spark作业的依赖关系和执行顺序。以下是实现自己的Hadoop/Spark作业工作流...
评论 4
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值