引言
使用Hue可以方便的通过界面制定Oozie的工作流,支持Hive、Pig、Spark、Java、Sqoop、MapReduce、Shell等等。Spark?当让可以,但是自带是spark1的,那能不能支持Spark2的呢?接下来本文章就主要讲述如何使用Hue创建Spark1和Spark2的Oozie工作流。
安装spark2-lib到oozie
环境说明
spark2 :
CDH内
jar包在:111节点的 /opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/jars
oozie:
CDH内,
主地址:192.168.1.100:11000/oozie
lib目录是:HDFS:/user/oozie/share/lib/lib_20180606144037/
操作系统:centos
步骤说明
1>添加lib到oozie-lib下
#先进入一个能够管理员的账号,这里使用import账号(为什么叫这个账号,不重要)
#建立本地jars目录
mkdir sparkyyh
#拷贝spark
cp /opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/jars/* sparkyyh
#把oozie的相关jar放进sparkyyh,否则,会出现 oozie-spark的MainClass找不到
cp /opt/cloudera/parcels/CDH/lib/oozie/oozie-sharelib-yarn/lib/spark/oozie-sharelib-spark*.jar sparkyyh
#传到hgfs的lib上(现在知道为啥import了吧,因为这是一个管理员账户,可以sudo)
sudo -u hdfs hadoop fs -mkdir /user/oozie/share/lib/lib_20180606144037/sparkyyh
sudo -u hdfs hadoop fs -put ./sparkyyh/*.jar /user/oozie/share/lib/lib_20180606144037/sparkyyh
#记得最后要保持一致
sudo -u hdfs hadoop fs -chown -R oozie:oozie /user/oozie/share/lib/
sudo -u hdfs hadoop fs –chmod -R 775 /user/oozie/share/lib/lib_20180606144037/sparkyyh
效果图如下:
[import@slave113 yyh]$ oozie admin -oozie http://192.168.1.100:11000/oozie -shareliblist spark2
[Available ShareLib]
spark2
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/JavaEWAH-0.3.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/RoaringBitmap-0.5.11.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/ST4-4.0.4.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/antlr-2.7.7.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/antlr-runtime-3.4.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/antlr4-runtime-4.5.3.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/aopalliance-repackaged-2.4.0-b34.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/apache-log4j-extras-1.2.17.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/arpack_combined_all-0.1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/avro-1.7.6-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/avro-ipc-1.7.6-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/avro-mapred-1.7.6-cdh5.7.0-hadoop2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/bonecp-0.7.1.RELEASE.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/breeze-macros_2.11-0.12.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/breeze_2.11-0.12.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/calcite-avatica-1.2.0-incubating.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/calcite-core-1.2.0-incubating.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/calcite-linq4j-1.2.0-incubating.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/chill-java-0.8.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/chill_2.11-0.8.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-cli-1.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-codec-1.10.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-collections-3.2.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-compiler-3.0.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-compress-1.4.1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-crypto-1.0.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-dbcp-1.4.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-httpclient-3.1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-io-2.4.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-lang-2.6.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-lang3-3.5.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-logging-1.1.3.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-math3-3.4.1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-net-2.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/commons-pool-1.5.4.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/compress-lzf-1.0.3.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/core-1.1.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/datanucleus-api-jdo-3.2.6.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/datanucleus-core-3.2.10.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/datanucleus-rdbms-3.2.9.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/derby-10.11.1.1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/eigenbase-properties-1.1.5.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/fastutil-6.3.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/flume-ng-configuration-1.6.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/flume-ng-core-1.6.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/flume-ng-sdk-1.6.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/gson-2.2.4.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hive-metastore-1.1.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hive-serde-1.1.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hive-shims-0.23-1.1.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hive-shims-1.1.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hive-shims-common-1.1.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hive-shims-scheduler-1.1.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hk2-api-2.4.0-b34.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hk2-locator-2.4.0-b34.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/hk2-utils-2.4.0-b34.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/httpcore-4.4.4.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/ivy-2.4.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jackson-annotations-2.6.5.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jackson-core-2.6.5.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jackson-databind-2.6.5.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jackson-jaxrs-1.8.8.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jackson-module-paranamer-2.6.5.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jackson-module-scala_2.11-2.6.5.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jackson-xc-1.8.8.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/janino-3.0.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/javassist-3.18.1-GA.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/javax.annotation-api-1.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/javax.inject-2.4.0-b34.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/javax.servlet-api-3.1.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/javax.ws.rs-api-2.0.1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/javolution-5.5.1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jcl-over-slf4j-1.7.5.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jdo-api-3.0.1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jersey-client-2.22.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jersey-common-2.22.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jersey-container-servlet-2.22.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jersey-container-servlet-core-2.22.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jersey-guava-2.22.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jersey-media-jaxb-2.22.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jersey-server-2.22.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jetty-6.1.26.cloudera.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jetty-util-6.1.26.cloudera.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jline-2.12.1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/joda-time-2.9.3.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jodd-core-3.5.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/json-20090211.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/json4s-ast_2.11-3.2.11.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/json4s-core_2.11-3.2.11.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/json4s-jackson_2.11-3.2.11.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jta-1.1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jtransforms-2.4.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/jul-to-slf4j-1.7.5.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/kryo-shaded-3.0.3.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/libfb303-0.9.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/libthrift-0.9.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/logredactor-1.0.3.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/lz4-1.3.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/metrics-core-3.1.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/metrics-graphite-3.1.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/metrics-json-3.1.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/metrics-jvm-3.1.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/mina-core-2.0.0-M5.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/minlog-1.3.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/netty-3.8.0.Final.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/netty-all-4.0.42.Final.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/objenesis-2.1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/opencsv-2.3.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/oro-2.0.8.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/osgi-resource-locator-1.0.1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/paranamer-2.3.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/parquet-avro-1.5.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/parquet-column-1.5.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/parquet-common-1.5.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/parquet-encoding-1.5.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/parquet-format-2.1.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/parquet-hadoop-1.5.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/parquet-hadoop-bundle-1.5.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/parquet-jackson-1.5.0-cdh5.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/pmml-model-1.2.15.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/pmml-schema-1.2.15.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/py4j-0.10.4.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/pyrolite-4.13.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/scala-compiler-2.11.8.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/scala-library-2.11.8.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/scala-parser-combinators_2.11-1.0.4.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/scala-reflect-2.11.8.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/scala-xml_2.11-1.0.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/scalap-2.11.8.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/shapeless_2.11-2.0.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/snappy-0.2.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-catalyst_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-core_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-graphx_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-hive-exec_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-hive_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-launcher_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-mllib-local_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-mllib_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-network-common_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-network-shuffle_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-repl_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-sketch_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-sql_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-streaming-flume-sink_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-streaming-flume_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-streaming_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-tags_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-unsafe_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spark-yarn_2.11-2.1.0.cloudera1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spire-macros_2.11-0.7.4.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/spire_2.11-0.7.4.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/stax-api-1.0.1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/stream-2.7.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/stringtemplate-3.2.1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/univocity-parsers-2.2.1.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/validation-api-1.1.0.Final.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/xbean-asm5-shaded-4.4.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/jars/xz-1.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/oozie-sharelib-spark-4.1.0-cdh5.9.0.jar
hdfs://master110.weiboyi.com:8020/user/oozie/share/lib/lib_20180606144037/spark2/oozie-sharelib-spark.jar
2>添加此lib到oozie系统中(更新Oozie的share-lib)
#添加
oozie admin -oozie http://192.168.1.100:11000/oozie -sharelibupdate
#确认
oozie admin -oozie http://192.168.1.100:11000/oozie -shareliblist
应当返回
测试oozie中的spark2–local
从下一张图可以看出,要添加sparkyyh属性
通过以下的参数可以看出,sparkyyh已经成功添加
这是8088运行
运行结果如下,是分析一个hdfs上文件的WC(代码多low,不重要,因为要的是能操作hdfs)
测试oozie中的spark2–cluster(代码中无local)
首先这个过程是踩了俩天的坑,才踩出来的
直接提交client或者cluster,将会报错
2018-06-27 09:10:18,710 WARN org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[manager100.weiboyi.com] USER[hive] GROUP[-] TOKEN[] APP[yyhTestWorkflow-spark] JOB[0000402-180621160350771-oozie-oozi-W] ACTION[0000402-180621160350771-oozie-oozi-W@spark-5908] Launcher exception: Library directory '/data3/yarn/nm/usercache/hive/appcache/application_1528447799658_20866/container_1528447799658_20866_01_000070/./assembly/target/scala-2.11/jars' does not exist; make sure Spark is built.
java.lang.IllegalStateException: Library directory '/data3/yarn/nm/usercache/hive/appcache/application_1528447799658_20866/container_1528447799658_20866_01_000070/./assembly/target/scala-2.11/jars' does not exist; make sure Spark is built.
at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
at org.apache.spark.launcher.CommandBuilderUtils.findJarsDir(CommandBuilderUtils.java:347)
at org.apache.spark.launcher.YarnCommandBuilderUtils$.findJarsDir(YarnCommandBuilderUtils.scala:38)
at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:526)
at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:814)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:169)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
at yyh.WC$.main(WC.scala:22)
at yyh.WC.main(WC.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:256)
at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:207)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:49)
at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:231)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
意思就是你没有指定某些参数,而系统在找不到路径的情况下,会找默认的路径./assembly/target/scala-2.11/jars,但是路径中没有而报错
这说明参数错了么?不是,这是oozie4.10版本和spark2的bug:https://issues.apache.org/jira/browse/OOZIE-2606
根据这个issue的思路,这里选用给出jars路径的方式(这里走了一个弯路,他提示包重复:
,因为当时我上传的tmp路径在nm的共享之中,所以,传了两遍,所以报错,删掉上传tmp就行了)
给出jars的路径采用:
--num-executors 3 --driver-memory 1g --executor-memory 1g --conf spark.yarn.jars=hdfs://master110.weiboyi.com:8020/user/hive/yyhtest/RemoteShellExecutor.jar
其中RemoteShellExecutor.jar是没有用的jar包,只是为了设置spark.yarn.jars路径:是不是很机智啊!!!!因为放有用的冲突,而且这个加载感觉有顺序
/
下面放运行情况:
测试
运行成功率很高
调度
rm8088资源管理
邮箱通知