目前公司的spark平台不能直接采用shell进行登陆和任务发布,通过hue进行任务的发布,并可以完成相关的任务执行。当然在目前的条件下,能够算是完成了预期的目标,后期随着大家对spark平台的认识的加深和理解的深入,进而进行spark优化。
在进行任务发布相关流程讲解之前,首先对相关的环境进行配置,有关环境配置一定要与spark平台的保持一致:
1、公司的安装spark版本为2.3.0,目前spark主要支持Java、scala、python的开发,编程语言版本:
1)、java版本:支持Java版本为1.8(必备),对应开发环境选择:IDEA
2)、scala版本:2.11.x,对应开发环境选择:IDEA
3)、python版本:2.7,对应开发环境选择:anaconda
spark版本信息:CDH 6.1.1版本信息.pdf
2.任务发布流程:
1)新建maven项目:注意选择Java版本为1.8
2)填写GroupId和ArtifactId:
3)设置保存路径:
4)开启Auto-Import:
5)编辑pom:
注:自动生成部分不可修改,后续需要加入相关的依赖项,如:spark版本、Scala 版本等等,常用的部分见附录1;
6)选择scala版本:
注:此处一定要确定选择如图的版本并且确保与代码中的Scala版本保持一致
7)配置环境
src目录下面添加scala目录,并设置为Source Folders
test目录下面添加scala目录和resources目录,并分别设置为Source Folders和Test Resource Folders
8)新建实例data_save_sc:
在src/scala目录下,右键New一个Scala Class,命名为ScalaPi,类型选择为Object
在此实例中编写代码,并进行调试,完成编辑及调试,方可进行下一步;
9)设置环境,打jar包:
10)生成jar包:
11)登陆hue,进入spark界面:
12)上载jar包:
离线任务的核心难点在于jar包生成,对于相关的版本支持等需要进行调整
4.0.0
<groupId>word_cnt</groupId>
<artifactId>spark_test</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<spark.version>2.3.0</spark.version>
<cupid.sdk.version>3.3.8-public</cupid.sdk.version>
<scala.version>2.11.8</scala.version>
<scala.binary.version>2.11</scala.binary.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
<exclusions>
<exclusion>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</exclusion>
<exclusion>
<groupId>org.scala-lang</groupId>
<artifactId>scalap</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<minimizeJar>false</minimizeJar>
<shadedArtifactAttached>true</shadedArtifactAttached>
<artifactSet>
<includes>
<!-- Include here the dependencies you
want to be packed in your fat jar -->
<include>*:*</include>
</includes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
<exclude>**/log4j.properties</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/services/org.apache.spark.sql.sources.DataSourceRegister</resource>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.3.2</version>
<executions>
<execution>
<id>scala-compile-first</id>
<phase>process-resources</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>scala-test-compile-first</id>
<phase>process-test-resources</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>