1.Scala入门
想用好Spark生态,必须学习Scala语言。想学习一门语言,必须先学会写hello world。传送门:Scala菜鸟教程
2.Scala整合Maven
1)下载一个支持Scala、Maven的IDE(如:Scala IDE for Eclipse )
2)创建一个Maven工程
3)选择合适的Scala版本,并在pom文件中定义scala.version属性
4)在pom文件中定义scala.tools.version属性(注意要与scala.version匹配)
5)在maven中央仓库 查找以下库的合适版本,注意要与scala.tools.version匹配,且这些库两两之间版本必须匹配
(通过maven中央仓库的详情页面中的Compile Dependencies 表格可识别库之间的版本配套关系)
spark-core、spark-sql、spark-streaming、kafka、spark-streaming-kafka、mongo-spark-connector
pom文件示例:
<project xmlns=
"http://maven.apache.org/POM/4.0.0"
xmlns:xsi=
"http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation=
"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"
>
<modelVersion>
4.0
.
0
</modelVersion>
<groupId>cn.com.flaginfo.demo</groupId>
<artifactId>scala-test</artifactId>
<version>
0.0
.
1
-SNAPSHOT</version>
<name>${project.artifactId}</name>
<description>My wonderfull scala app</description>
<inceptionYear>
2010
</inceptionYear>
<licenses>
<license>
<name>My License</name>
<url>http:
//....</url>
<distribution>repo</distribution>
</license>
</licenses>
<properties>
<maven.compiler.source>
1.6
</maven.compiler.source>
<maven.compiler.target>
1.6
</maven.compiler.target>
<encoding>UTF-
8
</encoding>
<scala.tools.version>
2.11
</scala.tools.version>
<scala.version>
2.11
.
0
</scala.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<!-- https:
//mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-cor