版本信息
spark2.1.1
scala 2.11.8
1.SBT
新建SBT项目
选择scala:2.11.8
添加依赖:spark_core
name := "SBTTest"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1"
scala wordcount代码
object WordCount {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setMaster("local[4]").setAppName("FirstSparkDemo")
val sc = new SparkContext(conf)
//② 从Scala 集合或者文件系统中创建RDD
//通过Scala 集合创建 RDD
//从HDFS 上创建RDD
val lines = sc.textFile("hdfs://172.17.11.85:9000/input/file.txt")
val flatRDD = lines.flatMap(_.split(" "))
println("size:"+lines.partitions.size)
flatRDD.map((_, 1)).reduceByKey(_ + _).collect.foreach(println)
}
}
选择SBTshell,输入compile,package
.
查看jar包
2.Maven
新建Maven项目
在pom文件中添加依赖
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.1.1</version>
</dependency>
</dependencies>
scala wordcount代码
编译打包
查看Jar包