用sbt构建Spark项目——WordCount

一、环境配置

1、sbt：http://www.scala-sbt.org/download.html    选择windows的SBT-0.13.12.MSI即可，然后安装

2、在系统环境中path后追加“sbt安装目录/bin”

3、用cmd进入本地命令窗，输入“sbt”，等待jar包下载完成

4、进入C:\Users\***\.sbt\0.13\plugins，编辑plugins.sbt文件，添加两个插件，代码如下

addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")

若无plugins文件夹和plugins.sbt，手动创建即可

二、用sbt创建eclipse项目

1、进入你的eclipse中的workspace文件夹，新建一个文件夹SbtWordCount，然后新建build.sbt文件，输入如下配置信息

name := "sbt-wordcount" 
  
version := "1.0"

scalaVersion := "2.10.6"
autoScalaLibrary := false
EclipseKeys.createSrc := EclipseCreateSrc.Default + EclipseCreateSrc.Resource
EclipseKeys.createSrc := EclipseCreateSrc.Default + EclipseCreateSrc.ManagedClasses

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-core_2.10" % "1.5.2" % "provided",
  "org.apache.spark" % "spark-mllib_2.10" % "1.5.2" % "provided",
  "org.apache.spark" % "spark-examples_2.10" % "1.1.1" % "provided"
)

resolvers ++= Seq( 
      // HTTPS is unavailable for Maven Central  
      "Maven Repository"     at "http://repo.maven.apache.org/maven2",  
      "Apache Repository"    at "https://repository.apache.org/content/repositories/releases",  
      "JBoss Repository"     at "https://repository.jboss.org/nexus/content/repositories/releases/",  
      "MQTT Repository"      at "https://repo.eclipse.org/content/repositories/paho-releases/",  
      "Cloudera Repository"  at "http://repository.cloudera.com/artifactory/cloudera-repos/",
      "le_bigdata_mining"    at "http://10.150.144.28/nexus/content/repositories/releases/",  
      Resolver.mavenLocal  
)

2、在命令行窗口中，进入到SbtWordCount目录中，然后输入“sbt eclipse”，等待出现Successfully created Eclipse ....就可以了

3、进入Scala IDE （eclipse）中，import该项目

4、若发现没有src目录，就手动创建。要想在windows下本地执行spark，还得需要winutils.exe，所以将其放在null/bin目录中，可以参考上一篇文章windows中用scala-IDE开发spark—— WordCount

windows中用scala-IDE开发spark—— WordCount