用sbt构建Spark项目——WordCount

一、环境配置

1、sbt:http://www.scala-sbt.org/download.html    选择windows的SBT-0.13.12.MSI即可,然后安装

2、在系统环境中path后追加“sbt安装目录/bin”

3、用cmd进入本地命令窗,输入“sbt”,等待jar包下载完成

4、进入C:\Users\***\.sbt\0.13\plugins,编辑plugins.sbt文件,添加两个插件,代码如下

addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")

若无plugins文件夹和plugins.sbt,手动创建即可

二、用sbt创建eclipse项目

1、进入你的eclipse中的workspace文件夹,新建一个文件夹SbtWordCount,然后新建build.sbt文件,输入如下配置信息

name := "sbt-wordcount" 
  
version := "1.0"

scalaVersion := "2.10.6"
autoScalaLibrary := false
EclipseKeys.createSrc := EclipseCreateSrc.Default + EclipseCreateSrc.Resource
EclipseKeys.createSrc := EclipseCreateSrc.Default + EclipseCreateSrc.ManagedClasses

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-core_2.10" % "1.5.2" % "provided",
  "org.apache.spark" % "spark-mllib_2.10" % "1.5.2" % "provided",
  "org.apache.spark" % "spark-examples_2.10" % "1.1.1" % "provided"
)

resolvers ++= Seq( 
      // HTTPS is unavailable for Maven Central  
      "Maven Repository"     at "http://repo.maven.apache.org/maven2",  
      "Apache Repository"    at "https://repository.apache.org/content/repositories/releases",  
      "JBoss Repository"     at "https://repository.jboss.org/nexus/content/repositories/releases/",  
      "MQTT Repository"      at "https://repo.eclipse.org/content/repositories/paho-releases/",  
      "Cloudera Repository"  at "http://repository.cloudera.com/artifactory/cloudera-repos/",
      "le_bigdata_mining"    at "http://10.150.144.28/nexus/content/repositories/releases/",  
      Resolver.mavenLocal  
)
2、在命令行窗口中,进入到SbtWordCount目录中,然后输入“sbt eclipse”,等待出现Successfully created Eclipse ....就可以了

3、进入Scala IDE (eclipse)中,import该项目

4、若发现没有src目录,就手动创建。要想在windows下本地执行spark,还得需要winutils.exe,所以将其放在null/bin目录中,可以参考上一篇文章windows中用scala-IDE开发spark—— WordCount

windows中用scala-IDE开发spark—— WordCount



阅读更多
个人分类: Spark
想对作者说点什么? 我来说一句

没有更多推荐了,返回首页

关闭
关闭
关闭