-
直接执行 sbt 会在当前目录下面创建 target 目录
-
sbt 的目录格局一般为 lib/ (该目录下存储与编译相关的 jar 文件)
project/ src/main/scala/ src/main/test/scala -
复制 jar 文件 spark-assembly *hadoop2.5.1.jar 到 lib 目录下
[root@localhost word]# find ../spark -name “spark*jar” |grep assem
../spark/assembly/target/scala-2.10/spark-assembly-1.1.2-SNAPSHOT-hadoop2.5.1.jar
../spark/dist/lib/spark-assembly-1.1.2-SNAPSHOT-hadoop2.5.1.jar
[root@localhost word]# cp ../spark/dist/lib/spark-assembly-1.1.2-SNAPSHOT-hadoop2.5.1.jar lib/
[root@localhost word]# ls lib
spark-assembly-1.1.2-SNAPSHOT-hadoop2.5.1.jar -
编辑 wordcount.scala
import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.SparkContext._
object wordCount{def main(args: Array[String]){ if (args.length == 0) { System.err.println("Usage bin/spark-submit [options] --class wordCount wordCount.jar <file1:URI>") System.err.println("Usage bin/spark-submit [options] --class wordCount wordCount.jar hdfs://172.16.1.141:9000/test.txt") System.exit(1); } val conf = new SparkConf().setAppName("WordCount") val sc = new SparkContext(conf) val doc = sc.textFile(args(0)) doc.cache() val words = doc.flatMap(_.split("")) val pairs = words.map( x=> (x,1)) val res = pairs.reduceByKey(_+_) res.collect().foreach(println) sc.stop() }
}
-
编辑 build.sbt
[root@localhost word]# cat build.sbt
name := “wordCount”
[blank line]
version := “1.0”
[blank line]
scalaVersion := “2.11.4”
6 . 编译打包 成 jar 文件[root@localhost word]# sbt package -Dsbt.ivy.home=/root/.ivy2
[info] Set current project to wordCount (in build file:/opt/htt/temp_20140611/java/word/)
[info] Updating {file:/opt/htt/temp_20140611/java/word/}word…
[info] Resolving jline#jline;2.12 …
[info] Done updating.
[info] Compiling 2 Scala sources to /opt/htt/temp_20140611/java/word/target/scala-2.11/classes…
[warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list
[info] Packaging /opt/htt/temp_20140611/java/word/target/scala-2.11/wordcount_2.11-1.0.jar …
[info] Done packaging.
[success] Total time: 11 s, completed Jan 5, 2015 8:37:38 AM
[root@localhost word]#- 编译 class 文件到当前目录
scalac src/main/scala/wordCount.scala -cp lib/spark-assembly-1.1.2-SNAPSHOT-hadoop2.5.1.jar
-
调用spark 执行
../spark/bin/spark-submit –class wordCount target/scala-2.11/wordcount_2.11-1.0.jar hdfs://172.16.1.141:9000/opt/old/htt/test/test.txt
参考文章: http://www.aboutyun.com/thread-8587-1-1.html
sbt介绍 sbt是一个代码编译工具,是scala界的mvn,可以编译scala,java等,需要java1.6以上。 sbt项目环境建立 sbt编译需要固定的目录格式,并且需要联网,sbt会将依赖的jar包下载到用户home的.ivy2下面,目录结构如下: 以上建立目录如下: 然后拷贝spark安装目录的sbt目录的 sbt脚本和sbt的jar包 由于spark的sbt脚本默认查找./sbt目录,修改如下 拷贝spark的jar包到,sbt的lib目录 建立build.sbt配置文件,各行需要有一个空行分割 由于spark的sbt脚本需要到project的build.properties文件找sbt的版本号,我们建立该文件,增加如下内容: Spark WordCount程序编写及编译 建立WordCount.scala源文件,假设需要包为spark.example 添加具体的程序代码,并保存 到spark_wordcount目录,执行编译: 打成jar包 编译过程,sbt需要上网下载依赖工具包,jna,scala等。编译完成后可以在target/scala-2.10/目录找到打包好的jar WordCount执行 可以参考Spark分布式运行于hadoop的yarn上的方法,写一个执行脚本 然后,拷贝一个名为testWordCount.txt的文件进hdfs 然后执行脚本,过一会就可以看到结果了 |