sbt的目录层次
|--build.sbt
|--lib
|--project
|--src
| |--main
| | |--scala
| |--test
| |--scala
|--sbt
|--target
拷贝spark安装目录的sbt目录的 sbt脚本和sbt的jar包
cp /path/to/spark/sbt/sbt* ~/wordcount/
由于spark的sbt脚本默认查找./sbt目录,修改如下
JAR=sbt/sbt-launch-${SBT_VERSION}.jar
to
JAR=sbt-launch-${SBT_VERSION}.jar
拷贝spark的jar包到,sbt的lib目录
cp /path/to/spark/assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar \
> ~/spark_wordcount/lib/
建立build.sbt配置文件,各行需要有一个空行分割
name := "WordCount"
[this is bank line]
version := "1.0.0"
[this is bank line]
scalaVersion := "2.10.3"
由于spark的sbt脚本需要到project的build.properties文件找sbt的版本号,我们建立该文件,增加如下内容:
sbt.version=0.12.4
创建wordcount
mkdir -p ~/wordcount/src/main/scala/spark/example
vi -p ~/wordcount/src/main/scala/spark/example/WordCount.scala
到wordcount目录,执行编译:
./sbt compile
打jar 包
./sbt package
编译过程,sbt需要上网下载依赖工具包,jna,scala等。编译完成后可以在target/scala-2.10/目录找到打包好的jar
WordCount执行
可以参考Spark分布式运行于hadoop的yarn上的方法,写一个执行脚本
#!/usr/bin/env bash
SPARK_JAR=./assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar \
./bin/spark-class org.apache.spark.deploy.yarn.Client \
--jar ~/spark_wordcount/target/scala-2.10/wordcount_2.10-1.0.0.jar \
--class spark.example.WordCount \
--args yarn-standalone \
--args /testWordCount.txt \
--args /resultWordCount \
--num-workers 3 \
--master-memory 4g \
--worker-memory 2g \
--worker-cores 2
然后,拷贝一个名为testWordCount.txt的文件进hdfs
hdfs dfs -copyFromLocal ./testWordCount.txt /testWordCount.txt