用SBT编译Spark的WordCount程序

sbt的目录层次

    |--build.sbt
    |--lib
    |--project
    |--src
    |   |--main
    |   |    |--scala
    |   |--test
    |         |--scala
    |--sbt
    |--target

拷贝spark安装目录的sbt目录的 sbt脚本和sbt的jar包

 cp /path/to/spark/sbt/sbt* ~/wordcount/

由于spark的sbt脚本默认查找./sbt目录，修改如下

JAR=sbt/sbt-launch-${SBT_VERSION}.jar
to
JAR=sbt-launch-${SBT_VERSION}.jar


拷贝spark的jar包到，sbt的lib目录

    cp /path/to/spark/assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar \
    > ~/spark_wordcount/lib/ 

建立build.sbt配置文件,各行需要有一个空行分割

    name := "WordCount"
    [this is bank line]
    version := "1.0.0"
    [this is bank line]
    scalaVersion := "2.10.3"

由于spark的sbt脚本需要到project的build.properties文件找sbt的版本号，我们建立该文件，增加如下内容：

    sbt.version=0.12.4

创建wordcount

mkdir -p ~/wordcount/src/main/scala/spark/example
  vi -p ~/wordcount/src/main/scala/spark/example/WordCount.scala

到wordcount目录，执行编译：

./sbt compile

打jar 包

./sbt package

编译过程，sbt需要上网下载依赖工具包，jna，scala等。编译完成后可以在target/scala-2.10/目录找到打包好的jar

WordCount执行
可以参考Spark分布式运行于hadoop的yarn上的方法，写一个执行脚本

#!/usr/bin/env bash

SPARK_JAR=./assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar \
    ./bin/spark-class org.apache.spark.deploy.yarn.Client \
      --jar ~/spark_wordcount/target/scala-2.10/wordcount_2.10-1.0.0.jar \
      --class  spark.example.WordCount \
      --args yarn-standalone \
      --args /testWordCount.txt \
      --args /resultWordCount \
      --num-workers 3 \
      --master-memory 4g \
      --worker-memory 2g \
      --worker-cores 2

然后，拷贝一个名为testWordCount.txt的文件进hdfs

  hdfs dfs -copyFromLocal ./testWordCount.txt /testWordCount.txt

转自 http://www.aboutyun.com/thread-8587-1-1.html