Spark系列(2)—A Standalone App in Scala

spark application 编译是用sbt编译并运行的，下面介绍下官网上实例。

1.编写源码

建立一个文件夹，命名为simpleApp

$mkdir simpleApp

另外还要创建一些列文件夹，其结构如下：

$ cd ..
$ cd simpleApp/
$ find .
.
./simple.sbt
./src
./src/main
./src/main/scala
./src/main/scala/simpleApp.scala

其中有两个文件，一个是simple.sbt,这个是sbt的编译配置文件，另一个simpleApp.scala是此app的源文件。其代码分别如下

simple.sbt

name := "Simple Project"

version := "1.0"

scalaVersion := "2.10.3"

libraryDependencies += "org.apache.spark" %% "spark-core" % "0.9.1"

resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

simpleApp.scala

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._

object SimpleApp {
  def main(args: Array[String]) {
    val logFile = "/home/jpan/Software/spark-0.9.1/README.md" // Should be some file on your system
    val sc = new SparkContext("local", "Simple App", "/home/jpan/Software/spark-0.9.1/",
    List("target/scala-2.10/simple-project_2.10-1.0.jar"))
    val logData = sc.textFile(logFile, 2).cache()
    val numAs = logData.filter(line => line.contains("a")).count()
    val numBs = logData.filter(line => line.contains("b")).count()
    println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
  }
}

2.运行代码

进入到simpleApp文件夹下，运行

$ ~/Software/spark-0.9.1/sbt/sbt package

出错啦！错误信息如下：

awk: fatal: cannot open file `./project/build.properties' for reading (No such file or directory)
Attempting to fetch sbt
/home/jpan/Software/spark-0.9.1/sbt/sbt: line 35: /sbt/sbt-launch-.jar: No such file or directory
/home/jpan/Software/spark-0.9.1/sbt/sbt: line 35: /sbt/sbt-launch-.jar: No such file or directory
Our attempt to download sbt locally to /sbt/sbt-launch-.jar failed. Please install sbt manually from http://www.scala-sbt.org/

查看错误信息，分析sbt代码，发现sbt有点问题，在0.9.1这个版本中用的相对地址，当你进入simpleApp这个目录时，导致版本信息和jar包找不到。修改$SPARK_HOME/sbt/stb文件

把以下两句

 SBT_VERSION=`awk -F "=" '/sbt\\.version/ {print $2}' ./project/build.properties`
 JAR=/sbt/sbt-launch-${SBT_VERSION}.jar

修改为

#SBT_VERSION=`awk -F "=" '/sbt\\.version/ {print $2}' $SPARK_HOME/project/build.properties`
#JAR=$SPARK_HOME/sbt/sbt-launch-${SBT_VERSION}.jar

当然，你首先得在自己的系统变量中把$SPARK_HOME设置好。

再次运行，成功啦！

jpan@jpan-Beijing:~/Software/simpleApp$ ~/Software/spark-0.9.1/sbt/sbt package
Launching sbt from /home/jpan/Software/spark-0.9.1/sbt/sbt-launch-0.12.4.jar
[info] Set current project to Simple Project (in build file:/home/jpan/Software/simpleApp/)
[info] Updating {file:/home/jpan/Software/simpleApp/}default-f9c2c0...
[info] Resolving com.codahale.metrics#metrics-graphite;3.0.0 ...
[info] Done updating.
[info] Compiling 1 Scala source to /home/jpan/Software/simpleApp/target/scala-2.10/classes...
[info] Packaging /home/jpan/Software/simpleApp/target/scala-2.10/simple-project_2.10-1.0.jar ...
[info] Done packaging.
[success] Total time: 14 s, completed May 8, 2014 5:59:34 PM

jpan@jpan-Beijing:~/Software/simpleApp$ ~/Software/spark-0.9.1/sbt/sbt run
Launching sbt from /home/jpan/Software/spark-0.9.1/sbt/sbt-launch-0.12.4.jar
[info] Set current project to Simple Project (in build file:/home/jpan/Software/simpleApp/)
[info] Running SimpleApp 
14/05/08 18:00:14 WARN util.Utils: Your hostname, jpan-Beijing resolves to a loopback address: 127.0.1.1; using 192.168.1.102 instead (on interface eth0)
14/05/08 18:00:14 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address
14/05/08 18:00:16 INFO slf4j.Slf4jLogger: Slf4jLogger started
14/05/08 18:00:16 INFO Remoting: Starting remoting
14/05/08 18:00:16 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@jpan-Beijing.local:40021]
14/05/08 18:00:16 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@jpan-Beijing.local:40021]
14/05/08 18:00:16 INFO spark.SparkEnv: Registering BlockManagerMaster
14/05/08 18:00:16 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20140508180016-da5a
....................
Lines with a: 62, Lines with b: 35
14/05/08 18:00:19 INFO network.ConnectionManager: Selector thread was interrupted!
[success] Total time: 6 s, completed May 8, 2014 6:00:19 PM

注：和Spark Admin交流说上面那个错误不是BUG，Spark下面的sbt/sbt只用于编译Spark项目，不用于运行其它项目。其它项目可以直接运行标准的sbt就行，即直接在simpleAPP目录下运行下面语句即可：

$sbt package
$sbt run