看example源码学spark系列(4)-DriverSubmissionTest

先运行

jpan@jpan-Beijing:~/Software/spark-0.9.1$ ./bin/run-example org.apache.spark.examples.DriverSubmissionTest 3
Environment variables containing SPARK_TEST:
System properties containing spark.test:
Alive for 1 out of 3 seconds
Alive for 2 out of 3 seconds

发现打印出来的信息有点问题，SPARK_TEST后面什么都没有，我们分析下源码

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.spark.examples

import scala.collection.JavaConversions._

/** Prints out environmental information, sleeps, and then exits. Made to
  * test driver submission in the standalone scheduler. */
object DriverSubmissionTest {
  def main(args: Array[String]) {
    if (args.size < 1) {
      println("Usage: DriverSubmissionTest <seconds-to-sleep>")
      System.exit(0)
    }
    val numSecondsToSleep = args(0).toInt

    val env = System.getenv()
    val properties = System.getProperties()

    println("Environment variables containing SPARK_TEST:")
    env.filter{case (k, v) => k.contains("SPARK_TEST")}.foreach(println)

    println("System properties containing spark.test:")
    properties.filter{case (k, v) => k.toString.contains("spark.test")}.foreach(println)

    for (i <- 1 until numSecondsToSleep) {
      println(s"Alive for $i out of $numSecondsToSleep seconds")
      Thread.sleep(1000)
    }
  }
}

从源码中可以看出，程序是打印系统变量中包含SPARK_TEST属性的值。但是我本机的env 和 properties如下（这是在spark-shell下）：

scala> val env = System.getenv()
env: java.util.Map[String,String] = {TERM=xterm, XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0, HADOOP_COMMON_LIB_NATIVE_DIR=/home/jpan/Software/hadoop-2.2.0/lib/native, SSH_AGENT_PID=2091, JAVA_HOME=/usr/share/jdk1.7.0_51, SSH_AGENT_LAUNCHER=upstart, LESSCLOSE=/usr/bin/lesspipe %s %s, UPSTART_SESSION=unix:abstract=/com/ubuntu/upstart-session/1000/2035, SESSION_MANAGER=local/jpan-Beijing:@/tmp/.ICE-unix/2199,unix/jpan-Beijing:/tmp/.ICE-unix/2199, LC_NUMERIC=zh_CN.UTF-8, GNOME_DESKTOP_SESSION_ID=this-is-deprecated, COMPIZ_CONFIG_PROFILE=ubuntu, IM_CONFIG_PHASE=1, GDMSESSION=ubuntu, MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path, PWD=/home/jpan/Software/spark-0.9.1, SESSIONTYPE=gnome-session, GTK_IM_MODULE=ibus, MASTER=spark://jpan-Beijing:7077, XDG_GREETER_DATA_DIR=/va...
scala> val properties = System.getProperties()
properties: java.util.Properties = 
{java.runtime.name=Java(TM) SE Runtime Environment, =/usr/share/jdk1.7.0_51/jre/lib/amd64, java.vm.version=24.51-b03, java.vm.vendor=Oracle Corporation, java.vendor.url=http://java.oracle.com/, path.separator=:, java.vm.name=Java HotSpot(TM) 64-Bit Server VM, file.encoding.pkg=sun.io, user.country=US, sun.java.launcher=SUN_STANDARD, sun.os.patch.level=unknown, java.vm.specification.name=Java Virtual Machine Specification, user.dir=/home/jpan/Software/spark-0.9.1, java.runtime.version=1.7.0_51-b13, java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment, java.endorsed.dirs=/usr/share/jdk1.7.0_51/jre/lib/endorsed, os.arch=amd64, java.io.tmpdir=/tmp, line.separator=
, java.vm.specification.vendor=Oracle Corporation, os.name=Linux, sun.jn...

其中并没有包含spark_test字样，所以没有打印出来。于是我自己见了项目，并把SPARK_TEST改为MASTER，把spark_test改为sun.boot.library.path，运行结果如下：

jpan@jpan-Beijing:~/Mywork/spark_test/DriverSubmissionTest$ sbt "project driversubmissiontest" "run 3"
[info] Set current project to DriverSubmissionTest (in build file:/home/jpan/Mywork/spark_test/DriverSubmissionTest/)
[info] Set current project to DriverSubmissionTest (in build file:/home/jpan/Mywork/spark_test/DriverSubmissionTest/)
[info] Compiling 1 Scala source to /home/jpan/Mywork/spark_test/DriverSubmissionTest/target/scala-2.10/classes...
[info] Running main.scala.DriverSubmissionTest 3
Environment variables containing SPARK_TEST:
(MASTER,spark://jpan-Beijing:7077)
System properties containing spark.test:
(sun.boot.library.path,/usr/share/jdk1.7.0_51/jre/lib/amd64)
Alive for 1 out of 3 seconds
Alive for 2 out of 3 seconds
[success] Total time: 9 s, completed Jun 4, 2014 3:49:10 PM

即可以看到结果。

源码分析：

这个源码比较简单，就是打印出系统信息，并sleep几秒后退出。程序里面有意思的是filter中的case(k,v)，它自动把等好前面的赋值为K，后面赋值为V。

scala语法里有。