Spark --- 启动、运行、关闭过程

计算PI值

// scalastyle:off println
package org.apache.spark.examples

import scala.math.random

import org.apache.spark._

/** Computes an approximation to pi */
object SparkPi {
   
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("Spark Pi")
    val spark = new SparkContext(conf)
    val slices = if (args.length > 0) args(0).toInt else 2
    val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow
    val count = spark.parallelize(1 until n, slices).map { i =>
      val x = random * 2 - 1
      val y = random * 2 - 1
      if (x*x + y*y < 1) 1 else 0
    }.reduce(_ + _)
    println("Pi is roughly " + 4.0 * count / n)
    spark.stop()
  }
}

流程分析

[abc@search-engine---dev4 spark]$ ./bin/run-example SparkPi
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/06/07 03:43:20 INFO SparkContext: Running Spark version 1.6.1
16/06/07 03:43:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
#进行acls用户权限认证
16/06/07 03:43:20 INFO SecurityManager: Changing view acls to: abc
16/06/07 03:43:20 INFO SecurityManager: Changing modify acls to: abc
16/06/07 03:43:20 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(abc); users with modify permissions: Set(abc)
16/06/07 03:43:21 INFO Utils: Successfully started service 'sparkDriver' on port 40568.
16/06/07 03:43:23 INFO Slf4jLogger: Slf4jLogger started
#启动远程监听服务，端口是36739，Spark的通信工作由akka来实现
16/06/07 03:43:23 INFO Remoting: Starting remoting
16/06/07 03:43:23 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@127.0.0.1:36739]
16/06/07 03:43:23 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port