Spark env

1.给driver的分配core

local => 1

local[5] =>5

local[*] =>当前进程能够给虚拟机分配的最大core数

非本地模式下都为0

2.初始化 SecurityManager

3.初始化RPCEnv,同时初始化Rpc工厂(默认为netty)然后使用工厂创建RpcEnv

4.使用AkkaUtil创建ActorSystem

5.初始化序列化器,并实例化对象

val serializer = instantiateClassFromConf[Serializer](
"spark.serializer", "org.apache.spark.serializer.JavaSerializer")
logDebug(s"Using serializer: ${serializer.getClass}")
//暂时没有用到该序列化方式
val closureSerializer = instantiateClassFromConf[Serializer](
"spark.closure.serializer", "org.apache.spark.serializer.JavaSerializer")

6.创建MapOutputTrackerMaster

val mapOutputTracker = if (isDriver) {
new MapOutputTrackerMaster(conf)
} else {
new MapOutputTrackerWorker(conf)
}
7.初始化MapOutputTrackerMasterEndpoint，并将其向RpcEnv注册，以Netty为例
dispatcher.registerRpcEndpoint(name, endpoint)
8.获取ShuffleManager的配置,默认为sort
val shortShuffleMgrNames = Map(
"hash" -> "org.apache.spark.shuffle.hash.HashShuffleManager",
"sort" -> "org.apache.spark.shuffle.sort.SortShuffleManager",
"tungsten-sort" -> "org.apache.spark.shuffle.sort.SortShuffleManager")
9.初始化MemoryManager
val useLegacyMemoryManager = conf.getBoolean("spark.memory.useLegacyMode", false)
val memoryManager: MemoryManager =
if (useLegacyMemoryManager) {
    //固定内存
new StaticMemoryManager(conf, numUsableCores)
  } else {
UnifiedMemoryManager(conf, numUsableCores)
  }
10.创建blockTransferService，扩展自ShuffleClient
val blockTransferService = 
new NettyBlockTransferService(conf, securityManager, numUsableCores)
11.创建BlockManagerMaster和BlockManager,通过注册
BlockManagerMasterEndpoint实现，并传入NettyRpcEnv和LiveListenerBus
val blockManagerMaster = new BlockManagerMaster(registerOrLookupEndpoint(
  BlockManagerMaster.DRIVER_ENDPOINT_NAME,
new BlockManagerMasterEndpoint(rpcEnv, isLocal, conf, listenerBus)),
  conf, isDriver)

// NB: blockManager is not valid until initialize() is called later.
//executorId="driver" 
//val blockManager = new BlockManager(executorId, rpcEnv, blockManagerMaster,
  serializer, conf, memoryManager, mapOutputTracker, shuffleManager,
blockTransferService, securityManager, numUsableCores)
12.创建broadcastManager
valbroadcastManager = new BroadcastManager(isDriver, conf, securityManager)
13.创建cacheManager
val cacheManager = new CacheManager(blockManager)
14.创建MetricsSystem
val metricsSystem = if (isDriver) {
// Don't start metrics system right now for Driver.
  // We need to wait for the task scheduler to give us an app ID.
  // Then we can start the metrics system.
  //不会立即启动，需要等待TaskScheduler给分配AppId
MetricsSystem.createMetricsSystem("driver", conf, securityManager)
} else {
// We need to set the executor ID before the MetricsSystem is created because sources and
  // sinks specified in the metrics configuration file will want to incorporate this executor's
  // ID into the metrics they report.
//需要传递ExecutorId
conf.set("spark.executor.id", executorId)
val ms = MetricsSystem.createMetricsSystem("executor", conf, securityManager)
  ms.start()
  ms
}
15.为后续下载依赖文件指定存放路径
// Set the sparkFiles directory, used when downloading dependencies.  In local mode,
// this is a temporary directory; in distributed mode, this is the executor's current working
// directory.
val sparkFilesDir: String = if (isDriver) {
  Utils.createTempDir(Utils.getLocalDir(conf), "userFiles").getAbsolutePath
} else {
"."
}
16.初始化 OutputCommitCoordinator,向RpcEnv注册，并返回outputCommitCoordinatorRef

val outputCommitCoordinator = mockOutputCommitCoordinator.getOrElse {
new OutputCommitCoordinator(conf, isDriver)
}
val outputCommitCoordinatorRef = registerOrLookupEndpoint("OutputCommitCoordinator",
new OutputCommitCoordinatorEndpoint(rpcEnv, outputCommitCoordinator))