Spark Cluster Mode Overview

Basics

Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext object in your main program (called the driver program).

flow

Specifically, to run on a cluster, the SparkContext can connect to several types of cluster managers (either Spark’s own standalone cluster manager, Mesos or YARN), which allocate resources across applications. Once connected, Spark acquires executors on nodes in the cluster, which are processes that run computations and store data for your application. Next, it sends your application code (defined by JAR or Python files passed to SparkContext) to the executors. Finally, SparkContext sends tasks to the executors to run.

Architect

• Each application gets its own executor processes, which stay up for
  the duration of the whole application and run tasks in multiple
  threads. This has the benefit of isolating applications from each
  other, on both the scheduling side (each driver schedules its own
  tasks) and executor side (tasks from different applications run in
  different JVMs). However, it also means that data cannot be shared
  across different Spark applications (instances of SparkContext)
  without writing it to an external storage system.
• Spark is agnostic to the underlying cluster manager. As long as it
  can acquire executor processes, and these communicate with each
  other, it is relatively easy to run it even on a cluster manager that
  also supports other applications (e.g. Mesos/YARN).
• The driver program must listen for and accept incoming connections
  from its executors throughout its lifetime (e.g., see
  spark.driver.port in the network config section). As such, the driver
  program must be network addressable from the worker nodes.
• Because the driver schedules tasks on the cluster, it should be run
  close to the worker nodes, preferably on the same local area network.
  If you’d like to send requests to the cluster remotely, it’s better
  to open an RPC to the driver and have it submit operations from
  nearby than to run a driver far away from the worker nodes.