Spark学习chapter8 Tunning and Debugging Spark

Configuring Spark with SparkConf

Components of Execution:Jobs, Tasks, and Stages

Spark Execution

1. User Code defines a DAG(directed acyclic graph) of RDDs
   Operations on RDDs create new RDDs that refer back to their parents, thereby creating a graph.
2. Action force translation of the DAG to an execution plan
   When you call an action on an RDD it must be computed. This requires computing its parent RDDs as well. Spark’s scheduler sumbits a job to compute all needed RDDs. That job wil have one or more stages, which are parallel waves of computation composed of tasks. Each stage will correspond one or more RDDs in the DAG. A single stage can correspond to mutiple RDDs due to pipelining.
3. Tasks are scheduler and executed on a cluster
   Stages are porcessed in order, with individual launching to compute segments of the RDDs. Once the final stage is finished in a job, the action is complete.

filter() and coalesce()

Memory Management

1. RDD storage
   When you call persist() and cache() on an RDD, its partitions will be stored in memory buffers. Spark will limit the amount of memory used when caching to a certain fraction of the JVM’s overall heap, set by spark.storage.memoryFraction. If this limit is exceeded, older partitions will be dropped from memory.
2. Shuffle and aggregation buffers
   When performing shuffle operations, Spark will create intermediate buffers for storing shuffle output data. spark.shuffle.memeoryFraction
3. User Code
   User code has access to everything lefet by JVM heap after the space for RDD storage and shuffle storage are allocated.

By default, Spark will leave 60% of space for RDD storage, 20% for shuffle memory, and the remaining 20% for user programs.

Hardware Provisioning

hardware provisioning
spark.cores.max