Configuring Spark with SparkConf
Components of Execution:Jobs, Tasks, and Stages
Spark Execution
- User Code defines a DAG(directed acyclic graph) of RDDs
Operations on RDDs create new RDDs that refer back to their parents, thereby creating a graph. - Action force translation of the DAG to an execution plan
When you call an action on an RDD it must be computed. This requires computing its parent RDDs as well. Spark’s scheduler sumbits a job to compute all needed RDDs. That job wil have one or more stages, which are parallel waves of computation composed of tasks. Each stage will correspond one or more RDDs in the DAG. A single stage can correspond to mutiple RDDs due to pipelining. - Tasks are scheduler and executed on a cluster
Stages are porcessed in order, with individual launching to compute segments of the RDDs. Once the final stage is finished in a job, the action is complete.
filter() and coalesce()
Memory Management
- RDD storage
When you call persist() and cache() on an RDD, its partitions will be stored in memory buffers. Spark will limit the amount of memory used when caching to a certain fraction of the JVM’s overall heap, set by spark.storage.memoryFraction. If this limit is exceeded, older partitions will be dropped from memory. - Shuffle and aggregation buffers
When performing shuffle operations, Spark will create intermediate buffers for storing shuffle output data. spark.shuffle.memeoryFraction - User Code
User code has access to everything lefet by JVM heap after the space for RDD storage and shuffle storage are allocated.
By default, Spark will leave 60% of space for RDD storage, 20% for shuffle memory, and the remaining 20% for user programs.
Hardware Provisioning
hardware provisioning
spark.cores.max