Spark Trouble Shooting and Performance Tuning

- spark master server - more memory 

export SPARK_DAEMON_MEMORY=5g

spark.ui.retainedJobs 500   # 默认都是1000
spark.ui.retainedStages 500

spark.history.retainedApplications application in memory

in spark-env.sh

- Runtime error -> add more memory and CPU to executor

• missing output location
• failed to connect to host

- executor lost, task lost and timeout -> increase spark.network.timeout

==========================================

- idle executor -> more partitions and lower below values

spark.locality.wait
spark.locality.wait.process
spark.locality.wait.node
spark.locality.wait.rack

- spark task keep failing 

spark.scheduler.executorTaskBlacklistTime 30000

blacklist executor for 30s  

- spark higher API (DataFrame, DataSet, Spark SQL) over lower API

higher API is optimized by catalyst and tungsten

- balanced yarn resource to let executor run multiple tasks

- proper partition size

- avoid too many shuffle data