Spark Standalone spark-default.conf

Example:

spark.master spark://master:7077

spark.eventLog.enabled true

spark.eventLog.dir hdfs://namenode:8021/directory

spark.serializer org.apache.spark.serializer.KryoSerializer

spark.driver.memory 5g

spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

spark.eventLog.enabled true
spark.eventLog.dir hdfs://flashHadoop/spark/log
spark.eventLog.compress true

spark.local.dir=/data1/data/spark/local

spark.ui.enabled true
spark.ui.killEnabled false
spark.ui.port 18080

spark.history.ui.port 18080
spark.history.fs.cleaner.enabled true
spark.history.fs.logDirectory hdfs://flashHadoop/spark/log
spark.history.fs.cleaner.interval 5d
spark.history.fs.cleaner.maxAge 8d

2018.12.17 Turning for driver and exec memory

spark.driver.supervise true
spark.driver.cores 2
spark.driver.memory 3g

spark.executor.cores 2
spark.executor.memory 3g

spark.driver.extraJavaOption -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:MaxGCPauseMillis=100 -XX:-ResizePLAB -XX
:+ParallelRefProcEnabled -XX:+AlwaysPreTouch -XX:ParallelGCThreads=16 -XX:ConcGCThreads=8 -XX:G1HeapWastePercent=3 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1MixedGCLiveThresholdPer
cent=85 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy

spark.executor.extraJavaOptions -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:MaxGCPauseMillis=100 -XX:-ResizePLAB -XX
:+ParallelRefProcEnabled -XX:+AlwaysPreTouch -XX:ParallelGCThreads=16 -XX:ConcGCThreads=8 -XX:G1HeapWastePercent=3 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1MixedGCLiveThresholdPer
cent=85 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy

Spark Job Submit → Read default conf,,,, 调优后。

转载于:https://www.cnblogs.com/hit-zb/p/10401737.html