spark-troubleshooting-OOM

java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError：GC overhead limit exceeded

driver OOM:

* 增加 spark submit –driver-memory 大小*

Executor OOM:

增加job的并行度

增加spark.executor.memory

降低 spark.storage.memoryFraction

减少shuffle reduce端task缓冲的大小
set (“spark.reducer.maxSizeInFligth”,”24)

export SPARK_JAVA_OPTS +=”-Dspark.kryoserializer.buffer.mb=10 -Dspark.cleaner.ttl=43200”