CDH SparkOnYarn执行中executor内存限制问题

因为配置了spark如下参数，启动spark-shell报错

export SPARK_WORKER_CORES=5

export SPARK_WORKER_INSTANCES=4

export SPARK_WORKER_MEMORY=50G

export SPARK_WORKER_WEBUI_PORT=8081

export SPARK_EXECUTOR_CORES=2

export SPARK_EXECUTOR_MEMORY=20G

 

 

15/07/07 15:52:47 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)

java.lang.IllegalArgumentException: Required executor memory (20480+1433 MB) is above the max threshold (8192 MB) of this cluster!