假设一个 n 个主机的 YARN 集群,每个主机具有 c 个内核和 m GB 内存。
yarn
yarn.nodemanager.resource.memory-mb = m-0.5*n
yarn.nodemanager.resource.cpu-vcores = c-4
yarn.scheduler.maximum-allocation-mb>(spark.yarn.executor.memoryOverhead+spark.executor.memory )
spark
spark.executor.cores =4、5或6
spark.executor.instances = n*spark.executor.cores
spark.executor.num= yarn.nodemanager.resource.memory-mb/spark.executor.cores 可以整除
spark.executor.memory=yarn.nodemanager.resource.memory-mb/yarn.nodemanager.resource.cpu-vcores/spark.executor.num(<=64G)
spark.yarn.executor.memoryOverhead=executorMemory * 0.10 (5-10%)
spark.driver.memory+spark.yarn.driver.memoryOverhead= 12%*
yarn.nodemanager.resource.memory-mb
yarn.nodemanager.resource.memory-mb大于50 GB时为12 GB
yarn.nodemanager.resource.memory-mb在12 GB到50 GB之间时为4