Determine Hadoop Memory Configuration Settings

本文中涉及到的脚本来自hortonworks，通过它可以计算出针对yarn和MapReduce的内存设置。
脚本名称：yarn-utils.py，参数如下图所示：

[hadoop@hdp01 ~]$ wget http://public-repo-1.hortonworks.com/HDP/tools/2.3.4.0/hdp_manual_install_rpm_helper_files-2.3.4.0.3485.tar.gz
[hadoop@hdp01 ~]$ tar -xzf hdp_manual_install_rpm_helper_files-2.3.4.0.3485.tar.gz
[hadoop@hdp01 ~]$ mv hdp_manual_install_rpm_helper_files-2.3.4.0.3485 hdp
[hadoop@hdp01 ~]$ cd hdp/scripts
[hadoop@hdp01 scripts]$  python yarn-utils.py -c 4 -m 16 -d 2 -k True
 Using cores=4 memory=16GB disks=2 hbase=True
 Profile: cores=4 memory=12288MB reserved=4GB usableMem=12GB disks=2
 Num Container=4
 Container Ram=3072MB
 Used Ram=12GB
 Unused Ram=4GB
 yarn.scheduler.minimum-allocation-mb=3072
 yarn.scheduler.maximum-allocation-mb=12288
 yarn.nodemanager.resource.memory-mb=12288
 mapreduce.map.memory.mb=3072
 mapreduce.map.java.opts=-Xmx2457m
 mapreduce.reduce.memory.mb=3072
 mapreduce.reduce.java.opts=-Xmx2457m
 yarn.app.mapreduce.am.resource.mb=3072
 yarn.app.mapreduce.am.command-opts=-Xmx2457m
 mapreduce.task.io.sort.mb=1228

以上参数的位置如下图所示：

参考文献：
1、Determine HDP Memory Configuration Settings
2、Demystify Apache Tez Memory Tuning - Step by Step