Hadoop调优手册(1)——HDFS核心参数

• 以下配置均针对Hadoop3.x系列

---

1.NameNode、DataNode内存配置

* 配置文件

hadoop-env.sh

* 配置参数

由hadoop-env.sh文件中的描述可知：
（1）守护进程将在 _OPT参数中选择 Xmx设置作为Hadoop内存大小。
（2）Hadoop的内存默认是动态分配的：若在 _OPT参数中没有设置 Xmx，则JVM会根据服务器内存大小，自动为Hadoop分配内存。

# The maximum amount of heap to use (Java -Xmx).  If no unit
# is provided, it will be converted to MB.  Daemons will
# prefer any Xmx setting in their respective _OPT variable.
# There is no default; the JVM will autoscale based upon machine
# memory size.
# export HADOOP_HEAPSIZE_MAX=

# The minimum amount of heap to use (Java -Xms).  If no unit
# is provided, it will be converted to MB.  Daemons will
# prefer any Xms setting in their respective _OPT variable.
# There is no default; the JVM will autoscale based upon machine
# memory size.
# export HADOOP_HEAPSIZE_MIN=

所以，我们需要在以下两个参数中分别设置NameNode和DataNode的内存：

HDFS_NAMENODE_OPTS
HDFS_DATANODE_OPTS

继续看hadoop-env.sh文件中的描述：
NameNode 的具体参数：

###
# NameNode specific parameters
###

# Default log level and output location for file system related change
# messages. For non-namenode daemons, the Java property must be set in
# the appropriate _OPTS if one wants something other than INFO,NullAppender
# Java property: hdfs.audit.logger
# export HDFS_AUDIT_LOGGER=INFO,NullAppender

# Specify the JVM options to be used when starting the NameNode.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
# a) Set JMX options
# export