第5章 YARN:资源调度平台
5.2 YARN参数解读与调优
yarn-site.xml文件默认参数:
http://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
5.2.1 ResourceManager相关配置参数
参数 | 默认值 | 说明 |
---|---|---|
yarn.resourcemanager.address | ${yarn.resourcemanager.hostname}:8032 | ResourceManager 对客户端暴露的地址 |
yarn.resourcemanager.scheduler.address | ${yarn.resourcemanager.hostname}:8030 | ResourceManager 对ApplicationMaster暴露的访问地址2 |
yarn.resourcemanager.resource-tracker.address | ${yarn.resourcemanager.hostname}:8031 | ResourceManager 对NodeManager暴露的地址 |
yarn.resourcemanager.admin.address | ${yarn.resourcemanager.hostname}:8033 | ResourceManager 对管理员暴露的访问地址 |
yarn.resourcemanager.webapp.address | ${yarn.resourcemanager.hostname}:8088 | ResourceManager对外WebUI地址 |
yarn.resourcemanager.scheduler.class | ..capacity.CapacityScheduler | 启用的资源调度器主类,目前可用的有FIFO、Capacity Scheduler和Fair Scheduler |
yarn.resourcemanager.resource-tracker.client.thread-count | 50 | 处理来自NodeManager的RPC请求的Handler数目 |
yarn.resourcemanager.scheduler.client.thread-count | 50 | 处理来自ApplicationMaster的RPC请求的Handler数目 |
yarn.scheduler.minimum-allocation-mb | 1024 | 单个可申请的最小内存资源量 |
yarn.scheduler.maximum-allocation-mb | 8192 | 单个可申请的最大内存资源量 |
yarn.scheduler.minimum-allocation-vcores | 1 | 单个可申请的最小虚拟CPU个数 |
yarn.scheduler.maximum-allocation-vcores | 32 | 单个可申请的最大虚拟CPU个数 |
yarn.resourcemanager.nodemanagers.heartbeat-interval-ms | 1000(毫秒) | NodeManager心跳间隔 |
..capacity.CapacityScheduler的完整名称: org.apache.Hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
5.2.2 NodeManager相关配置参数
参数 | 默认值 | 说明 |
---|---|---|
yarn.nodemanager.resource.memory-mb | 8192 | NodeManager总的可用物理内存(这个值通过一定要配置) |
yarn.nodemanager.vmem-pmem-ratio | 2.1 | 每使用1MB物理内存,最多可用的虚拟内存数 |
yarn.nodemanager.resource.cpu-vcores | 8 | NodeManager总的可用虚拟CPU个数 |
yarn.nodemanager.local-dirs | ${hadoop.tmp.dir}/nm-local-dir | 中间结果存放位置,这个参数通常会配置多个目录,已分摊磁盘IO负载。 |
yarn.nodemanager.log-dirs | ${yarn.log.dir}/userlogs | 日志存放地址(可配置多个目录) |
yarn.nodemanager.log.retain-seconds | 10800 | NodeManager上日志最多存放时间(不启用日志聚集功能时有效) |
yarn.nodemanager.aux-services | NodeManager上运行的附属服务,需配置成mapreduce_shuffle,才可运行MapReduce程序 |
5.2.3 mapred-site.xml
参数 | 默认值 | 说明 |
---|---|---|
mapreduce.job.reduces | 1 | 默认启动的reduce数 |
mapreduce.job.maps | 2 | 默认启动的map数 |
mapreduce.task.io.sort.factor | 10 | Reduce Task中合并小文件时,一次合并的文件数据 |
mapreduce.task.io.sort.mb | 100 | Map Task缓冲区所占内存大小 |
mapred.child.java.opts | -Xmx200m | jvm启动的子线程可以使用的最大内存 |
mapreduce.jobtracker.handler.count | 10 | JobTracker可以启动的线程数,一般为tasktracker节点的4% |
mapreduce.reduce.shuffle.parallelcopies | 5 | reuduce shuffle阶段并行传输数据的数量 |
mapreduce.tasktracker.http.threads | 40 | map和reduce是通过http进行数据传输的,这个是设置传输的并行线程数 |
mapreduce.map.output.compress | false | map输出是否进行压缩,如果压缩就会多耗cpu,但是减少传输时间,如果不压缩,就需要较多的传输带宽 |
mapreduce.reduce.shuffle.merge.percent | 0.66 | reduce归并接收map的输出数据可占用的内存配置百分比 |
mapreduce.reduce.shuffle.memory.limit.percent | 0.25 | 一个单一的shuffle的最大内存使用限制。 |
mapreduce.jobtracker.handler.count | 10 | 可并发处理来自tasktracker的RPC请求数,默认值10。 |
mapreduce.job.jvm.numtasks | 1 | 一个jvm可连续启动多个同类型任务,默认值1,若为-1表示不受限制。 |
mapreduce.tasktracker.reduce.tasks.maximum | 2 | 一个tasktracker并发执行的reduce数,建议为cpu核数 |
5.2.4 参数调优
参照
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/index.html
单击“Command Line Installation”超链接后的PDF图标,即可打开HDP安装文档。
单击“1.10. Determining HDP Memory Configuration Settings”条目,跳转到对应页面
单击“Download Companion Files”连接,可以看到两条命令
-
wget http:
//public-repo-
1.
hortonworks
.com
/HDP/tools/
2.6
.0
.3
/ hdp_manual_install_rpm_helper_files-
2.6
.0
.3
.8
.tar
.gz
-
-
tar zxvf hdp_manual_install_rpm_helper_files-
2.6
.0
.3
.8
.tar
.gz
- 1
- 2
- 3
下载:
-
[
root@node1 ~]
# wget http://public-repo-1.hortonworks.com/HDP/tools/2.6.0.3/hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
-
-
-
2017
-
05
-
23
23:
26:
13-- http:
//public-repo-
1.
hortonworks.com/HDP/tools/
2.6
.0
.3
/hdp_manual_install_rpm_helper_files-
2.6
.0
.3
.8
.tar.gz
-
Resolving
public-repo
-
1.hortonworks.com (
public-repo
-
1.hortonworks.com)
...
52.84
.167
.222,
52.84
.167
.38,
52.84
.167
.49,
...
-
Connecting to
public-repo
-
1.hortonworks.com (
public-repo
-
1.hortonworks.com)|
52.84
.167
.222|:
80... connected.
-
HTTP request sent, awaiting response...
200 OK
-
Length:
85173 (
83K) [application/x-tar]
-
Saving to: ‘hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8.tar.gz’
-
-
100%[==================================================================================================================================>]
85,
173
132KB/s
in
0.
6s
-
-
2017
-
05
-
23
23:
26:
14 (
132 KB/s) - ‘hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8.tar.gz’ saved [
85173/
85173]
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
解压缩
-
[
root@node1 ~]
# tar -zxvf hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/
-
.....
-
2.6
.0
.3
.8/configuration_files/zookeeper/configuration
.xsl
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/zookeeper/log4j
.properties
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/zookeeper/zoo
.cfg
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/pig/
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/pig/pig-env
.sh
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/pig/pig
.properties
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/pig/log4j
.properties
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/oozie/
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/oozie/oozie-log4j
.properties
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/oozie/oozie-env
.sh
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/oozie/oozie-site
.xml
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/oozie/adminusers
.txt
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/oozie/hadoop-config
.xml
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/oozie/oozie-
default
.xml
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/hbase-policy
.xml
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/regionservers
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/hadoop-metrics
.properties
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/hbase-env
.sh
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/hadoop-metrics
.properties
.master-GANGLIA
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/hadoop-metrics
.properties
.regionservers-GANGLIA
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/hbase-site
.xml
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/log4j
.properties
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/webhcat/
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/webhcat/webhcat-env
.sh
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/webhcat/webhcat-site
.xml
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/scripts/
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/scripts/directories
.sh
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/scripts/yarn-utils
.py
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/scripts/usersAndGroups
.sh
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/readme
.txt
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/HDP-CHANGES
.txt
-
[
root@node1 ~]
#
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
-
[root@node1 ~]
# cd hdp_manual_install_rpm_helper_files-2.6.0.3.8
-
[root@node1 hdp_manual_install_rpm_helper_files-
2.6
.0
.3
.8]
# ls
-
configuration_files HDP-CHANGES
.txt readme
.txt scripts
-
[root@nb0 hdp_manual_install_rpm_helper_files-
2.6
.0
.3
.8]
# cd scripts/
-
[root@nb0 scripts]
# ls
-
directories
.sh usersAndGroups
.sh yarn-utils
.py
- 1
- 2
- 3
- 4
- 5
- 6
假设,某个节点有4核8G内存1块硬盘,该节点同时安装HBase,通过下面命令即可获得优化参数
-
[root@nb0 scripts]
# python yarn-utils.py -c 4 -m 8 -d 1 -k True
-
Using cores=
4 memory=
8GB disks=
1 hbase=
True
-
Profile: cores=
4 memory=
5120MB reserved=
3GB usableMem=
5GB disks=
1
-
Num
Container=
3
-
Container
Ram=
1536MB
-
Used
Ram=
4GB
-
Unused
Ram=
3GB
-
yarn
.scheduler
.minimum-allocation-mb=
1536
-
yarn
.scheduler
.maximum-allocation-mb=
4608
-
yarn
.nodemanager
.resource
.memory-mb=
4608
-
mapreduce
.map
.memory
.mb=
1536
-
mapreduce
.map
.java
.opts=-
Xmx1228m
-
mapreduce
.reduce
.memory
.mb=
3072
-
mapreduce
.reduce
.java
.opts=-
Xmx2457m
-
yarn
.app
.mapreduce
.am
.resource
.mb=
3072
-
yarn
.app
.mapreduce
.am
.command-opts=-
Xmx2457m
-
mapreduce
.task
.io
.sort
.mb=
614
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
选项说明
选项 | 描述 |
---|---|
-c | 每一个客户机的核数目 |
-m | 每一个客户机拥有的内存总数 |
-d | 每一个客户机拥有的磁盘数目 |
-k | 如果Hbase安装了为”True”,否则为”False” |