第5章 YARN：资源调度平台
5.2 YARN参数解读与调优
yarn-site.xml文件默认参数：
http://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
5.2.1 ResourceManager相关配置参数
| 参数 | 默认值 | 说明 |
|---|---|---|
| yarn.resourcemanager.address | ${yarn.resourcemanager.hostname}:8032 | ResourceManager 对客户端暴露的地址 |
| yarn.resourcemanager.scheduler.address | ${yarn.resourcemanager.hostname}:8030 | ResourceManager 对ApplicationMaster暴露的访问地址2 |
| yarn.resourcemanager.resource-tracker.address | ${yarn.resourcemanager.hostname}:8031 | ResourceManager 对NodeManager暴露的地址 |
| yarn.resourcemanager.admin.address | ${yarn.resourcemanager.hostname}:8033 | ResourceManager 对管理员暴露的访问地址 |
| yarn.resourcemanager.webapp.address | ${yarn.resourcemanager.hostname}:8088 | ResourceManager对外WebUI地址 |
| yarn.resourcemanager.scheduler.class | ..capacity.CapacityScheduler | 启用的资源调度器主类，目前可用的有FIFO、Capacity Scheduler和Fair Scheduler |
| yarn.resourcemanager.resource-tracker.client.thread-count | 50 | 处理来自NodeManager的RPC请求的Handler数目 |
| yarn.resourcemanager.scheduler.client.thread-count | 50 | 处理来自ApplicationMaster的RPC请求的Handler数目 |
| yarn.scheduler.minimum-allocation-mb | 1024 | 单个可申请的最小内存资源量 |
| yarn.scheduler.maximum-allocation-mb | 8192 | 单个可申请的最大内存资源量 |
| yarn.scheduler.minimum-allocation-vcores | 1 | 单个可申请的最小虚拟CPU个数 |
| yarn.scheduler.maximum-allocation-vcores | 32 | 单个可申请的最大虚拟CPU个数 |
| yarn.resourcemanager.nodemanagers.heartbeat-interval-ms | 1000（毫秒） | NodeManager心跳间隔 |
..capacity.CapacityScheduler的完整名称： org.apache.Hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
5.2.2 NodeManager相关配置参数
| 参数 | 默认值 | 说明 |
|---|---|---|
| yarn.nodemanager.resource.memory-mb | 8192 | NodeManager总的可用物理内存（这个值通过一定要配置） |
| yarn.nodemanager.vmem-pmem-ratio | 2.1 | 每使用1MB物理内存，最多可用的虚拟内存数 |
| yarn.nodemanager.resource.cpu-vcores | 8 | NodeManager总的可用虚拟CPU个数 |
| yarn.nodemanager.local-dirs | ${hadoop.tmp.dir}/nm-local-dir | 中间结果存放位置，这个参数通常会配置多个目录，已分摊磁盘IO负载。 |
| yarn.nodemanager.log-dirs | ${yarn.log.dir}/userlogs | 日志存放地址（可配置多个目录） |
| yarn.nodemanager.log.retain-seconds | 10800 | NodeManager上日志最多存放时间（不启用日志聚集功能时有效） |
| yarn.nodemanager.aux-services | NodeManager上运行的附属服务,需配置成mapreduce_shuffle，才可运行MapReduce程序 |
5.2.3 mapred-site.xml
| 参数 | 默认值 | 说明 |
|---|---|---|
| mapreduce.job.reduces | 1 | 默认启动的reduce数 |
| mapreduce.job.maps | 2 | 默认启动的map数 |
| mapreduce.task.io.sort.factor | 10 | Reduce Task中合并小文件时，一次合并的文件数据 |
| mapreduce.task.io.sort.mb | 100 | Map Task缓冲区所占内存大小 |
| mapred.child.java.opts | -Xmx200m | jvm启动的子线程可以使用的最大内存 |
| mapreduce.jobtracker.handler.count | 10 | JobTracker可以启动的线程数，一般为tasktracker节点的4% |
| mapreduce.reduce.shuffle.parallelcopies | 5 | reuduce shuffle阶段并行传输数据的数量 |
| mapreduce.tasktracker.http.threads | 40 | map和reduce是通过http进行数据传输的，这个是设置传输的并行线程数 |
| mapreduce.map.output.compress | false | map输出是否进行压缩，如果压缩就会多耗cpu，但是减少传输时间，如果不压缩，就需要较多的传输带宽 |
| mapreduce.reduce.shuffle.merge.percent | 0.66 | reduce归并接收map的输出数据可占用的内存配置百分比 |
| mapreduce.reduce.shuffle.memory.limit.percent | 0.25 | 一个单一的shuffle的最大内存使用限制。 |
| mapreduce.jobtracker.handler.count | 10 | 可并发处理来自tasktracker的RPC请求数，默认值10。 |
| mapreduce.job.jvm.numtasks | 1 | 一个jvm可连续启动多个同类型任务，默认值1，若为-1表示不受限制。 |
| mapreduce.tasktracker.reduce.tasks.maximum | 2 | 一个tasktracker并发执行的reduce数，建议为cpu核数 |
5.2.4 参数调优
参照
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/index.html
单击“Command Line Installation”超链接后的PDF图标，即可打开HDP安装文档。
单击“1.10. Determining HDP Memory Configuration Settings”条目，跳转到对应页面
单击“Download Companion Files”连接，可以看到两条命令
-
wget http:
//public-repo-
1.
hortonworks
.com
/HDP/tools/
2.6
.0
.3
/ hdp_manual_install_rpm_helper_files-
2.6
.0
.3
.8
.tar
.gz
-
-
tar zxvf hdp_manual_install_rpm_helper_files-
2.6
.0
.3
.8
.tar
.gz
- 1
- 2
- 3
下载：
-
[
root@node1 ~]
# wget http://public-repo-1.hortonworks.com/HDP/tools/2.6.0.3/hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
-
-
-
2017
-
05
-
23
23:
26:
13-- http:
//public-repo-
1.
hortonworks.com/HDP/tools/
2.6
.0
.3
/hdp_manual_install_rpm_helper_files-
2.6
.0
.3
.8
.tar.gz
-
Resolving
public-repo
-
1.hortonworks.com (
public-repo
-
1.hortonworks.com)
...
52.84
.167
.222,
52.84
.167
.38,
52.84
.167
.49,
...
-
Connecting to
public-repo
-
1.hortonworks.com (
public-repo
-
1.hortonworks.com)|
52.84
.167
.222|:
80... connected.
-
HTTP request sent, awaiting response...
200 OK
-
Length:
85173 (
83K) [application/x-tar]
-
Saving to: ‘hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8.tar.gz’
-
-
100%[==================================================================================================================================>]
85,
173
132KB/s
in
0.
6s
-
-
2017
-
05
-
23
23:
26:
14 (
132 KB/s) - ‘hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8.tar.gz’ saved [
85173/
85173]
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
解压缩
-
[
root@node1 ~]
# tar -zxvf hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/
-
.....
-
2.6
.0
.3
.8/configuration_files/zookeeper/configuration
.xsl
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/zookeeper/log4j
.properties
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/zookeeper/zoo
.cfg
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/pig/
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/pig/pig-env
.sh
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/pig/pig
.properties
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/pig/log4j
.properties
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/oozie/
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/oozie/oozie-log4j
.properties
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/oozie/oozie-env
.sh
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/oozie/oozie-site
.xml
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/oozie/adminusers
.txt
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/oozie/hadoop-config
.xml
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/oozie/oozie-
default
.xml
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/hbase-policy
.xml
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/regionservers
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/hadoop-metrics
.properties
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/hbase-env
.sh
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/hadoop-metrics
.properties
.master-GANGLIA
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/hadoop-metrics
.properties
.regionservers-GANGLIA
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/hbase-site
.xml
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/hbase/log4j
.properties
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/webhcat/
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/webhcat/webhcat-env
.sh
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/configuration_files/webhcat/webhcat-site
.xml
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/scripts/
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/scripts/directories
.sh
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/scripts/yarn-utils
.py
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/scripts/usersAndGroups
.sh
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/readme
.txt
-
hdp_manual_install_rpm_helper_files
-
2.6
.0
.3
.8/HDP-CHANGES
.txt
-
[
root@node1 ~]
#
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
-
[root@node1 ~]
# cd hdp_manual_install_rpm_helper_files-2.6.0.3.8
-
[root@node1 hdp_manual_install_rpm_helper_files-
2.6
.0
.3
.8]
# ls
-
configuration_files HDP-CHANGES
.txt readme
.txt scripts
-
[root@nb0 hdp_manual_install_rpm_helper_files-
2.6
.0
.3
.8]
# cd scripts/
-
[root@nb0 scripts]
# ls
-
directories
.sh usersAndGroups
.sh yarn-utils
.py
- 1
- 2
- 3
- 4
- 5
- 6
假设，某个节点有4核8G内存1块硬盘，该节点同时安装HBase，通过下面命令即可获得优化参数
-
[root@nb0 scripts]
# python yarn-utils.py -c 4 -m 8 -d 1 -k True
-
Using cores=
4 memory=
8GB disks=
1 hbase=
True
-
Profile: cores=
4 memory=
5120MB reserved=
3GB usableMem=
5GB disks=
1
-
Num
Container=
3
-
Container
Ram=
1536MB
-
Used
Ram=
4GB
-
Unused
Ram=
3GB
-
yarn
.scheduler
.minimum-allocation-mb=
1536
-
yarn
.scheduler
.maximum-allocation-mb=
4608
-
yarn
.nodemanager
.resource
.memory-mb=
4608
-
mapreduce
.map
.memory
.mb=
1536
-
mapreduce
.map
.java
.opts=-
Xmx1228m
-
mapreduce
.reduce
.memory
.mb=
3072
-
mapreduce
.reduce
.java
.opts=-
Xmx2457m
-
yarn
.app
.mapreduce
.am
.resource
.mb=
3072
-
yarn
.app
.mapreduce
.am
.command-opts=-
Xmx2457m
-
mapreduce
.task
.io
.sort
.mb=
614
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
选项说明
| 选项 | 描述 |
|---|---|
| -c | 每一个客户机的核数目 |
| -m | 每一个客户机拥有的内存总数 |
| -d | 每一个客户机拥有的磁盘数目 |
| -k | 如果Hbase安装了为”True”，否则为”False” |
1680
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
