在YARN中，如何控制和监控map/reduce的并发数

最新推荐文章于 2022-10-10 16:07:22 发布

转载最新推荐文章于 2022-10-10 16:07:22 发布 · 405 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：https://yq.aliyun.com/articles/43501

文章标签：

#大数据 #ui

本文介绍了YARN中任务槽位的变化，从MapReduce 1的TaskTracker到YARN的NodeManager，并解释了如何通过配置内存和CPU核心来控制节点资源分配。此外，还提供了NameNode和ResourceManager的Web UI地址，以及Job History Server和集群利用率测量工具如Ambari和Ganglia的使用方法。

配置建议：

1.
In MR1, the mapred.tasktracker.map.tasks.maximum and mapred.tasktracker.reduce.tasks.maximum properties dictated how many map and reduce slots each TaskTracker had.

These properties no longer exist in YARN. Instead, YARN uses yarn.nodemanager.resource.memory-mb and yarn.nodemanager.resource.cpu-vcores, which control the amount of memory and CPU on each node, both available to both maps and reduces

Essentially:
YARN has no TaskTrackers, but just generic NodeManagers. Hence, there's no more Map slots and Reduce slots separation. Everything depends on the amount of memory in use/demanded

Using the web UI you can get lot of monitoring/admin kind of info:

NameNode - http://:50070/
Resource Manager - http://:8088/

其他配置参考：

There is a good guide on YARN configuration from Hortonworks
You may analyze your job in Job History server. It usually may be found on port 19888. Ambari andGanglia are also very good for cluster utilization measurement.