配置Hadoop运行在小内存主机上

最新推荐文章于 2022-01-20 21:52:26 发布

穷目楼

最新推荐文章于 2022-01-20 21:52:26 发布

阅读量1.8k

点赞数 2

分类专栏：大数据文章标签： Hadoop Yarn

本文链接：https://blog.csdn.net/skyupward/article/details/103641962

版权

本文介绍了如何调整Hadoop基于Yarn在小内存主机上的配置，以解决由于资源不足导致的任务失败问题。通过降低AM和mapred的内存请求，使得在2G内存的主机上能成功执行简单的grep任务。

摘要由CSDN通过智能技术生成

想把Hadoop基于Yarn部署到小内存主机上，默认的配置会由于资源不足导致简单的任务也无法执行成功。

2019-12-02 14:28:33,989 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1575266850654_0002
2019-12-02 14:28:33,989 INFO mapreduce.JobSubmitter: Executing with tokens: []
2019-12-02 14:28:34,316 INFO conf.Configuration: resource-types.xml not found
2019-12-02 14:28:34,317 INFO resource.ResourceUtils: Unable to find ‘resource-types.xml’.
2019-12-02 14:28:34,393 INFO impl.YarnClientImpl: Submitted application application_1575266850654_0002
2019-12-02 14:28:34,470 INFO mapreduce.Job: The url to track the job: http://vm-0-2-centos:8088/proxy/application_1575266850654_0002/
2019-12-02 14:28:34,471 INFO mapreduce.Job: Running job: job_1575266850654_0002
2019-12-02 14:28:43,766 INFO mapreduce.Job: Job job_1575266850654_0002 running in uber mode : false
2019-12-02 14:28:43,767 INFO mapreduce.Job: map 0% reduce 0%
2019-12-02 14:30:23,061 INFO mapreduce.Job: map 26% reduce 0%
2019-12-02 14:30:24,107 INFO mapreduce.Job: map 44% reduce 0%
2019-12-02 14:38:58,133 INFO mapreduce.Job: Task Id : attempt_1575266850654_0002_m_000001_0, Status : FAILED
AttemptID:attempt_1575266850654_0002_m_000001_0 Timed out after 600 secs
2019-12-02 14:38:58,148 INFO mapreduce.Job: Task Id : attempt_1575266850654_0002_m_000002_0, Status : FAILED

有时候还会发现Hadoop的常驻进程自动挂了，通过journal -xe查看日志可以看到由于资源的原因，被内核强制停了。

Dec 02 14:30:23 vm-0-2-centos kernel: Out of memory: Kill process 16044 (java) score 94 or sacrifice child
Dec 02 14:30:23 vm-0-2-centos kernel: Killed process 16044 (java), UID 1000, total-vm:2440728kB, anon-rss:174572kB, file-rss:2148kB, shme

导致Yarn无法正常执行任务的主要资源限制是在内存上，所以可以调整限制Yarn所能使用的物理内存