Cloudera配置问题

最新推荐文章于 2024-05-07 09:32:48 发布

你冻了我的冬天

最新推荐文章于 2024-05-07 09:32:48 发布

阅读量1.4k

点赞数 2

分类专栏： Cloudera hadoop bug解决文章标签： bug cloudera

本文链接：https://blog.csdn.net/u012071918/article/details/76828053

版权

bug解决同时被 2 个专栏收录

5 篇文章 0 订阅

订阅专栏

Cloudera hadoop

2 篇文章 0 订阅

订阅专栏

很明显这个问题是因为yarn设置的相关内存不足，有过Apache Hadoop配置经验的人都会去改yarn-site.xml的配置文件，但是对于Cloudera来说它提供的配置文件分在好多目录，让你很迷惑。其实Cloudera是通过网页Cloudera Manager 来配置这些参数的。

Memory errors

Configuring YARN after executor memory errors

The following error can occur when the required executor memory is above the maximum threshold:

Caused by: com.spss.mapreduce.exceptions.JobException:
    java.lang.IllegalArgumentException: Required executor memory (1024+384  MB) is above the max
    threshold (1024 MB) of this cluster! Please  increase the value of
    'yarn.scheduler.maximum-allocation-mb'.

For Cloudera

Increase the yarn.nodemanager.resource.memory-mb to 8GB
- In the Cloudera Manager user interface, go to Yarn service > Configurations > Search Container Memory and increase the value to 8GB.
In the Cloudera Manager user interface, go to YARN service > Quick Links and select Dynamic Resource Pools.
Under Configuration, click edit for each of the available pools, and under YARN set the Max Running Apps value to 4.
Restart YARN, Spark, and the Analytic Server service.