hadoop 集群常见问题

1

Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=1536, maxMemory=1024

 conf.set("yarn.scheduler.maximum-allocation-mb","200");
 conf.set("yarn.app.mapreduce.am.resource.mb","200");
 conf.set("yarn.nodemanager.resource.memory-mb","200");

参考文献:
https://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits

2

Container [pid=1623,containerID=container_1553048494869_0002_02_000001] is running beyond virtual memory limits. Current usage: 82.7 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used. Killing container.

参考文献:
https://stackoverflow.com/questions/14110428/am-container-is-running-beyond-virtual-memory-limits

Failing this attempt.Diagnostics: Container [pid=5044,containerID=container_1572932711593_0002_02_000001] is running beyond virtual memory limits. Current usage: 115.0 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used. Killing container.

虚拟内存不足

在etc/Hadoop/yarn-site.xml文件中，修改检查虚拟内存的属性为false，如下：

<property> 
    <name>yarn.nodemanager.vmem-check-enabled</name> 
    <value>false</value> 
</property>

参考:
https://www.cnblogs.com/hxsyl/p/6503010.html

Exception in thread "main" java.io.IOException: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=1536, maxMemory=1024

内存问题:
指定内存容量

conf.set("yarn.app.mapreduce.am.resource.mb", "500");// 指定资源分配器