NodeManager死亡、内存溢出

1、查看nodemanager节点状态

yarn node -list

[bigdata@master logs]$ yarn node -list
18/10/17 16:01:40 INFO client.RMProxy: Connecting to ResourceManager at master/10.18.13.3:8032
Total Nodes:3
         Node-Id         Node-State    Node-Http-Address    Number-of-Running-Containers
    slave2:37193            RUNNING          slave2:8042                               0
    master:47245            RUNNING          master:8042                               0
    slave1:37263            RUNNING          slave1:8042                               0

2、停止nodemanager

sbin/yarn-daemon.sh stop nodemanager

如果内存过小，执行过程中消耗的内存过大就会导致内存溢出进程死亡；报错如下：

2018-10-16 22:15:43,968 ERROR org.apache.hadoop.util.Shell: Caught java.lang.OutOfMemoryError: unable to create new native thread. One possible reason is that ulimit setting of 'max user processes' is too low.
If so, do 'ulimit -u <largerNum>' and try again.2018-10-16 22:15:43,968 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Failed to launch container.
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:717)
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:562)
        at org.apache.hadoop.util.Shell.run(Shell.java:482)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-10-16 22:15:43,969 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1539688839298_0001_01_000147 transitioned from RUNNING to EXITED_WITH_FAILURE
2018-10-16 22:15:43,969 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1539688839298_0001_01_000147
2018-10-16 22:15:43,974 ERROR org.apache.hadoop.util.Shell: Caught java.lang.OutOfMemoryError: unable to create new native thread. One possible reason is that ulimit setting of 'max user processes' is too low.
If so, do 'ulimit -u <largerNum>' and try again.2018-10-16 22:15:43,974 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Failed to launch container.
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:717)
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:562)
        at org.apache.hadoop.util.Shell.run(Shell.java:482)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-10-16 22:15:43,982 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1539688839298_0001_000001 (auth:SIMPLE)

3、修改etc/hoadoop/yarn-site.xml  --可以根据机器的情况调大内存

<!-- 设置 NodeManager 上的可用内存量，单位MB -->
        <property>
                <name>yarn.nodemanager.resource.memory-mb</name>
                <value>20480</value>
        </property>

4、启动nodemanager

sbin/yarn-daemon.sh start nodemanager

上述命令为单独启动某一个节点的nodemanger，

5、启动所有节点的nodemanager

sbin/yarn-daemons.sh start nodemanager

6、或者修改之前执行stop-yarn.sh  修改完后再启动

 

7、etc/hoadoop/yarn-site.xml给定每个节点的内存的时候还可以给定cpu

<property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
    <value>10</value>
</property>