spark-shell在yarn client模式下启动报错

spark-shell在yarn client模式下启动报错

17/10/26 19:16:12 ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!
17/10/26 19:16:12 ERROR client.TransportClient: Failed to send RPC 8955925396344311292 to /192.168.0.16:39738: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(…)(Unknown Source)

查看NodeManager日志发现如下信息：

2017-10-26 19:43:21,787 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=3820,containerID=container_1509016963775_0001_02_000001] is running beyond virtual memory limits. Current usage: 339.0 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

由此可见是虚拟内存设置过小导致container被杀死，故需增加其大小。在yarn-site.xml中适量增大 yarn.nodemanager.vmem-pmem-ratio （默认值为2.1）即可。