Container marked as failed报错排查方案

解决这个问题的第一步是，不要直接照搬网上任何一篇博客的解决方案.

完整报错如下:

2020-05-02 16:37:37 INFO  RMProxy:98 - Connecting to ResourceManager at Desktop/192.168.0.103:8030
2020-05-02 16:37:37 INFO  YarnRMClient:54 - Registering the ApplicationMaster
2020-05-02 16:37:38 INFO  YarnAllocator:54 - Will request 2 executor container(s), each with 1 core(s) and 1408 MB memory (including 384 MB of overhead)
2020-05-02 16:37:38 INFO  YarnAllocator:54 - Submitted 2 unlocalized container requests.
2020-05-02 16:37:38 INFO  ApplicationMaster:54 - Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
2020-05-02 16:37:39 INFO  AMRMClientImpl:360 - Received new token for : Desktop:45307
2020-05-02 16:37:39 INFO  AMRMClientImpl:360 - Received new token for : Laptop:46733
2020-05-02 16:37:39 INFO  YarnAllocator:54 - Launching container container_1588395311466_0001_01_000002 on host Desktop for executor with ID 1
2020-05-02 16:37:39 INFO  YarnAllocator:54 - Launching container container_1588395311466_0001_01_000003 on host Laptop for executor with ID 2
2020-05-02 16:37:39 INFO  YarnAllocator:54 - Received 2 containers from YARN, launching executors on 2 of them.
2020-05-02 16:37:39 INFO  ContainerManagementProtocolProxy:81 - yarn.client.max-cached-nodemanagers-proxies : 0
2020-05-02 16:37:39 INFO  ContainerManagementProtocolProxy:81 - yarn.client.max-cached-nodemanagers-proxies : 0
2020-05-02 16:37:39 INFO  ContainerManagementProtocolProxy:260 - Opening proxy : Laptop:46733
2020-05-02 16:37:39 INFO  ContainerManagementProtocolProxy:260 - Opening proxy : Desktop:45307
2020-05-02 16:38:09 INFO  YarnAllocator:54 - Completed container container_1588395311466_0001_01_000003 on host: Laptop (state: COMPLETE, exit status: 1)
2020-05-02 16:38:09 WARN  YarnAllocator:66 - Container marked as failed: container_1588395311466_0001_01_000003 on host: Laptop. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1588395311466_0001_01_000003
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:585)
	at org.apache.hadoop.util.Shell.run(Shell.java:482)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)


Container exited with a non-zero exit code 1

2020-05-02 16:38:12 INFO  YarnAllocator:54 - Will request 1 executor container(s), each with 1 core(s) and 1408 MB memory (including 384 MB of overhead)
2020-05-02 16:38:12 INFO  YarnAllocator:54 - Submitted 1 unlocalized container requests.
2020-05-02 16:38:13 INFO  YarnAllocator:54 - Launching container container_1588395311466_0001_01_000004 on host Desktop for executor with ID 3
2020-05-02 16:38:13 INFO  YarnAllocator:54 - Received 1 containers from YARN, launching executors on 1 of them.
2020-05-02 16:38:13 INFO  ContainerManagementProtocolProxy:81 - yarn.client.max-cached-nodemanagers-proxies : 0
2020-05-02 16:38:13 INFO  ContainerManagementProtocolProxy:260 - Opening proxy : Desktop:45307

看见上面的红色字体没1408MB？

cd $HADOOP_HOME/etc/hadoop

修改yarn-site.xml

<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1500</value>
</property>
 

这个value里面的数值大于1408MB即可

重启hadoop，重新输入:

pyspark --master yarn

再次前往Desktop:8088查看log，报错消失

 

上述修改的依据是[1]

 

Reference:

[1]spark-submit提交参数说明以及与yarn-site.xml中参数的相互约束关系+spark运行架构图解(持续更新中)