Spark提交Yarn的集群pi任务报错

Spark-2.4.4用hadoop-2.8.5的yarn提交计算pi的任务出现错误。

cd $SPARK_HOME/bin
./spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client ../examples/jars/spark-examples_2.11-2.4.4.jar 10 4

报错信息如下

al-dir/usercache/hadoop/filecache/13_tmp]
2019-09-21 23:22:02,514 WARN org.apache.hadoop.hdfs.client.impl.BlockReaderFactory: I/O error constructing remote block reader.
java.nio.channels.ClosedByInterruptException
        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:659)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
        at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2878)
        at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:815)
        at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:740)
        at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:385)
            at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:696)
        at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:655)
        at org.apache.hadoop.hdfs.DFSInputStream.seekToBlockSource(DFSInputStream.java:1666)
        at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:900)
        at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:935)
        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:982)
        at java.io.DataInputStream.read(DataInputStream.java:100)
        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:96)
        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:64)
        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:125)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:369)
        at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:267)
        at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2019-09-21 23:22:02,516 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /192.168.44.101:50010 for block, add to deadNodes and continue. java.nio.channels.ClosedByInterruptException
java.nio.channels.ClosedByInterruptException
        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:659)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
        at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2878)
        at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:815)
        at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:740)
        at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:385)
        at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:696)

是因为yarn受到虚拟机分配资源的限制

处理方法

在hadoop的yarn.site.xml配置中添加

<property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
</property>

<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>

参数说明：
yarn.nodemanager.pmem-check-enabled
容器是否会执行物理内存限制 默认为True
执行物理内存限制时，当Container达到最大物理内存限制时，该参数为True时,会杀死任务

F)yarn.nodemanager.vmem-check-enabled
容器是否会执行虚拟内存限制 默认为True
执行虚拟限制时，当Container达到最大物理虚拟限制时，该参数为True时,会杀死任务
————————————————
关于yarn的更多参数信息可以参考：https://blog.csdn.net/pengzonglu7292/article/details/80550252

再次执行报错报错

INFO yarn.Client: Preparing resources for our AM container
19/09/23 16:39:51 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
19/09/23 16:39:57 INFO yarn.Client: Uploading resource file:/tmp/spark-35645810-52df-4c80-87d9-6e748cd53f8b/__spark_libs__4608376108230472640.zip -> hdfs://hadoop1:9000/user/hadoop/.sparkStaging/application_1569227950438_0001/__spark_libs__4608376108230472640.zip
19/09/23 16:40:09 WARN ui.JettyUtils: GET /jobs/ failed: java.util.NoSuchElementException
java.util.NoSuchElementException
        at java.util.Collections$EmptyIterator.next(Collections.java:4189)
        at org.apache.spark.util.kvstore.InMemoryStore$InMemoryIterator.next(InMemoryStore.java:373)
        at org.apache.spark.status.AppStatusStore.applicationInfo(AppStatusStore.scala:38)
        at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:275)
        at org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84)
        at org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84)
        at org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
        at org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584)
        at org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
        at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)

对于新的问题，是因为没有重启hadoop，重启之后就不会出现问题了。哈哈
还要注意吧所有节点上的yarn-site.xml中都添加哦！！！