Spark提交报错：1 node(s) are excluded in this operation

头顶榴莲树

已于 2022-02-25 14:15:05 修改

阅读量2.7k

点赞数

分类专栏： spark 文章标签： spark big data hdfs

于 2022-02-25 10:59:39 首次发布

本文链接：https://blog.csdn.net/qq_32068809/article/details/123128334

版权

spark 专栏收录该内容

7 篇文章 0 订阅

订阅专栏

提交spark on yarn作业报错：

主要错误信息“There are 1 datanode(s) running and 1 node(s) are excluded in this operation”，有一个datanode被排除

1088 [main] INFO  org.apache.spark.deploy.yarn.Client  - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
1089 [main] INFO  org.apache.spark.deploy.yarn.Client  - Will allocate AM container, with 1408 MB memory including 384 MB overhead
1089 [main] INFO  org.apache.spark.deploy.yarn.Client  - Setting up container launch context for our AM
1091 [main] INFO  org.apache.spark.deploy.yarn.Client  - Setting up the launch environment for our AM container
1106 [main] INFO  org.apache.spark.deploy.yarn.Client  - Preparing resources for our AM container
1155 [main] WARN  org.apache.spark.deploy.yarn.Client  - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
3343 [main] INFO  org.apache.spark.deploy.yarn.Client  - Uploading resource file:/tmp/spark-510bf868-cc29-4e2f-b1c0-55136c98eb32/__spark_libs__2655251803993337384.zip -> hdfs://dss0:8020/user/root/.sparkStaging/application_1645757337254_0003/__spark_libs__2655251803993
337384.zip3475 [Thread-7] INFO  org.apache.hadoop.hdfs.DataStreamer  - Exception in createBlockOutputStream blk_1073742400_1577
java.io.EOFException: Unexpected EOF while trying to read response from server
	at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:458)
	at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1762)
	at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1679)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
3477 [Thread-7] WARN  org.apache.hadoop.hdfs.DataStreamer  - Abandoning BP-242088412-192.168.78.12-1644827784895:blk_1073742400_1577
3483 [Thread-7] WARN  org.apache.hadoop.hdfs.DataStreamer  - Excluding datanode DatanodeInfoWithStorage[192.168.78.12:9866,DS-686c7ba7-eda5-442a-a041-cf44347b8bfc,DISK]
3503 [Thread-7] WARN  org.apache.hadoop.hdfs.DataStreamer  - DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/root/.sparkStaging/application_1645757337254_0003/__spark_libs__2655251803993337384.zip could only be written to 0 of the 1 minReplication nodes. There are 1 datanode(s) running and 1 node(s) are ex
cluded in this operation.	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2116)

执行"hadoop fs -rm -r -skipTrash /user/root/.sparkStaging/*"，清空掉该目录下的作业缓存文件后解决~，如果问题还存在可以尝试重启hdfs以及yarn，或者关闭一些服务使集群资源更加充足，然后再尝试提交作业