在hadoop yarn上运行spark报错

最新推荐文章于 2024-07-22 20:11:03 发布

wqq奋斗的小鸟

最新推荐文章于 2024-07-22 20:11:03 发布

阅读量3.1k

点赞数

文章标签： spark hadoop yarn

本文链接：https://blog.csdn.net/wqqGo/article/details/80702184

版权

参照相关教程，在yarn-site.xml配置文件中进行调整，并执行重启和集群格式化操作，成功解决了Spark在Hadoop YARN上运行时遇到的错误。

摘要由CSDN通过智能技术生成

hduser@master:/usr/local/hadoop/etc/hadoop$ HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop/ pyspark --master yarn --deploy-mode client
Python 2.7.14 |Anaconda, Inc.| (default, Dec  7 2017, 17:05:42) 
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
18/06/15 10:25:33 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/06/15 10:25:39 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
18/06/15 10:26:39 ERROR YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!
18/06/15 10:26:39 ERROR TransportClient: Failed to send RPC 7707247702813566843 to /