Exception in thread “main“ org.apache.spark.SparkException: When running with master ‘yarn‘ either

最新推荐文章于 2023-12-16 11:58:50 发布

南潇如梦

最新推荐文章于 2023-12-16 11:58:50 发布

阅读量3.2k

点赞数

分类专栏：日常踩坑记文章标签： spark apache scala

本文链接：https://blog.csdn.net/weixin_45682261/article/details/123490779

版权

在尝试运行Spark程序时遇到错误：'Exception in thread "main" org.apache.spark.SparkException: When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.'。此错误表明在使用YARN作为master时，HADOOP_CONF_DIR或YARN_CONF_DIR环境变量未设置。解决方案包括确保将yarn-site.xml, core-site.xml, hdfs-site.xml打包到应用程序中，并正确设置环境变量。" 122598413,10224566,Java利用Dom4j深入解析XML,"['Java', 'XML解析', 'Dom4j', 'XPath', '数据存储']

摘要由CSDN通过智能技术生成

在执行一个spark程序的时候，明明把yarn-site.xml,core-site.xml,hdfs-site.xml都打包进去了，

可是还是报错

Exception in thread "main" org.apache.spark.SparkException: When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.
        at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657)
        at org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:290)
        at org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:251)
        at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:120)
        at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$1.<init>(SparkSubmit.scala:907)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:907)
        at org.apache.spark.deploy.SparkSubmit.doS