运行Spark任务时解决WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set

 

WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.

下面配置解决上述警告

方式1

1) 在HDFS上创建/spark-yarn/jars目录,并且将spark下的jar包上传到该目录

hadoop fs -mkdir -p /spark-yarn/jars

cd /opt/installs/spark/jars/

hadoop fs -put * /spark-yarn/jars/

2) 修改/opt/installs/spark/conf/spark-defaults.conf文件 spark.yarn.jars=hdfs://hadoop10:8020/spark-yarn/jars/*.jar

方式2

1）使用归档命令创建 archive: jar cv0f spark-libs.jar -C $SPARK_HOME/jars/ .

2）hdfs下创建目录：hdfs dfs -mkdir -p /system/SparkJars/jar

     上传jar包到HDFS: hdfs dfs -put spark-libs.jar /system/SparkJars/jar

3）切换到/opt/installs/spark/conf/下，在spark-default.conf中设置

spark.yarn.archive=hdfs://hadoop10:8020/system/SparkJars/jar/spark-libs.jar

该方法同时可以对Spark任务进行调优

官方解释：

Running Spark on YARN - Spark 3.5.1 Documentation