WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
下面配置解决上述警告
方式1
1) 在HDFS上创建/spark-yarn/jars目录,并且将spark下的jar包上传到该目录
hadoop fs -mkdir -p /spark-yarn/jars
cd /opt/installs/spark/jars/
hadoop fs -put * /spark-yarn/jars/
2) 修改/opt/installs/spark/conf/spark-defaults.conf文件 spark.yarn.jars=hdfs://hadoop10:8020/spark-yarn/jars/*.jar
方式2
1)使用归档命令创建 archive: jar cv0f spark-libs.jar -C $SPARK_HOME/jars/ .
2)hdfs下创建目录:hdfs dfs -mkdir -p /system/SparkJars/jar
上传jar包到HDFS: hdfs dfs -put spark-libs.jar /system/SparkJars/jar
3)切换到/opt/installs/spark/conf/下,在spark-default.conf中设置
spark.yarn.archive=hdfs://hadoop10:8020/system/SparkJars/jar/spark-libs.jar
该方法同时可以对Spark任务进行调优
官方解释: