spark执行优化——依赖上传到HDFS(spark.yarn.jar和spark.yarn.archive的使用)

最新推荐文章于 2024-07-17 12:19:49 发布
最新推荐文章于 2024-07-17 12:19:49 发布
阅读量7.2k 收藏 18
点赞数 7
分类专栏： 大数据-spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/OMars/article/details/110425423
版权

1.简述

使用yarn的方式提交spark应用时，在没有配置spark.yarn.archive或者spark.yarn.jars时， 看到输出的日志在输出Neither spark.yarn.jars nor spark.yarn.archive is set；一段指令后，会看到不停地上传本地jar到HDFS上，内容如下，这个过程会非常耗时。可以通过在spark-defaults.conf配置里添加spark.yarn.archive或spark.yarn.jars来缩小spark应用的启动时间。

 Will allocate AM container, with 896 MB memory including 384 MB overhead
2020-12-01 11:16:11 INFO  Client:54 - Setting up container launch context for our AM
2020-12-01 11:16:11 INFO  Client:54 - Setting up the launch environment for our AM container
2020-12-01 11:16:11 INFO  Client:54 - Preparing resources for our AM container
2020-12-01 11:16:12 WARN  Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2020-12-01 11:16:14 INFO  Client:54 - Uploading resource file:/tmp/spark-897c6291-e0bd-47e6-8d42-7f67225c4819/__spark_libs__5294834939010995385.zip -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606792499194_0001/__spark_libs__5294834939010995385.zip
2020-12-01 11:16:18 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/wordcount.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606792499194_0001/wordcount.jar
2020-12-01 11:16:18 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/zookeeper-3.4.6.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606792499194_0001/zookeeper-3.4.6.jar
2020-12-01 11:16:18 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/xz-1.0.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606792499194_0001/xz-1.0.jar

2.spark官网对这两个配置的解释

在这里插入图片描述
中文释义大概如下
在这里插入图片描述

3.spark.yarn.jars使用

3.1 将spark根目录下jars里的所有jar包上传到HDFS
 hadoop fs -mkdir -p  /spark-yarn/jars
 hadoop fs -put /opt/module/spark-2.3.2-bin-hadoop2.7/jars/* /spark-yarn/jars/
3.2 修改spark-defaults.conf
spark.yarn.jars hdfs://hadoop122:9000/spark-yarn/jars/*.jar
3.3 效果
2020-12-01 13:53:52 INFO  Client:54 - Setting up container launch context for our AM
2020-12-01 13:53:52 INFO  Client:54 - Setting up the launch environment for our AM container
2020-12-01 13:53:52 INFO  Client:54 - Preparing resources for our AM container
2020-12-01 13:53:53 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/jars/JavaEWAH-0.3.2.jar
2020-12-01 13:53:53 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/jars/RoaringBitmap-0.5.11.jar
3.4 可能遇到的错误

ERROR client.TransportClient: Failed to send RPC RPC

Caused by: java.io.IOException: Failed to send RPC 5353749227723805834 to /192.168.10.122:58244: java.nio.channels.ClosedChannelException
	at org.apache.spark.network.client.TransportClient.lambda$sendRpc$2(TransportClient.java:237)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
	at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
	at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:431)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
	at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)

关闭通道异常，看上去是超时的问题，这个问题当运行spark-shell --master yarn-client时，可能也会出现。在yarn-site.xml里添加如下配置可以解决

<property>
		<name>yarn.nodemanager.pmem-check-enabled</name>
		<value>false</value>
</property>
<property>
		<name>yarn.nodemanager.vmem-check-enabled</name>
		<value>false</value>
</property>

4.spark.yarn.archive使用

4.1 将spark根目录下jars里的所有jar包上传到HDFS

打包要注意所有的jar都在zip包的根目录中

cd /opt/module/spark-2.3.2-bin-hadoop2.7/jars/
zip -q -r spark_jars_2.3.2.zip *
hadoop fs -mkdir /spark-yarn/zip
hadoop fs -put spark_jars_2.3.2.zip /spark-yarn/zip/
4.2 修改spark-defaults.conf
spark.yarn.archive hdfs://hadoop122:9000/spark-yarn/zip/spark_jars_2.3.2.zip
4.3 效果
2020-12-01 14:41:53 INFO  Client:54 - Setting up container launch context for our AM
2020-12-01 14:41:53 INFO  Client:54 - Setting up the launch environment for our AM container
2020-12-01 14:41:53 INFO  Client:54 - Preparing resources for our AM container
2020-12-01 14:41:54 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/zip/spark_jars_2.3.2.zip
2020-12-01 14:41:54 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/wordcount.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606801972366_0009/wordcount.jar
2020-12-01 14:41:55 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/zstd-jni-1.3.2-2.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606801972366_0009/zstd-jni-1.3.2-2.jar
2020-12-01 14:41:55 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/zookeeper-3.4.6.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606801972366_0009/zookeeper-3.4.6.jar
2020-12-01 14:41:55 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/xz-1.0.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606801972366_0009/xz-1.0.jar
2020-12-01 14:41:55 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/xmlenc-0.52.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606801972366_0009/xmlenc-0.52.jar
2020-12-01 14:41:55 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/xml-apis-1.3.04.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606801972366_0009/xml-apis-1.3.04.jar
2020-12-01 14:41:55 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/xercesImpl-2.9.1.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606801972366_0009/xercesImpl-2.9.1.jar
2020-12-01 14:41:55 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/xbean-asm5-shaded-4.4.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606801972366_0009/xbean-asm5-shaded-4.4.jar
2020-12-01 14:41:55 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/spark-core_2.11-2.3.2.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606801972366_0009/spark-core_2.11-2.3.2.jar
4.4 可能遇到的错误

应用的driver日志

错误: 找不到或无法加载主类 org.apache.spark.deploy.yarn.ApplicationMaster

如果像如下的打包方式，就会保留目录的层级到zip包中，就会报错如上

zip -q -r spark_jars_2.3.2.zip /opt/module/spark-2.3.2-bin-hadoop2.7/jars/*

在这里插入图片描述

5.效果对比

spark官网关于这两个配置的说明有以下两块
在这里插入图片描述
Preparations处有个说明如果未指定spark.yarn.archive或者spark.yarn.jars，Spark将创建一个zip文件，包含所有$SPARK_HOME/jars路径下jar包，并将其上传到分布式缓存。
在这里插入图片描述
spark.yarn.archive有个说明是，说如果两个参数都配置，应用程序会优先使用 spark.yarn.archive会代替 spark.yarn.jars 在所有容器中使用打包文件。

为了更好的看出效果，我分别就配置的几种情况，提交yarn-cluster任务查看控制台输出的日志情况

5.1 未配置spark.yarn.jars/spark.yarn.archive

上传：__spark_conf__xxx.zip、__spark_libs__xxx.zip、jar包、wordcount.jar

2020-12-02 14:40:01 INFO  Client:54 - Will allocate AM container, with 1408 MB memory including 384 MB overhead
2020-12-02 14:40:01 INFO  Client:54 - Setting up container launch context for our AM
2020-12-02 14:40:01 INFO  Client:54 - Setting up the launch environment for our AM container
2020-12-02 14:40:01 INFO  Client:54 - Preparing resources for our AM container
2020-12-02 14:40:02 WARN  Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2020-12-02 14:40:04 INFO  Client:54 - Uploading resource file:/tmp/spark-343f5dda-9476-4330-b2ed-407ec6aa00e9/__spark_libs__2239070030081220213.zip -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0003/__spark_libs__2239070030081220213.zip
2020-12-02 14:40:07 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/wordcount.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0003/wordcount.jar
2020-12-02 14:40:07 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/zstd-jni-1.3.2-2.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0003/zstd-jni-1.3.2-2.jar
2020-12-02 14:40:07 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/zookeeper-3.4.6.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0003/zookeeper-3.4.6.jar
...
...
2020-12-02 14:40:14 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/aopalliance-repackaged-2.4.0-b34.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0003/aopalliance-repackaged-2.4.0-b34.jar
2020-12-02 14:40:14 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/activation-1.1.1.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0003/activation-1.1.1.jar
2020-12-02 14:40:14 INFO  Client:54 - Uploading resource file:/tmp/spark-343f5dda-9476-4330-b2ed-407ec6aa00e9/__spark_conf__7169080322038125856.zip -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0003/__spark_conf__.zip
2020-12-02 14:40:14 INFO  SecurityManager:54 - Changing view acls to: root
...
5.2 配置spark.yarn.jars

上传：__spark_conf__xxx.zip、（未上传）jar包、wordcount.jar

2020-12-02 14:47:07 INFO  Client:54 - Will allocate AM container, with 1408 MB memory including 384 MB overhead
2020-12-02 14:47:07 INFO  Client:54 - Setting up container launch context for our AM
2020-12-02 14:47:07 INFO  Client:54 - Setting up the launch environment for our AM container
2020-12-02 14:47:07 INFO  Client:54 - Preparing resources for our AM container
2020-12-02 14:47:08 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/jars/JavaEWAH-0.3.2.jar
2020-12-02 14:47:08 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/jars/RoaringBitmap-0.5.11.jar
2020-12-02 14:47:08 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/jars/ST4-4.0.4.jar
2020-12-02 14:47:08 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/jars/activation-1.1.1.jar
...
...
2020-12-02 14:47:08 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/jars/zookeeper-3.4.6.jar
2020-12-02 14:47:08 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/jars/zstd-jni-1.3.2-2.jar
2020-12-02 14:47:08 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/wordcount.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0004/wordcount.jar
2020-12-02 14:47:09 WARN  Client:66 - Same name resource file:///home/workspace/wordcount/lib/zstd-jni-1.3.2-2.jar added multiple times to distributed cache
2020-12-02 14:47:09 WARN  Client:66 - Same name resource file:///home/workspace/wordcount/lib/zookeeper-3.4.6.jar added multiple times to distributed cache
...
...
2020-12-02 14:47:09 WARN  Client:66 - Same name resource file:///home/workspace/wordcount/lib/scala-parser-combinators_2.11-1.0.4.jar added multiple times to distributed cache
2020-12-02 14:47:09 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/scalap-2.11.0.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0004/scalap-2.11.0.jar
2020-12-02 14:47:09 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/scala-logging_2.11-3.5.0.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0004/scala-logging_2.11-3.5.0.jar
2020-12-02 14:47:09 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/scala-library-2.11.12.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0004/scala-library-2.11.12.jar
2020-12-02 14:47:09 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/scala-compiler-2.11.12.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0004/scala-compiler-2.11.12.jar
...
...
2020-12-02 14:47:11 WARN  Client:66 - Same name resource file:///home/workspace/wordcount/lib/aopalliance-repackaged-2.4.0-b34.jar added multiple times to distributed cache
2020-12-02 14:47:11 WARN  Client:66 - Same name resource file:///home/workspace/wordcount/lib/activation-1.1.1.jar added multiple times to distributed cache
2020-12-02 14:47:11 INFO  Client:54 - Uploading resource file:/tmp/spark-b763698e-77bd-4004-8d71-c0eca5a1006d/__spark_conf__5511617263425666988.zip -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0004/__spark_conf__.zip
2020-12-02 14:47:11 INFO  SecurityManager:54 - Changing view acls to: root
5.3 配置spark.yarn.archive

上传：__spark_conf__xxx.zip、jar包、wordcount.jar

2020-12-02 14:53:43 INFO  Client:54 - Will allocate AM container, with 1408 MB memory including 384 MB overhead
2020-12-02 14:53:43 INFO  Client:54 - Setting up container launch context for our AM
2020-12-02 14:53:43 INFO  Client:54 - Setting up the launch environment for our AM container
2020-12-02 14:53:43 INFO  Client:54 - Preparing resources for our AM container
2020-12-02 14:53:44 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/zip/spark_jars_2.3.2.zip
2020-12-02 14:53:44 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/wordcount.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0005/wordcount.jar
2020-12-02 14:53:44 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/zstd-jni-1.3.2-2.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0005/zstd-jni-1.3.2-2.jar
2020-12-02 14:53:45 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/zookeeper-3.4.6.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0005/zookeeper-3.4.6.jar
...
...
2020-12-02 14:53:51 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/aopalliance-repackaged-2.4.0-b34.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0005/aopalliance-repackaged-2.4.0-b34.jar
2020-12-02 14:53:51 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/activation-1.1.1.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0005/activation-1.1.1.jar
2020-12-02 14:53:51 INFO  Client:54 - Uploading resource file:/tmp/spark-0ce3c5f1-6083-499a-b5cd-1a5700e74bf3/__spark_conf__6154114087017136415.zip -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0005/__spark_conf__.zip
2020-12-02 14:53:51 INFO  SecurityManager:54 - Changing view acls to: root
5.4 配置spark.yarn.jars&spark.yarn.archive

上传：__spark_conf__xxx.zip、jar包、wordcount.jar

2020-12-02 14:59:08 INFO  Client:54 - Will allocate AM container, with 1408 MB memory including 384 MB overhead
2020-12-02 14:59:08 INFO  Client:54 - Setting up container launch context for our AM
2020-12-02 14:59:08 INFO  Client:54 - Setting up the launch environment for our AM container
2020-12-02 14:59:08 INFO  Client:54 - Preparing resources for our AM container
2020-12-02 14:59:09 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/zip/spark_jars_2.3.2.zip
2020-12-02 14:59:09 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/wordcount.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0006/wordcount.jar
2020-12-02 14:59:10 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/zstd-jni-1.3.2-2.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0006/zstd-jni-1.3.2-2.jar
2020-12-02 14:59:10 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/zookeeper-3.4.6.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0006/zookeeper-3.4.6.jar
...
...
2020-12-02 14:59:16 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/aopalliance-repackaged-2.4.0-b34.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0006/aopalliance-repackaged-2.4.0-b34.jar
2020-12-02 14:59:16 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/lib/activation-1.1.1.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0006/activation-1.1.1.jar
2020-12-02 14:59:16 INFO  Client:54 - Uploading resource file:/tmp/spark-3451a4e5-fb97-45a5-85dc-36f645fb7db3/__spark_conf__4794321975937827120.zip -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0006/__spark_conf__.zip
2020-12-02 14:59:16 INFO  SecurityManager:54 - Changing view acls to: root
5.5 配置spark.yarn.jars并且上传了程序所有的依赖jar到HDFS /spark-yarn/jars/ 下

上传：__spark_conf__xxx.zip、wordcount.jar

2020-12-02 15:19:30 INFO  Client:54 - Will allocate AM container, with 1408 MB memory including 384 MB overhead
2020-12-02 15:19:30 INFO  Client:54 - Setting up container launch context for our AM
2020-12-02 15:19:30 INFO  Client:54 - Setting up the launch environment for our AM container
2020-12-02 15:19:30 INFO  Client:54 - Preparing resources for our AM container
2020-12-02 15:19:31 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/jars/JavaEWAH-0.3.2.jar
2020-12-02 15:19:31 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/jars/RoaringBitmap-0.5.11.jar
2020-12-02 15:19:31 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/jars/ST4-4.0.4.jar
2020-12-02 15:19:31 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/jars/activation-1.1.1.jar
...
...
2020-12-02 15:19:31 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/jars/zookeeper-3.4.6.jar
2020-12-02 15:19:31 INFO  Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoop122:9000/spark-yarn/jars/zstd-jni-1.3.2-2.jar
2020-12-02 15:19:31 INFO  Client:54 - Uploading resource file:/home/workspace/wordcount/wordcount.jar -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0007/wordcount.jar
2020-12-02 15:19:31 WARN  Client:66 - Same name resource file:///home/workspace/wordcount/lib/zstd-jni-1.3.2-2.jar added multiple times to distributed cache
2020-12-02 15:19:31 WARN  Client:66 - Same name resource file:///home/workspace/wordcount/lib/zookeeper-3.4.6.jar added multiple times to distributed cache
...
...
2020-12-02 15:19:31 WARN  Client:66 - Same name resource file:///home/workspace/wordcount/lib/scala-parser-combinators_2.11-1.0.4.jar added multiple times to distributed cache
2020-12-02 15:19:31 WARN  Client:66 - Same name resource file:///home/workspace/wordcount/lib/scalap-2.11.0.jar added multiple times to distributed cache
2020-12-02 15:19:31 WARN  Client:66 - Same name resource file:///home/workspace/wordcount/lib/scala-logging_2.11-3.5.0.jar added multiple times to distributed cache
2020-12-02 15:19:31 WARN  Client:66 - Same name resource file:///home/workspace/wordcount/lib/scala-library-2.11.12.jar added multiple times to distributed cache
2020-12-02 15:19:31 WARN  Client:66 - Same name resource file:///home/workspace/wordcount/lib/scala-compiler-2.11.12.jar added multiple times to distributed cache
...
...
2020-12-02 15:19:31 WARN  Client:66 - Same name resource file:///home/workspace/wordcount/lib/aopalliance-repackaged-2.4.0-b34.jar added multiple times to distributed cache
2020-12-02 15:19:31 WARN  Client:66 - Same name resource file:///home/workspace/wordcount/lib/activation-1.1.1.jar added multiple times to distributed cache
2020-12-02 15:19:32 INFO  Client:54 - Uploading resource file:/tmp/spark-e9eca080-9f38-4fe5-ae5f-663a3e54c718/__spark_conf__3182593855268881202.zip -> hdfs://hadoop122:9000/user/root/.sparkStaging/application_1606887365975_0007/__spark_conf__.zip
2020-12-02 15:19:32 INFO  SecurityManager:54 - Changing view acls to: root
结论：

5.5（配置spark.yarn.jars并且上传了程序所有的依赖jar到HDFS /spark-yarn/jars/ 下）的方式更能减少资源的上传

参考文章：
https://blog.csdn.net/liyaya0201/article/details/105277681
https://www.cnblogs.com/yyy-blog/p/11110388.html
https://www.jianshu.com/p/e44e948b8d5f
https://blog.csdn.net/u012957549/article/details/89361485

Marcel Lou
关注
专栏目录
spark.yarn.archive spark.yarn.jars 的设置
naruto00001的专栏
03-16 7189
为什么要设置呢？摘自Spark-2.3.0官方文档 http://spark.apache.org/docs/latest/running-on-yarn.html#preparations（个人理解：spark运行所需的jar包，不设置的话每次运行就需要上传到yarn管理的各个节点的缓存，很麻烦很影响性能。如果设置了，比如说放在HDFS上，就不需要每次都上传而是从HDFS上读取，能快那么一点点…...
十：WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set,解决案例
weizhonggui的博客
12-25 7327
在spark on yarn 上提交任务是，报错如下： 这是Spark的一个优化点： WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. INFO yarn.Client: Uploading re...
spark.yarn.archive spark.yarn.jars
我的祖传代码
03-25 1394
参考自：http://spark.apache.org/docs/latest/running-on-yarn.html#preparations 在spark on yarn模式下，/usr/local/spark-current2.3/conf下的spark-defaults.conf配置文件有一个spark.yarn.archive配置项 1.如果没有配置 #spark.yarn.a...
运行Spark任务时解决WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set
最新发布
weixin_56130021的博客
07-17 355
2) 修改/opt/installs/spark/conf/spark-defaults.conf文件 spark.yarn.jars=hdfs://hadoop10:8020/spark-yarn/jars/*.jar。3）在spark-default.conf中设置 spark.yarn.archive=hdfs://hadoop10:8020/system/SparkJars/jar/spark-libs.jar。该方法同时可以对Spark任务进行调优。下面配置解决上述警告。
spark优化——依赖包传入HDFS_spark.yarn.jar和spark.yarn.archive的使用
weixin_30607659的博客
06-30 582
一、参数说明 启动Spark任务时，在没有配置spark.yarn.archive或者spark.yarn.jars时， 会看到不停地上传jar，非常耗时；使用spark.yarn.archive可以大大地减少任务的启动时间，整个处理过程如下。 二、spark.yarn.archive使用 1.在本地创建zip文件 silent@bd01:~/env/spark$ cd jars/ ...
spark执行优化--禁止将依赖的Jar包传到HDFS(spark.yarn.jar和spark.yarn.archive的使用)
penriver的博客
04-26 1831
使用yarn的方式提交spark应用时，如果没有配置spark.yarn.archive或者spark.yarn.jars时， 输出的日志在输出Neither spark.yarn.jars nor spark.yarn.archive is set后，会将SPARK_HOME/jars下的所有jar打包并上传到HDFS上，内容如下，这个过程会非常耗时。可以通过如下2种方法避免此操作，缩小spark应用的启动时间 spark-defaults.conf配置里添加spark.yarn.archive或s
spark执行优化--将依赖的jar包上传至hdfs上（spark3.1.1）
weixin_42934205的博客
10-26 3601
spark执行过程优化------将spark运行的jar包上传到HDFS上，避免在提交任务时因为spark程序包过大导致的占用运行时间和cpu资源 起因是在提交spark任务的时候，因为服务器处于离线状态，好多的依赖包无法通过网络进行下载，所以打成的spark程序包比较大，大概在200~300MB之间，每次通过spark-on-yarn提交任务时，总是有一段时间将本地的spark程序上传到hdfs，占用了太多的网络资源以及cpu。 所以，通过配置spark.yarn.archive和spark.yarn.
spark执行优化——依赖上传到HDFS二(-conf spark.yarn.dist.jars或者--jars 的使用)
OMars的博客
08-26 3477
1.说明 之前整理过一篇类似文章，但是这个spark.yarn.jar配置的目录最好只是放spark jars目录下的jar包，如果放入其他的jar包，很大概率会有冲突，而且如果项目比较多，jar包引入的内容版本不尽相同，也不太利于管理。题主这里有一个spark的分析项目，引入了很多依赖，如果只是配置了spark.yarn.jars,上传jar包的过程仍然很慢，所以还是需要把项目的依赖jar包上传到HDFS,经过查阅资料和翻查官网，发现了application-jar , --jars 都是即可以使用本地也
java提交spark任务到yarn平台的配置讲解共9页.pdf.zip
10-28
Java提交Spark任务到YARN平台是一项常见的大数据处理操作，它结合了Java的编程能力和Spark的高性能计算框架，以及Hadoop的资源管理系统YARN。在这个过程中，开发者需要理解多个关键概念和配置步骤，以便有效地调度和...
Spark on YARN 模式启动后去HDFS上加载依赖包
muyingmiao的专栏
10-18 1549
1.在使用Spark on YARN模式时，发现作业提交缓慢 通过官网描述： 也就是说，：如果想要在yarn端（yarn的节点）访问spark的runtime jars，需要指定spark.yarn.archive 或者 spark.yarn.jars。如果都这两个参数都没有指定，spark就会把$SPARK_HOME/jars/所有的jar上传到分布式缓存中。这也是之前任务提交特别慢的原因。 ...
Spark on Yarn运行时加载的jar包
upupfeng的博客
08-22 3725
记录下spark on yarn运行时会加载的jar包
spark.yarn.jar和spark.yarn.archive的使用
weixin_30826761的博客
11-29 756
启动Spark任务时，在没有配置spark.yarn.archive或者spark.yarn.jars时， 会看到不停地上传jar非常耗时；使用spark.yarn.archive可以大大地减少任务的启动时间，整个处理过程如下 1.在本地创建zip文件 hzlishuming@hadoop691:~/env/spark$ cd jars/ hzlishuming@hadoop691:~/...
Spark On Yarn --jars/spark.yarn.jars 踩坑 与 提升spark submit速度
BITDDD小栈
04-14 5907
一.引言 Spark submit 可以选择 --jars 传入本地的jar也可以 --jars 传入HDFS的jar包，经过半下午的实验，终于搞清了两者的关系以及 spark.yarn.jars 和它们的区别 二.--jars的使用 1.--jars 传入本地jar包 --jars a.jar,b.jar,c.jar 传入本地jar包时，只需要输入通道机的jar包地址即可，随后spark-submit脚本会执行Uploading resource的操作将本地的jar包拷贝到spar..
spark.yarn.archive 的正确设置方法
微电子学与固体电子学-俞驰
05-02 3223
spark.yarn.archive 的设置方法有很多，下面三种 ①spark.yarn.jars hdfs://Desktop:9000/spark/jars/* ②spark.yarn.jars hdfs://Desktop:9000/spark/jars/*.jar ③spark.yarn.jars hdfs://Desktop:9000/spark/jars/ 上述配置真的正确...
Spark 在YARN上运行
大强博客
01-31 5127
在YARN上运行Spark 安全 在YARN上启动Spark 添加其他JAR 准备工作 组态 调试您的应用程序 Spark特性 重要笔记 的Kerberos YARN特定的Kerberos配置 Kerberos故障排除 配置外部随机播放服务 使用Apache Oozie启动您的应用程序 使用Spark History Server替换Spark Web U...
Spark2.1.1<性能调优Spark运行时jar从yarn端访问>
Gpwner的博客
06-19 3600
1.为什么要让运行时Jar可以从yarn端访问spark2以后，原有lib目录下的大JAR包被分散成多个小JAR包，原来的spark-assembly-*.jar已经不存在每一次我们运行的时候，如果没有指定 spark.yarn.archive or spark.yarn.jars Spark将在安装路径下的Jar目录，将其所有的Jar包打包然后将其上传到分布式缓存（官网上的原话是：To ma
使用java远程提交spark任务到yarn集群
weixin_43039757的博客
05-17 421
公司需求中，需要用到java远程提交spark任务，方式还是用yarn提供的方法提交任务。如果你也想远程提交flink任务，请看这篇文章。
spark.yarn.jars的配置
xinji
10-14 2978
1.原因: 官方文档 http://spark.apache.org/docs/latest/running-on-yarn.html#preparations To make Spark runtime jars accessible from YARN side, you can specify spark.yarn.archive or spark.yarn.jars. For deta...
spark运行时出现Neither spark.yarn.jars nor spark.yarn.archive is set错误的解决办法（图文详解）...
weixin_34023863的博客
02-26 676
问题详情 　　每次提交spark任务到yarn的时候，总会出现uploading resource（打包spark jars并上传）到hdfs上。恶劣情况下，会在这里卡住很久。 17/01/13 17:21:47 INFO Client: Preparing resources for our AM container 17/01/13 17:2...
hadoop集群各组件性能调优[spark，hbase，hive，hdfs...]
03-16
Hadoop集群各组件的性能调优需要根据具体情况进行，以下是一些常见的调优方法： 1. Spark性能调优：可以通过调整Spark的内存分配、并行度、缓存等参数来提高性能。例如，可以增加executor的内存大小，提高并行度，使用广播变量等。 2. HBase性能调优：可以通过调整HBase的缓存、预分区、压缩等参数来提高性能。例如，可以增加region server的缓存大小，预分区表以减少热点访问，使用压缩算法等。 3. Hive性能调优：可以通过调整Hive的查询优化、数据压缩、分区等参数来提高性能。例如，可以使用分区表来减少扫描数据量，使用压缩算法减少磁盘IO，使用索引等。 4. HDFS性能调优：可以通过调整HDFS的块大小、副本数、缓存等参数来提高性能。例如，可以增加块大小以减少寻道时间，增加副本数以提高可靠性和读取速度，使用缓存等。 总之，性能调优需要根据具体情况进行，需要对系统进行全面的分析和测试，才能找到最优的调优方案。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值