pyspark写入mysql内存溢出_当我用sparksubmit运行作业.py，它总是写着文件'pyspark.zip文件'不存在...

最新推荐文章于 2022-12-20 10:11:17 发布
最新推荐文章于 2022-12-20 10:11:17 发布
阅读量516 收藏
点赞数
文章标签： pyspark写入mysql内存溢出
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_29871753/article/details/113960572
版权

环境：spark-2.1

当我使用spark submit运行作业.py，它总是说文件pyspark.zip不存在。在

但还是不行。我的run.sh如下：#!/bin/sh

/usr/lib/software/spark/spark-2.1/bin/spark-submit \

--master yarn-cluster \

--driver-memory 5G \

--num-executors 12 \

./job.py

我的job.py如下：

^{pr2}$

我运行命令：nohup/运行.sh&gt；日志2&gt；1&amp；。在

日志文件如下：nohup: 忽略输入

Warning: Master yarn-cluster is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead.

18/08/21 15:29:28 WARN util.NativeCodeLoader main: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

18/08/21 15:29:29 INFO yarn.Client main: Requesting a new application from cluster with 2313 NodeManagers

18/08/21 15:29:29 INFO yarn.Client main: Verifying our application has not requested more than the maximum memory capability of the cluster (26624 MB per container)

18/08/21 15:29:29 INFO yarn.Client main: Will allocate AM container, with 5632 MB memory including 512 MB overhead

18/08/21 15:29:29 INFO yarn.Client main: Setting up container launch context for our AM

18/08/21 15:29:29 INFO yarn.Client main: Setting up the launch environment for our AM container

18/08/21 15:29:29 INFO yarn.Client main: Preparing resources for our AM container

18/08/21 15:29:31 INFO yarn.Client main: Source and destination file systems are the same. Not copying viewfs://xxxx-cluster/home/spark/jar/spark-2.1.2-bin-U1/spark-2.1.2-bin-U1.zip

18/08/21 15:29:31 INFO yarn.Client main: Source and destination file systems are the same. Not copying viewfs://xxxx-cluster/home/spark/files/2.1/log4j.properties

18/08/21 15:29:31 INFO yarn.Client main: Uploading resource file:/home/job.py -> viewfs://xxxx-cluster/home/spark/stagingDir/.sparkStaging/application_1534831354007_17334/job.py

18/08/21 15:29:32 INFO yarn.Client main: Uploading resource file:/usr/lib/software/spark/spark-2.1/python/lib/pyspark.zip -> viewfs://xxxx-cluster/home/spark/stagingDir/.sparkStaging/application_1534831354007_17334/pyspark.zip

18/08/21 15:29:33 INFO yarn.Client main: Uploading resource file:/usr/lib/software/spark/spark-2.1/python/lib/py4j-0.10.4-src.zip -> viewfs://xxxx-cluster/home/spark/stagingDir/.sparkStaging/application_1534831354007_17334/py4j-0.10.4-src.zip

18/08/21 15:29:33 INFO yarn.Client main: Uploading resource file:/tmp/spark-cae7c762-dc5e-4081-8b17-c1c136072780/__spark_conf__4062455229532468331.zip -> viewfs://xxxx-cluster/home/spark/stagingDir/.sparkStaging/application_1534831354007_17334/__spark_conf__.zip

18/08/21 15:29:34 INFO spark.SecurityManager main: Changing view acls to: hdp_lbg_ectech,dr.who

18/08/21 15:29:34 INFO spark.SecurityManager main: Changing modify acls to: hdp_lbg_ectech

18/08/21 15:29:34 INFO spark.SecurityManager main: Changing view acls groups to:

18/08/21 15:29:34 INFO spark.SecurityManager main: Changing modify acls groups to:

18/08/21 15:29:34 INFO spark.SecurityManager main: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hdp_lbg_ectech, dr.who); groups with view permissions: Set(); users with modify permissions: Set(hdp_lbg_ectech); groups with modify permissions: Set()

18/08/21 15:29:34 INFO yarn.Client main: Submitting application application_1534831354007_17334 to ResourceManager

18/08/21 15:29:34 INFO impl.YarnClientImpl main: Submitted application application_1534831354007_17334

18/08/21 15:29:35 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: SUBMITTED)

18/08/21 15:29:35 INFO yarn.Client main:

client token: N/A

diagnostics: N/A

ApplicationMaster host: N/A

ApplicationMaster RPC port: -1

queue: root.spark.hdp_lbg_ectech.spark

start time: 1534836574582

final status: UNDEFINED

tracking URL: http://xxxxxxxxxxx:9088/proxy/application_1534831354007_17334/

user: hdp_lbg_ectech

18/08/21 15:29:36 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: SUBMITTED)

18/08/21 15:29:37 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: SUBMITTED)

18/08/21 15:29:38 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: SUBMITTED)

18/08/21 15:29:39 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: SUBMITTED)

18/08/21 15:29:40 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:40 INFO yarn.Client main:

client token: N/A

diagnostics: N/A

ApplicationMaster host: N/A

ApplicationMaster RPC port: -1

queue: root.spark.hdp_lbg_ectech.spark

start time: 1534836574582

final status: UNDEFINED

tracking URL: http://xxxxxxxxxxx:9088/proxy/application_1534831354007_17334/

user: hdp_lbg_ectech

18/08/21 15:29:41 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:42 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:43 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:44 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:45 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:46 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:47 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:48 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:49 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:50 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:51 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:52 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:53 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:54 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:55 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:56 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:57 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:58 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:29:59 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:00 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:01 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:02 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:03 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:04 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:05 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:06 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:07 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:08 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:09 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:10 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:11 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:12 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:13 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:14 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:15 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:16 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:17 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:18 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:19 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:20 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:21 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:22 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:23 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:24 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:25 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:26 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:27 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:28 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:29 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:30 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:31 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:32 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:33 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:34 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:35 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:36 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:37 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:38 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:39 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:40 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:41 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:42 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:43 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:44 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:45 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:46 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:47 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:48 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:49 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:50 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:51 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:52 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:53 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:54 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:55 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:56 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:57 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:58 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:30:59 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:00 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:01 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:02 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:03 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:04 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:05 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:06 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:07 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:08 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:09 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:10 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:11 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:12 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:13 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:14 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:15 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:16 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:17 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:18 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:19 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:20 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:21 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:22 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:23 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:24 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:25 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:26 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:27 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:28 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:29 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:30 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:31 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:32 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:33 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:34 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:35 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:36 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:37 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:38 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:39 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:40 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:41 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:42 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:43 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:44 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:45 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:46 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:47 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:48 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:49 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:50 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:57 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:58 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:31:59 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:00 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:02 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:03 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:04 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:05 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:06 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:07 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:08 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:09 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:10 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:11 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:12 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:13 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:14 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:15 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:16 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:17 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:18 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:19 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:20 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:21 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)

18/08/21 15:32:22 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: FAILED)

18/08/21 15:32:22 INFO yarn.Client main:

client token: N/A

diagnostics: Application application_1534831354007_17334 failed 4 times due to AM Container for appattempt_1534831354007_17334_000004 exited with exitCode: -1000

For more detailed output, check application tracking page:http://xxxxxxxxxxx:9088/proxy/application_1534831354007_17334/Then, click on links to logs of each attempt.

Diagnostics: File does not exist: hdfs://xxxx:8020/home/spark/stagingDir/.sparkStaging/application_1534831354007_17334/pyspark.zip

java.io.FileNotFoundException: File does not exist: hdfs://xxxx:8020/home/spark/stagingDir/.sparkStaging/application_1534831354007_17334/pyspark.zip

at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1135)

at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1127)

at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)

at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1127)

at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)

at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)

at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)

at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1672)

at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)

at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Failing this attempt. Failing the application.

ApplicationMaster host: N/A

ApplicationMaster RPC port: -1

queue: root.spark.hdp_lbg_ectech.spark

start time: 1534836574582

final status: FAILED

tracking URL: http://xxxxxx:9088/cluster/app/application_1534831354007_17334

user: hdp_lbg_ectech

Exception in thread "main" org.apache.spark.SparkException: Application application_1534831354007_17334 finished with failed status

at org.apache.spark.deploy.yarn.Client.run(Client.scala:1185)

at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1231)

at org.apache.spark.deploy.yarn.Client.main(Client.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:744)

at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)

at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

18/08/21 15:32:22 INFO util.ShutdownHookManager Thread-3: Shutdown hook called

18/08/21 15:32:22 INFO util.ShutdownHookManager Thread-3: Deleting directory /tmp/spark-cae7c762-dc5e-4081-8b17-c1c136072780

大多数帖子说“pyspark.zip文件不存在”的原因是某个地方的代码将主文件设置为local，但我找不到任何地方将主文件设置为local，在我的run.sh中，我设置了yarn-cluster

你能帮我解决这个问题吗？非常感谢你！在

皮实妞
关注
[网络安全自学篇] 六十三.hack the box渗透之OpenAdmin题目及蚁剑管理员提权（四）
杨秀璋的专栏
03-27 1万+
在撰写这篇文章之前，我先简单分享下hack the box实验感受。hack the box是一个在线渗透平台，模拟了真实环境且难度较大，各种Web渗透工具及操作串联在一起，挺有意思的。本文详细讲解了hack the box机器配置，通过OpenAdmin题目分享管理员权限Flag获取流程，希望对您有所帮助。同时，该网站题目细节不便于透露，推荐大家亲自去尝试，本文更多的分享一些思想。
[网络安全自学篇] 九十一.阿里云搭建LNMP环境及实现PHP自定义网站IP访问 (1)
杨秀璋的专栏
07-25 6337
这是作者网络安全自学教程系列，主要是关于安全工具和实践操作的在线笔记，特分享出来与博友们学习，希望您喜欢，一起进步。本文主要分享作阿里云搭建LNMP环境及实现PHP自定义网站IP访问，同时详细介绍走过的坑，还发了朋友圈求助大家，再次感谢。本来想重新设置一个专栏分享基于阿里云的PHP网站搭建，但考虑到搞安全的也会面临各种靶场及个人网站搭建的情况，所以将这篇文章放置在“网络安全自学篇”中，后续还会分享PHP如何记录后台IP及Python定位经纬度等知识，希望对您有所帮助~
pyspark相关包.zip
02-09
在Windows下安装使用pyspark所依赖的软件，含python-3.7.2-amd64.exe，scala-2.11.8.msi，spark-2.4.0-bin-hadoop2.7.tgz，winutils-master.zip，其中winutils-master.zip中有hadoop-2.6.0到hadoop-3.0.0各版本，使用说明详见：drguo.blog.csdn.net/article/details/113769453
Spark读取本地文件时报错：java.io.FileNotFountException:file doesn't exist
intersting的博客
10-09 3668
在确认文件存在的情况下，出现这个问题的原因是：在集群中运行的pyspark。例如使用命令bin/pyspark --master yarn启动pyspark 解决方式1：让每个Worker节点的相应位置都有要读取的数据文件。 解决方式2：直接将数据文件上传到hdfs，达到数据共享。...
【Pyspark】Spark导入zip文件/上传zip文件
sunflower_sara的机器学习园地
07-17 2596
Spark上传zip文件 步骤一、启动spark的时候addfile zip文件 #zip文件原始路径 file_path = "./filename.zip" #启动spark spark = SparkSession.builder.appName("space_name").enableHiveSupport().getOrCreate() sc = spark.sparkContext # 添加文件到spark的空间,位于根目录下 sc.addFile(file_path) ...
记一次spark-submi 提交python脚本 遇到的问题
qq_42337191的博客
07-01 320
一、通过spark-submit 提交报错如下 yarn运行模式 spark用的版本是2.4.0是支持pandas_udf的，而且通过pyspark的shell命令行一条条执行 都是没有问题的 但是将代码作为文件用spark submit提交就报这个错误 二、解决办法： @pandas_udf(returnType=“string”, PandasUDFType.GROUPED_AGG)中的修改为@pandas_udf(StringType(),PandasUDFType.GROUPED_AGG)
Learning PySpark(pdf+epub+mobi+code_file).zip
05-15
Learning PySpark Learning PySpark Learning PySpark
pyspark写入mysql内存溢出_Spark开发-Spark内存溢出原因以及解决方式
weixin_30128407的博客
02-02 1323
Dpark内存溢出Spark内存溢出堆内内存溢出堆外内存溢出堆内内存溢出java.lang.OutOfMemoryError: GC overhead limit execeededjava.lang.OutOfMemoryError: Java heap space具体说明Heap size JVM堆的设置是指java程序运行过程中JVM可以调配使用的内存空间的设置.JVM在启动的时候会自动设...
pyspark写入mysql内存溢出_小数据玩转pyspark（1）
weixin_36212212的博客
02-09 398
sqoop：导入结构化数据kafka：导入流式数据HDFS：文件存储形式(数据存在Hadoop上是，存在HDFS)怎么访问呢？最传统的肯定是MR，后期有Hive(其实就是把MR通过sql转换了一下，Hive本身并没有存储功能，存储还是HDFS)，现在也可以用spark进行数据操作(spark SQL)，PIG是一种脚本式操作语言，可以直接操作HDFSHBASE：键值对存储形式(MPP的数据库)MR...
HDF.zip_hdf
09-24
HDF（Hierarchical Data Format）是一种用于存储和组织大量科学数据的文件格式。它由美国国家超级计算应用中心（NCSA）开发，设计目的是为了高效地存储和检索大规模的多维数组和非结构化数据。HDF5是HDF的最新版本，...
大数据技术之_19_Spark学习_07_Spark 性能调优 + 数据倾斜调优 + 运行资源调优 + 程序开发调优 + Shuffle 调优 + GC 调优 + Spark 企业应用案例
黑泽君
05-04 2035
大数据技术之_19_Spark学习_07_Spark 性能调优 + 数据倾斜调优 + 运行资源调优 + 程序开发调优 + Shuffle 调优 + GC 调优 + Spark 企业应用案例
文件上传漏洞
m0_55772907的博客
04-28 3219
什么是文件上传漏洞 文件上传漏洞是指由于程序员在对用户文件上传部分的控制不足或者处理缺陷，而导致的用户可以越过其本身权限向服务器上上传可执行的动态脚本文件。这里上传的文件可以是木马，病毒，恶意脚本或者WebShell等。“文件上传”本身没有问题，有问题的是文件上传后，服务器怎么处理、解释文件。如果服务器的处理逻辑做的不够安全，则会导致严重的后果。 什么是webshell WebShell就是以asp、php、jsp或者cgi等网页文件形式存在的一种命令执行环境，也可以将其称之为一种网页后门。攻击者
Diagnostics: File file:/tmp/spark-***/__spark_libs__***.zip does not exist
weixin_30706691的博客
07-04 446
Diagnostics: File file:/tmp/spark-c03df206-c90e-4c97-a2d6-a5d3fdb17811/__spark_libs__303213348409500965.zip does not exist 使用了cluster和client的deploy mode都不好使。期间使用cluster模式下，还报出了jar包找不到的异常。。 ...
pySpark ModuleNotFoundError: No module named ‘XXX‘
最新发布
Han_Lin-的博客
12-20 1348
pySpark ModuleNotFoundError: No module named 'XXX'
pyspark在yarn上运行出现的问题
a857553315的博客
09-07 1062
在yarn上运行使用Python编写的spark的任务代码
zip命令的坑 pyspark --py-files导入zip包不生效 / python导入zip包不生效
热门推荐
我们仍旧还是那年追逐着C++的少年的影子
01-21 1万+
背景 pyspark作业在cluster模式提交时候除了执行的文件外，如果有其他依赖的话，需要用–py-files导入，如果是导入一个python包(带__init__.py的文件夹)的话，需要先打包成zip包。(只是一种方式，也可以打包环境上传HDFS，和本文无关不多说) 如: 想要调用的request包如果dn节点的机器上没有的话，可以把源码直接打成zip然后才可以调用 --py-files ...
2.11.3PySpark--安装
寒暄的博客
03-27 532
总目录：https://www.jianshu.com/p/e406a9bc93a9 Python - 子目录：https://www.jianshu.com/p/50b432cb9460 全称Python on Spark，使用spark提供的pyspark库编写spark应用。 pyspark的安装很特殊，为了保持版本一致性，需要从我们spark的安装目录中的Python目录下...
spark三种清理数据的方式：UDF，自定义函数，spark.sql；Python中的zip()与*zip()函数详解//及python中的*args和**kwargs
sinat_26566137的博客
07-30 1278
（1）UDF的方式清理数据 import sys reload(sys) sys.setdefaultencoding('utf8') import re import json from pyspark.sql import SparkSession from pyspark.sql import Row from pyspark.sql.functions import udf fro...
pyspark安装和使用
weixin_44768806的博客
03-11 2075
PySpark笔记 PySpark:Python on Spark 基于python语言,使用spark提供的pythonAPI库，进行spark应用程序的编程 ============================================================== 一、搭建PySpark的环境 1.windows上进行pyspark环境配置 步骤： -1.在windows上搭...
Node.js文件操作：读取与写入实战解析
- **文件创建**：使用`fs.writeFile()`在写入时会自动创建文件，如果需要确保文件存在，可以使用`fs.mkdirSync()`创建目录，`fs.openSync()`创建文件并返回文件描述符（fd）。 - **文件追加**：若要追加内容而不是...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值