spark on yarn 运行报错 prelaunch.err.

最新推荐文章于 2023-12-20 12:06:44 发布
最新推荐文章于 2023-12-20 12:06:44 发布
阅读量6.3k 收藏 2
点赞数 2
分类专栏： Spark 文章标签： spark on yarn
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/dudefu011/article/details/88874553
版权

错误代码如下：

[2019-03-28 17:50:04.728]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
/hadoop/yarn/local/usercache/defu.du/appcache/application_1553763476904_0007/container_e03_1553763476904_0007_02_000005/launch_container.sh:行33: $PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:$HADOOP_CONF_DIR:/usr/hdp/3.1.0.0-78/hadoop/*:/usr/hdp/3.1.0.0-78/hadoop/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:$PWD/__spark_conf__/__hadoop_conf__: 坏的替换

[2019-03-28 17:50:04.729]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
/hadoop/yarn/local/usercache/defu.du/appcache/application_1553763476904_0007/container_e03_1553763476904_0007_02_000005/launch_container.sh:行33: $PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:$HADOOP_CONF_DIR:/usr/hdp/3.1.0.0-78/hadoop/*:/usr/hdp/3.1.0.0-78/hadoop/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:$PWD/__spark_conf__/__hadoop_conf__: 坏的替换


19/03/28 17:50:02 INFO BlockManagerMaster: Removal of executor 4 requested
19/03/28 17:50:02 INFO BlockManagerMasterEndpoint: Trying to remove executor 4 from BlockManagerMaster.
19/03/28 17:50:02 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 4
19/03/28 17:50:07 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/03/28 17:50:09 ERROR YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!
19/03/28 17:50:09 INFO SparkUI: Stopped Spark web UI at http://10.24.2.247:4040
19/03/28 17:50:09 INFO DAGScheduler: Job 0 failed: collect at wordcount.scala:18, took 22.786662 s
Exception in thread "main" org.apache.spark.SparkException: Job 0 cancelled because SparkContext was shut down
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:837)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:835)
	at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
	at org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:835)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onStop(DAGScheduler.scala:1900)
	at org.apache.spark.util.EventLoop.stop(EventLoop.scala:83)
	at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1813)
	at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1931)
	at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1361)
	at org.apache.spark.SparkContext.stop(SparkContext.scala:1930)
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:112)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2055)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2074)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
	at org.apache.spark.rdd.RDD.collect(RDD.scala:944)
	at com.xinyi.scala.wordcount$.main(wordcount.scala:18)
	at com.xinyi.scala.wordcount.main(wordcount.scala)
19/03/28 17:50:09 INFO DAGScheduler: ShuffleMapStage 0 (map at wordcount.scala:17) failed in 22.611 s due to Stage cancelled because SparkContext was shut down
19/03/28 17:50:09 INFO DiskBlockManager: Shutdown hook called
19/03/28 17:50:09 ERROR TransportClient: Failed to send RPC 5014435794729518644 to /10.24.5.209:41900: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
	at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
19/03/28 17:50:09 INFO ShutdownHookManager: Shutdown hook called
19/03/28 17:50:09 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful
java.io.IOException: Failed to send RPC 5014435794729518644 to /10.24.5.209:41900: java.nio.channels.ClosedChannelException
	at org.apache.spark.network.client.TransportClient.lambda$sendRpc$2(TransportClient.java:237)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
	at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
	at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:431)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
	at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
19/03/28 17:50:09 INFO ShutdownHookManager: Deleting directory C:\Users\defu.du\AppData\Local\Temp\spark-f7b7718f-582b-4c97-b06a-0e53c0431f73
19/03/28 17:50:09 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
19/03/28 17:50:09 ERROR Utils: Uncaught exception in thread Yarn application state monitor
org.apache.spark.SparkException: Exception thrown in awaitResult: 
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
	at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
	at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.requestTotalExecutors(CoarseGrainedSchedulerBackend.scala:567)
	at org.apache.spark.scheduler.cluster.YarnSchedulerBackend.stop(YarnSchedulerBackend.scala:95)
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:155)
	at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:508)
	at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1814)
	at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1931)
	at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1361)
	at org.apache.spark.SparkContext.stop(SparkContext.scala:1930)
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:112)
Caused by: java.io.IOException: Failed to send RPC 5014435794729518644 to /10.24.5.209:41900: java.nio.channels.ClosedChannelException
	at org.apache.spark.network.client.TransportClient.lambda$sendRpc$2(TransportClient.java:237)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
	at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
	at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:431)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
	at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
19/03/28 17:50:09 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

解决方案：
设置参数：set(“spark.yarn.am.extraJavaOptions”,"-Dhdp.version=3.1.0.0-78")

夜下探戈
关注
  • 2
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
Spark on Yarn模式部署.docx
10-03
Spark on Yarn 模式部署是一种常见的 Spark 应用场景，本文将详细介绍 Spark on Yarn 模式部署的步骤和配置过程。 标题解释 Spark on Yarn 模式部署是指将 Spark 应用程序部署在 Yarn 集群上，使得 Spark 能够使用 ...
Container exited with a non-zero exit code 1. Error file: prelaunch.err.org.apache.hadoop.mapreduce.
热门推荐
colby_chenlun的博客
05-31 2万+
这两天闲的慌，装了一下新版本的HADOOP 3.2.1，的HA，装好之后，测试wordcount程序时报下面的错误 For more detailed output, check the application tracking page: http://hadoop102:8088/cluster/app/application_1590917926429_0002 Then click on links to logs of each attempt. . Failing the applicati
集群配置的错误
qq_41927934的博客
11-05 2706
在集群上运行jar程序的时候，出现以下这个错误信息： Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild 解决方法 1.打开配置文件yarn
运行Mapreduce集群时候出现报错：Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 40
未来社会二十年发展的核心技术趋势由ABCD四个字母组成，分别是AI（人工智能）、BlockChain（区块链）、Cloud（云）、和Data（大数据）每一次进步都有新的认知和感触
11-20 3333
运行Mapreduce集群时候出现报错：Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 40...............，本文主要针对该问题进行详细的图文说明帮助解决该问题。
hadoop集群迁移， hive任务执行报错：Container exited with a non-zero exit code 1. Error file: prelaunch.err
A_single_cat的博客
12-02 5006
hadoop 集群迁移 ， apache原生版本迁移到cdh版本时 ， hive任务执行报错： Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : 因为未改动脚本， 以为是集群问题，期间尝试了以下方法， 发现并无效果： #在mapred-site.xml文件中添加mapreduce
MapReduce程序运行异常Container exited with a non-zero exit code 1. Error file: prelaunch.err.
qq_38780623的博客
11-21 1万+
Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Hadoop mapreduce 报错:Container exited with a non-zero exit code 127. Error file: prelaunch.err.
Natsu_natsu的博客
10-16 1113
Hadoop mapreduce 报错:Container exited with a non-zero exit code 127. Error file: prelaunch.err.
SPARK2_ON_YARN-2.4.0.cloudera2.jar
03-10
SPARK2_ON_YARN-2.4.0 jar包下载
spark yarn模式的搭建.docx
06-15
Spark On Yarn完全分布式集群环境搭建文档。 分为如下几部分： 1、环境的准备; 2、Zookeeper完全分布式搭建; 3、Hadoop2.0 HA集群搭建步骤介绍; 4、Spark On Yarn搭建介绍; 5、集群启动介绍; 最新最全的java培训视频...
Sparkonyarn集群搭建详细过程.pdf
10-15
Spark on YARN 集群搭建详细过程 _title_：Spark on YARN 集群搭建详细过程 _description_：本文详细介绍了 Spark on YARN 集群搭建的过程，包括配置主机 hosts 文件、免密码登录、安装 Java、安装 Scala 等步骤。...
yarn-v0.23.2.tar.gz
07-22
yarn-v0.23.2.tar.gz 在安装ambari,源码编译的时候下载的文件有问题 手动下载 地址 https://github.com/yarnpkg/yarn/releases/download/v0.23.2/yarn-v0.23.2.tar.gz
hadoop在本地测试之后，将jar包上传到集群中报错 Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : /opt/modul
qq2259583140的博客
10-30 213
输入hadoop classpath。修改mapred-site.xml。如图，把classpath放到其中。
spark开发问题汇总
盛源的博客
09-23 3413
1 可能导致内存溢出的聚合函数 1 错误代码 Container exited with a non-zero exit code 134. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err 2 函数 collect_set 、 collect_list 3 原因 某个key值(null, unknown, 空串)过多, 被聚合的value值过多, 导致范围(数组超范围)超限; 3 解决方法 1) 调大内存 2) .
执行hadoop-3.3.0 wc 报错Container exited with a non-zero exit code 1. Error file: prelaunch.err.
top_explore的博客
01-27 7673
报错 Application application_1611732673035_0001 failed 2 times due to AM Container for appattempt_1611732673035_0001_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2021-01-27 02:32:14.002]Exception from container-launch. Container id:
Flink任务提交问题汇总
微信搜：import_bigdata，大数据领域硬核原创作者
02-24 1853
全网最全大数据面试提升手册！flink 任务提交问题汇总1. 提交flink自带任务WordCount.jar遇到的问题：2. 提交flink 批处理任务时遇到的问题3. flink定时任务，mysql连接超时问题4. flink checkpoint 恢复失败5. yarn 增加并行任务数量配置6. 流处理flink程序在hadoop集群跑了一段时间莫名挂掉1. 提交flink自带任务WordC...
Flink中遇到的问题
index_test的博客
02-24 2629
解决遇到的一些问题
spark-streaming kafka Blacklisting behavior can be configured via spark.blacklist.*.
剑雨江湖
03-08 4226
[2022-03-08 15:23:14.742]Container exited with a non-zero exit code 50. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/sr.
Spark standalone和yarn模式安装部署
不以物喜的博客
05-15 1038
1 安装 1） 下载 https://archive.apache.org/dist/spark/ 2） 解压下载安装包 tar -zxvf spark-2.1.1-bin-hadoop2.7.tgz 2 standalone模式 2.1 说明 构建一个由 Master + Slave 构成的 Spark 集群，Spark 运行在集群中。这个要和 Hadoop 中的 Standalone 区别开来. 这里的 Standalone 是指只用 Spark 来搭建一个集群, 不需要借助其他的框架.是相对于 Yar
Container exited with a non-zero exit code 13. Error file: prelaunch.err.
最新发布
窦永厚
12-20 1208
spark提交报这个错，主要原因是你在代码中指的的是local模式，二用spark-submit提交是指定了yarn集群模式，所以报这个错了，他找不到你的yarn集群。
W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
06-09
这个问题通常是由于 TensorFlow 数据集加载时出错导致的。你可以尝试以下几种方法来解决问题： 1. 检查数据集路径是否正确，并确保数据集文件存在。 2. 尝试重新下载数据集并重新运行代码。 3. 检查 TensorFlow 版本是否最新，并且升级到最新版本。 4. 检查是否有足够的内存和磁盘空间来加载数据集。 5. 尝试使用其他数据集加载方法，例如 tf.data.TFRecordDataset()。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值