异常-User class threw exception: java.lang.IllegalStateException: Cannot call methods on a stopped Spa...

最新推荐文章于 2023-09-09 05:19:40 发布
最新推荐文章于 2023-09-09 05:19:40 发布
阅读量1.3k 收藏
点赞数
文章标签： java 大数据 scala
原文链接：http://www.cnblogs.com/QuestionsZhang/p/10840538.html
版权

1 详细信息

User class threw exception: java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.

This stopped SparkContext was created at:
 
org.apache.spark.SparkContext.<init>(SparkContext.scala:76)
com.wm.bigdata.spark.etl.RentOrderDistributedEtl$.main(RentOrderDistributedEtl.scala:227)
com.wm.bigdata.spark.etl.RentOrderDistributedEtl.main(RentOrderDistributedEtl.scala)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:678)
 
The currently active SparkContext was created at:
 
(No active SparkContext.)
 
at org.apache.spark.SparkContext.assertNotStopped(SparkContext.scala:100)
at org.apache.spark.SparkContext$$anonfun$newAPIHadoopRDD$1.apply(SparkContext.scala:1186)
at org.apache.spark.SparkContext$$anonfun$newAPIHadoopRDD$1.apply(SparkContext.scala:1185)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.SparkContext.withScope(SparkContext.scala:699)
at org.apache.spark.SparkContext.newAPIHadoopRDD(SparkContext.scala:1185)
at org.apache.phoenix.spark.PhoenixRDD.<init>(PhoenixRDD.scala:49)
at org.apache.phoenix.spark.PhoenixRelation.buildScan(PhoenixRelation.scala:39)
at org.apache.spark.sql.execution.datasources.DataSourceStrategy$$anonfun$10.apply(DataSourceStrategy.scala:293)
at org.apache.spark.sql.execution.datasources.DataSourceStrategy$$anonfun$10.apply(DataSourceStrategy.scala:293)
at org.apache.spark.sql.execution.datasources.DataSourceStrategy$$anonfun$pruneFilterProject$1.apply(DataSourceStrategy.scala:326)
at org.apache.spark.sql.execution.datasources.DataSourceStrategy$$anonfun$pruneFilterProject$1.apply(DataSourceStrategy.scala:325)
at org.apache.spark.sql.execution.datasources.DataSourceStrategy.pruneFilterProjectRaw(DataSourceStrategy.scala:381)
at org.apache.spark.sql.execution.datasources.DataSourceStrategy.pruneFilterProject(DataSourceStrategy.scala:321)
at org.apache.spark.sql.execution.datasources.DataSourceStrategy.apply(DataSourceStrategy.scala:289)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:63)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:63)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:93)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2$$anonfun$apply$2.apply(QueryPlanner.scala:78)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2$$anonfun$apply$2.apply(QueryPlanner.scala:75)
at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1334)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2.apply(QueryPlanner.scala:75)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2.apply(QueryPlanner.scala:67)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:93)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2$$anonfun$apply$2.apply(QueryPlanner.scala:78)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2$$anonfun$apply$2.apply(QueryPlanner.scala:75)
at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1334)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2.apply(QueryPlanner.scala:75)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2.apply(QueryPlanner.scala:67)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:93)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2$$anonfun$apply$2.apply(QueryPlanner.scala:78)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2$$anonfun$apply$2.apply(QueryPlanner.scala:75)
at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1334)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2.apply(QueryPlanner.scala:75)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2.apply(QueryPlanner.scala:67)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:93)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2$$anonfun$apply$2.apply(QueryPlanner.scala:78)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2$$anonfun$apply$2.apply(QueryPlanner.scala:75)
at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1334)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2.apply(QueryPlanner.scala:75)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2.apply(QueryPlanner.scala:67)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:93)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2$$anonfun$apply$2.apply(QueryPlanner.scala:78)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2$$anonfun$apply$2.apply(QueryPlanner.scala:75)
at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1334)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2.apply(QueryPlanner.scala:75)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2.apply(QueryPlanner.scala:67)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:93)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2$$anonfun$apply$2.apply(QueryPlanner.scala:78)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2$$anonfun$apply$2.apply(QueryPlanner.scala:75)
at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1334)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2.apply(QueryPlanner.scala:75)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2.apply(QueryPlanner.scala:67)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:93)
at org.apache.spark.sql.execution.QueryExecution.sparkPlan$lzycompute(QueryExecution.scala:72)
at org.apache.spark.sql.execution.QueryExecution.sparkPlan(QueryExecution.scala:68)
at org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:77)
at org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:77)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$toString$3.apply(QueryExecution.scala:207)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$toString$3.apply(QueryExecution.scala:207)
at org.apache.spark.sql.execution.QueryExecution.stringOrError(QueryExecution.scala:99)
at org.apache.spark.sql.execution.QueryExecution.toString(QueryExecution.scala:207)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:75)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.Dataset.withNewRDDExecutionId(Dataset.scala:3346)
at org.apache.spark.sql.Dataset.foreach(Dataset.scala:2716)
at com.wm.bigdata.spark.etl.RentOrderDistributedEtl$$anonfun$main$1$$anonfun$apply$mcVJ$sp$1.apply(RentOrderDistributedEtl.scala:284)
at com.wm.bigdata.spark.etl.RentOrderDistributedEtl$$anonfun$main$1$$anonfun$apply$mcVJ$sp$1.apply(RentOrderDistributedEtl.scala:263)
at scala.collection.immutable.List.foreach(List.scala:392)
at com.wm.bigdata.spark.etl.RentOrderDistributedEtl$$anonfun$main$1.apply$mcVJ$sp(RentOrderDistributedEtl.scala:263)
at com.wm.bigdata.spark.etl.RentOrderDistributedEtl$$anonfun$main$1.apply(RentOrderDistributedEtl.scala:262)
at com.wm.bigdata.spark.etl.RentOrderDistributedEtl$$anonfun$main$1.apply(RentOrderDistributedEtl.scala:262)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at com.wm.bigdata.spark.etl.RentOrderDistributedEtl$.main(RentOrderDistributedEtl.scala:262)
at com.wm.bigdata.spark.etl.RentOrderDistributedEtl.main(RentOrderDistributedEtl.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:678)
 
错误原因
　　自己关闭资源的位置写在任务循环代码之内了 
sqlContext.sparkSession.close()
sc.stop()

详细检查写结束的位置，写在最后，解决问题。低级失误

转载于:https://www.cnblogs.com/QuestionsZhang/p/10840538.html

a1346277
关注
zeppelin报错 java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext. This stop
yy的博客
05-19 5591
报错:java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext. This stopped SparkContext was created at: 原因之一是之前你在spark webui里面吧zeppelin给kill掉了;zeppelin以为他自己还连着spark,直接就去拿sparkContext,然而并没有,所以报错; 解决方法: zeppelin页面,右上角点击,选int...
structured streaming getOrCreate报错：java.lang.IllegalStateException: Cannot call methods on a stopped
yy的博客
04-26 560
解决办法： result.writeStream.... 之后加一行阻塞spark等待任务结束 spark.streams.awaitAnyTermination()
User class threw exception: java.lang.UnsupportedOperationException: empty collection
可乐
10-13 1230
原因： spark代码中有访问这个表：t_sd_xxxx 表的生命周期只有十天，然后生命周期把历史的数据清理了 这个表里的分区已经没有了 20/10/13 20:43:36 ERROR ApplicationMaster: User class threw exception: java.lang.UnsupportedOperationException: empty collection java.lang.UnsupportedOperationException: empty collection
User class threw exception: org.apache.spark.SparkException: Task not serializable问题解决
LZX的博客
02-22 3191
问题描述： 计划将sparksql查询结果数据通过alibaba.druid.pool.存入mysql，结果报错序列化问题 User class threw exception: org.apache.spark.SparkException: Task not serializable at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:345) at org.apache.spark.util
Zeppelin错误：无法在已停止的SparkContext上调用方法。停止SparkContext
Book_Sea的博客
09-09 420
为了解决这个错误，我们可以在调用SparkContext的方法之前检查其状态，或者在每次任务执行之前创建一个新的SparkContext实例。首先，让我们了解一下SparkContext是什么。解决"Cannot call methods on a stopped SparkContext"错误的一种方法是确保在调用SparkContext的方法之前，检查SparkContext的状态。这样，我们可以确保始终有一个有效的SparkContext，并避免调用已停止的SparkContext上的方法。
Spark代码提交到Yarn报错：java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/java8
大数据跟我学i
03-19 1536
保存信息： ERROR Client: Application diagnostics message: User class threw exception: java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/java8/JFunction2$mcIII$sp at com.czxy.WordCount_Spark$.main(WordCount_Spark.scala:29)
记录:java.lang.IllegalStateException: Cannot call sendError() after the response has...【解决方案】
热门推荐
**My Coding Family**
03-20 1万+
记录:java.lang.IllegalStateException: Cannot call sendError() after the response has...【解决方案】
ERROR yarn.ApplicationMaster: User class threw exception: java.lang.OutOfMemoryError: Java heap space
最新发布
11-03
该错误是由于Java堆内存不足导致的。可以通过调整JVM的堆内存大小来解决该问题。根据引用，可以将JVM heap设置为4G或更高，同时将-Xmx和-Xms设置为相同的值，以减少内存抖动带来的性能影响。另外，根据引用，也可以...
User class threw exception: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog':
07-25
这个问题是由于在实例化'org.apache.spark.sql.hive.HiveExternalCatalog'时出现了错误导致的。可能的原因是缺少必要的依赖项或配置不正确。您可以尝试以下步骤来解决这个问题： 1. 确保您的项目中包含了所有必需的...
spark User class threw exception: java.lang.NoSuchMethodError
bitcarmanlee的博客
02-08 3570
在spark 集群上运行job的时候，经常会出现类似如下错误提示： User class threw exception: java.lang.NoSuchMethodError: org.apache.spark.WritableConverter$.writableWritableConverterFn(Lscala/reflect/ClassTag;)Lscala/Function0; at...
Spark User class threw exception: java.sql.SQLException: No suitable driver
yisun123456的博客
12-26 1695
Spark on Yarn:java.sql.SQLException: No suitable driver found for jdbc:microsoft:sqlserver://localhost\\db_instance_name:1433;databaseName=db_name 本文只是针对当前特定环境下，出现的问题找不到sqljdbc驱动的案例。具体出现原因，可能是spark版本...
User class threw exception: org.apache.hadoop.mapred.InvalidInputException: Input path does not exis
我的博客
07-19 693
临时处理个小任务，结果要放到hdfs中。 刚开始是设置传参 ： 任务提交的时候在脚本中传参，但是这种一运行就报错：说路径已经存在了！！！试了几次，已进行传参就报已存在的错误。 解决办法：自己直接在config.properties中给它设置死了，不进行传参，要修改在properties中修改！！！ ...
User class threw exception: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
weixin_43621063的博客
06-04 872
User class threw exception: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver1. 增加依赖2. 允许hive 1. 增加依赖 <dependency> <groupId>mysql</groupId> <artifactId&g...
出现java.lang.IllegalStateException的几种情况
lalaandbei的专栏
10-23 961
抛出异常：ERROR [Engine] StandardWrapperValve[jsp]: Servlet.service() for servlet jsp threw exceptionjava.lang.IllegalStateException: getOutputStream() has already been called for this response综合分析原因如下：这是w
【pyspark】No active SparkContext
littlemichelle
08-14 3550
运行pyspark的时候出现了以下错误： Py4JJavaError: An error occurred while calling o2198.csv. : java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext. This stopped SparkContext was created at: pyspark java.lang.IllegalStateException: Cann.
解决java.lang.IllegalStateException: Cannot call sendError() after the response has been committe
oak_javaLearner的博客
09-03 3716
错误截图： 错误代码： 错误原因：输出流关闭之后，socket也已经关闭，不能再次发送response，所以导致错误的就是return的返回值信息，这时候把return res;改成return null;就可以了，这样就不会因为需要返回信息而需要再次发送response，错误就解决了 附上参考的大佬博客：点击跳转 ...
TasksetManager冲突导致SparkContext异常关闭
liujianhuiouc的专栏
03-16 2524
背景介绍 当正在悠闲敲着代码的时候，业务方兄弟反馈接收到大量线上运行的spark streaming任务的告警短信，查看应用的web页面信息，发现spark应用已经退出了，第一时间拉起线上的应用，再慢慢的定位故障原因。本文代码基于spark 1.6.1。 问题定位 登陆到线上机器，查看错误日志，发现系统一直报Cannot call methods on a stopped Spark
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值