spark.SparkContext: Error initializing SparkContext.

最新推荐文章于 2023-12-05 14:30:41 发布
最新推荐文章于 2023-12-05 14:30:41 发布
阅读量1.8k 收藏
点赞数 1
分类专栏： 大数据 文章标签： spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/dragon_hooray/article/details/107044004
版权

spark.SparkContext: Error initializing SparkContext.

报错：

(base) [bigdata@worker01 ~]$ spark2-shell 
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
20/06/30 15:40:49 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Exception when registering SparkListener
	at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2364)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:553)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
	at org.apache.spark.repl.Main$.createSparkSession(Main.scala:103)
	at $line3.$read$$iw$$iw.<init>(<console>:15)
	at $line3.$read$$iw.<init>(<console>:43)
	at $line3.$read.<init>(<console>:45)
	at $line3.$read$.<init>(<console>:49)
	at $line3.$read$.<clinit>(<console>)
	at $line3.$eval$.$print$lzycompute(<console>:7)
	at $line3.$eval$.$print(<console>:6)
	at $line3.$eval.$print(<console>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
	at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
	at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
	at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
	at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
	at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
	at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
	at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
	at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
	at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
	at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(SparkILoop.scala:79)
	at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(SparkILoop.scala:79)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SparkILoop.scala:79)
	at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:79)
	at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:79)
	at scala.tools.nsc.interpreter.ILoop.savingReplayStack(ILoop.scala:91)
	at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:78)
	at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:78)
	at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:78)
	at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
	at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:77)
	at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:110)
	at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)
	at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
	at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
	at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
	at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
	at org.apache.spark.repl.Main$.doMain(Main.scala:76)
	at org.apache.spark.repl.Main$.main(Main.scala:56)
	at org.apache.spark.repl.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:892)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.FileNotFoundException: Lineage directory /var/log/spark2/lineage doesn't exist or is not writable.
	at com.cloudera.spark.lineage.LineageWriter$.checkLineageConfig(LineageWriter.scala:158)
	at com.cloudera.spark.lineage.NavigatorAppListener.<init>(ClouderaNavigatorListener.scala:30)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2740)
	at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2732)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
	at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
	at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
	at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2732)
	at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2353)
	at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2352)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2352)
	... 62 more
org.apache.spark.SparkException: Exception when registering SparkListener
  at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2364)
  at org.apache.spark.SparkContext.<init>(SparkContext.scala:553)
  at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
  at org.apache.spark.repl.Main$.createSparkSession(Main.scala:103)
  ... 55 elided
Caused by: java.io.FileNotFoundException: Lineage directory /var/log/spark2/lineage doesn't exist or is not writable.
  at com.cloudera.spark.lineage.LineageWriter$.checkLineageConfig(LineageWriter.scala:158)
  at com.cloudera.spark.lineage.NavigatorAppListener.<init>(ClouderaNavigatorListener.scala:30)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2740)
  at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2732)
  at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
  at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
  at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
  at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
  at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
  at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2732)
  at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2353)
  at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2352)
  at scala.Option.foreach(Option.scala:257)
  at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2352)
  ... 62 more
<console>:14: error: not found: value spark
       import spark.implicits._
              ^
<console>:14: error: not found: value spark
       import spark.sql
              ^
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.3.0.cloudera2
      /_/
         
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_162)
Type in expressions to have them evaluated.
Type :help for more information.

scala>

解决:赋予他写的权限

chmod 777 /var/log/spark2/lineage

完成！

ERROR spark.SparkContext:Error initializing SparkContext.Call From to failed on connection exception
zhengzaifeidelushang的博客
11-11 1397
Spark集群搭建完，节点启动Spark-shell报错 报错如下： bigdata1节点启动Spark-shell报错： ERROR repl.Main: Failed to initialize Spark session.java.net.ConnectException: Call From bigdata1/192.168.239.131 to bigdata1:8020 failed on connection exception: java.net.ConnectException: 拒绝连接
Spark应用启动报错：Error initializing SparkContext. java.lang.IllegalArgumentException: System memory 25952
甘蓝的专栏
07-18 606
源码层面分析并解决Spark应用启动报错：Error initializing SparkContext. java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859
忽然遇到报错：ERROR spark.SparkContext: Error initializing SparkContext.
热门推荐
matiji66
05-06 2万+
java.lang.IllegalArgumentException: System memory 100663296 must be at least 4.718592E8. Please use a larger heap size. 在Eclipse里开发Spark项目，尝试直接在spark里运行程序的时候，遇到下面这个报错： 很明显，这是JVM申请的memory不够
ERROR SparkContext: Error initializing SparkContext.
吃橘子
06-14 5223
在Intellij idea中测试写好的Spark程序,运行时报如下错误: Using Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties 20/06/14 21:45:56 INFO SparkContext: Running Spark version 2.3.4 20/06/14 21:45:57 ERROR SparkContext: Error initializing SparkContext. or
ERROR SparkContext: Error initializing SparkContext. （）
yujinlong2002的博客
03-20 2284
在配置yarn的配置文件中设置的内存过小，不能够支持spark程序的运行所以出现了报错。在将spark程序提交到yarn中运行的时出现以下报错。
spark报错：ERROR SparkContext: Error initializing SparkContext.
Big_data_implicit_
11-28 3456
今天刚好spark history server 配好，就出现了这个错误：ERROR SparkContext: Error initializing SparkContext. 最后一再仔细看，才发现是spark conf下的spark-defaults.conf文件中的错误，把hdfs搞错了，正确的应该是： spark.eventLog.dir hdfs://linux01.pub:9000/spark-logs ...
Spark ERROR SparkContext: Error initializing SparkContext. Name node is in safe mode.
m0_56528620的博客
04-02 2583
ERROR SparkContext: Error initializing SparkContext. org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create file/sparklog/local-1648871203592.lz4.inprogress. Name node is in safe mode.
解决ERROR spark.SparkContext: Error initializing SparkContext
mo
11-25 8493
今天在开启spark-shell时报了如下错误: 从报错的日志上面来看可以知道它说SparkContext初始化异常，并且和hadoop的端口通信的时候显示连接失败。刚开始的时候我检查了启动的spark进程发现也没问题。 [root@hadoop102 logs]# showjps.sh ===================== root@hadoop102 ======================= 7315 Worker 7395 Jps 7237 Master ===============
提交spark程序到yarn出现ERROR SparkContext: Error initializing SparkContext.
piduzi的博客
02-26 1万+
命令行输出的java异常栈ERROR SparkContext: Error initializing SparkContext.org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. ...
ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Could not pars
qq_53810226的博客
03-14 5950
ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Could not parse Master URL: ‘<pyspark.conf.SparkConf object at 0x7fb5da40a850>’ 报错信息 22/03/14 10:58:26 WARN NativeCodeLoader: Unable to load native-hadoop library f
spark报错：SparkContext: Error initializing SparkContext
啊帅和和的博客
11-28 6931
今天早上在写一个.scala代码时，控制台一直出现这个错误。 原本以为这个错误是找不到URL地址，不能使用local[*]这种模式，后来发现这个错误的根本是在上一行，也就是： SparkContext: Error initializing SparkContext 原因是初始化SparkContext时出现的问题。 1.val config:SparkConf = new SparkConf().setMaster(“Local[*]”).setAppName(“ahehe”) 2.val sc = n
ERROR SparkContext: Error initializing SparkContext.org.apache.spark.SparkException：无效的 Spark URL：
最新发布
HYSliuliuliu的博客
12-05 1万+
ERROR SparkContext: Error initializing SparkContext.org.apache.spark.SparkException：无效的 Spark URL：在idea中用scala编写spark的WordCount的案例时，老是报错，scala能运行其他程序scala没有问题，依赖按照视频教程也成功下载，Maven仓库也下载的巴巴适适，报的错误太多了，我节选有价值的错误信息：
spark Shell启动的时候报错：ERROR spark.SparkContext: Error initializing SparkContext.ConnectException
技术探求
07-18 5551
spark Shell启动的时候报错：ERROR spark.SparkContext: Error initializing SparkContext.ConnectException: Call From hadoop202/192.168.1.202 to hadoop202:9000 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: spark连接HDFS拒绝连接，其实是犯了一
SparkContext: Error initializing SparkContext解决方法
koooooooo5的博客
11-01 2万+
SparkContext: Error initializing SparkContext解决方法 Spark在配置高可用集群中，出现报错 ERROR SparkContext: Error initializing SparkContext. java.net.ConnectException: Call From hadoop102/192.168.10.102 to hadoop102:8020 failed on connection exception: java.net.ConnectExcep
Spark 报错解决--Error initializing SparkContext
weixin_34049948的博客
05-14 1万+
在提交spark作业的时候，spark出现报错 ./spark-shell 19/05/14 05:37:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Setting defau...
Spark运行出现Error initializing SparkContext
qq_37543213的博客
05-12 3688
Spark运行出现Error initializing SparkContext1.问题描述2.问题解决 1.问题描述 idea中运行spark时，创建SparkConf和创建创建SparkContext报错 ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Invalid Spark URL: spark://HeartbeatReceiver@xxxxxxxxxxx:4126 2.
ERROR spark.SparkContext: Error initializing SparkContext
qq_32253371的博客
08-09 1646
ERROR spark.SparkContext: Error initializing SparkContext. org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby 解决
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

总要冲动一次

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值