SparkShell 启动报错，在线等大神解决，急！！！！

最新推荐文章于 2024-02-26 22:34:50 发布
最新推荐文章于 2024-02-26 22:34:50 发布
阅读量1.6k 收藏
点赞数 1
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/Snow_Face/article/details/79410309
版权
[root@hadoop01 bin]# ./spark-shell --master spark://hadoop
log4j:WARN No appenders could be found for logger (org.apa
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.htm
Using Spark's repl log4j profile: org/apache/spark/log4j-d
To adjust logging level use sc.setLogLevel("INFO")
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.6.3
/_/

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server
Type in expressions to have them evaluated.
Type :help for more information.
18/03/01 09:49:58 ERROR SparkDeploySchedulerBackend: Appli
18/03/01 09:49:58 WARN SparkDeploySchedulerBackend: Applic
18/03/01 09:49:58 WARN AppClient$ClientEndpoint: Drop Unre
18/03/01 09:49:58 ERROR MapOutputTrackerMaster: Error comm
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchr
at scala.concurrent.impl.Promise$DefaultPromise.tr
at scala.concurrent.impl.Promise$DefaultPromise.re
at scala.concurrent.impl.Promise$DefaultPromise.re
at scala.concurrent.Await$$anonfun$result$1.apply(
at scala.concurrent.BlockContext$DefaultBlockConte
at scala.concurrent.Await$.result(package.scala:10
at org.apache.spark.rpc.RpcTimeout.awaitResult(Rpc
at org.apache.spark.rpc.RpcEndpointRef.askWithRetr
at org.apache.spark.rpc.RpcEndpointRef.askWithRetr
at org.apache.spark.MapOutputTracker.askTracker(Ma
at org.apache.spark.MapOutputTracker.sendTracker(M
at org.apache.spark.MapOutputTrackerMaster.stop(Ma
at org.apache.spark.SparkEnv.stop(SparkEnv.scala:9
at org.apache.spark.SparkContext$$anonfun$stop$12.
at org.apache.spark.util.Utils$.tryLogNonFatalErro
at org.apache.spark.SparkContext.stop(SparkContext
at org.apache.spark.scheduler.cluster.SparkDeployS
at org.apache.spark.deploy.client.AppClient$Client
at org.apache.spark.deploy.client.AppClient$Client
at java.util.concurrent.Executors$RunnableAdapter.
at java.util.concurrent.FutureTask.runAndReset(Fut
at java.util.concurrent.ScheduledThreadPoolExecuto
at java.util.concurrent.ScheduledThreadPoolExecuto
at java.util.concurrent.ThreadPoolExecutor.runWork
at java.util.concurrent.ThreadPoolExecutor$Worker.
at java.lang.Thread.run(Thread.java:748)
18/03/01 09:49:59 ERROR Utils: Uncaught exception in threa
org.apache.spark.SparkException: Error communicating with
at org.apache.spark.MapOutputTracker.askTracker(Ma
at org.apache.spark.MapOutputTracker.sendTracker(M
at org.apache.spark.MapOutputTrackerMaster.stop(Ma
at org.apache.spark.SparkEnv.stop(SparkEnv.scala:9
at org.apache.spark.SparkContext$$anonfun$stop$12.
at org.apache.spark.util.Utils$.tryLogNonFatalErro
at org.apache.spark.SparkContext.stop(SparkContext
at org.apache.spark.scheduler.cluster.SparkDeployS
at org.apache.spark.deploy.client.AppClient$Client
at org.apache.spark.deploy.client.AppClient$Client
at java.util.concurrent.Executors$RunnableAdapter.
at java.util.concurrent.FutureTask.runAndReset(Fut
at java.util.concurrent.ScheduledThreadPoolExecuto
at java.util.concurrent.ScheduledThreadPoolExecuto
at java.util.concurrent.ThreadPoolExecutor.runWork
at java.util.concurrent.ThreadPoolExecutor$Worker.
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchr
at scala.concurrent.impl.Promise$DefaultPromise.tr
at scala.concurrent.impl.Promise$DefaultPromise.re
at scala.concurrent.impl.Promise$DefaultPromise.re
at scala.concurrent.Await$$anonfun$result$1.apply(
at scala.concurrent.BlockContext$DefaultBlockConte
at scala.concurrent.Await$.result(package.scala:10
at org.apache.spark.rpc.RpcTimeout.awaitResult(Rpc
at org.apache.spark.rpc.RpcEndpointRef.askWithRetr
at org.apache.spark.rpc.RpcEndpointRef.askWithRetr
at org.apache.spark.MapOutputTracker.askTracker(Ma
... 16 more
18/03/01 09:50:00 ERROR SparkContext: Error initializing S
java.lang.NullPointerException
at org.apache.spark.SparkContext.<init>(SparkConte
at org.apache.spark.repl.SparkILoop.createSparkCon
at $line3.$read$$iwC$$iwC.<init>(<console>:15)
at $line3.$read$$iwC.<init>(<console>:24)
at $line3.$read.<init>(<console>:26)
at $line3.$read$.<init>(<console>:30)
at $line3.$read$.<clinit>(<console>)
at $line3.$eval$.<init>(<console>:7)
at $line3.$eval$.<clinit>(<console>)
at $line3.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Na
at sun.reflect.NativeMethodAccessorImpl.invoke(Nat
at sun.reflect.DelegatingMethodAccessorImpl.invoke
at java.lang.reflect.Method.invoke(Method.java:498
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.
at org.apache.spark.repl.SparkIMain$Request.loadAn
at org.apache.spark.repl.SparkIMain.loadAndRunReq$
at org.apache.spark.repl.SparkIMain.interpret(Spar
at org.apache.spark.repl.SparkIMain.interpret(Spar
at org.apache.spark.repl.SparkILoop.reallyInterpre
at org.apache.spark.repl.SparkILoop.interpretStart
at org.apache.spark.repl.SparkILoop.command(SparkI
at org.apache.spark.repl.SparkILoopInit$$anonfun$i
at org.apache.spark.repl.SparkILoopInit$$anonfun$i
at org.apache.spark.repl.SparkIMain.beQuietDuring(
at org.apache.spark.repl.SparkILoopInit$class.init
at org.apache.spark.repl.SparkILoop.initializeSpar
at org.apache.spark.repl.SparkILoop$$anonfun$org$aSparkILoop.scala:974)
at org.apache.spark.repl.SparkILoopInit$class.runT
at org.apache.spark.repl.SparkILoop.runThunks(Spar
at org.apache.spark.repl.SparkILoopInit$class.post
at org.apache.spark.repl.SparkILoop.postInitializa
at org.apache.spark.repl.SparkILoop$$anonfun$org$a
at org.apache.spark.repl.SparkILoop$$anonfun$org$a
at org.apache.spark.repl.SparkILoop$$anonfun$org$a
at scala.tools.nsc.util.ScalaClassLoader$.savingCo
at org.apache.spark.repl.SparkILoop.org$apache$spa
at org.apache.spark.repl.SparkILoop.process(SparkI
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Na
at sun.reflect.NativeMethodAccessorImpl.invoke(Nat
at sun.reflect.DelegatingMethodAccessorImpl.invoke
at java.lang.reflect.Method.invoke(Method.java:498
at org.apache.spark.deploy.SparkSubmit$.org$apache
at org.apache.spark.deploy.SparkSubmit$.doRunMain$
at org.apache.spark.deploy.SparkSubmit$.submit(Spa
at org.apache.spark.deploy.SparkSubmit$.main(Spark
at org.apache.spark.deploy.SparkSubmit.main(SparkS
java.lang.NullPointerException
at org.apache.spark.SparkContext.<init>(SparkConte
at org.apache.spark.repl.SparkILoop.createSparkCon
at $iwC$$iwC.<init>(<console>:15)
at $iwC.<init>(<console>:24)
at <init>(<console>:26)
at .<init>(<console>:30)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Na
at sun.reflect.NativeMethodAccessorImpl.invoke(Nat
at sun.reflect.DelegatingMethodAccessorImpl.invoke
at java.lang.reflect.Method.invoke(Method.java:498
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.
at org.apache.spark.repl.SparkIMain$Request.loadAn
at org.apache.spark.repl.SparkIMain.loadAndRunReq$
at org.apache.spark.repl.SparkIMain.interpret(Spar
at org.apache.spark.repl.SparkIMain.interpret(Spar
at org.apache.spark.repl.SparkILoop.reallyInterpre
at org.apache.spark.repl.SparkILoop.interpretStart
at org.apache.spark.repl.SparkILoop.command(SparkI
at org.apache.spark.repl.SparkILoopInit$$anonfun$i
at org.apache.spark.repl.SparkILoopInit$$anonfun$i
at org.apache.spark.repl.SparkIMain.beQuietDuring(
at org.apache.spark.repl.SparkILoopInit$class.init
at org.apache.spark.repl.SparkILoop.initializeSpar
at org.apache.spark.repl.SparkILoop$$anonfun$org$aSparkILoop.scala:974)
at org.apache.spark.repl.SparkILoopInit$class.runT
at org.apache.spark.repl.SparkILoop.runThunks(Spar
at org.apache.spark.repl.SparkILoopInit$class.post
at org.apache.spark.repl.SparkILoop.postInitializa
at org.apache.spark.repl.SparkILoop$$anonfun$org$a
at org.apache.spark.repl.SparkILoop$$anonfun$org$a
at org.apache.spark.repl.SparkILoop$$anonfun$org$a
at scala.tools.nsc.util.ScalaClassLoader$.savingCo
at org.apache.spark.repl.SparkILoop.org$apache$spa
at org.apache.spark.repl.SparkILoop.process(SparkI
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Na
at sun.reflect.NativeMethodAccessorImpl.invoke(Nat
at sun.reflect.DelegatingMethodAccessorImpl.invoke
at java.lang.reflect.Method.invoke(Method.java:498
at org.apache.spark.deploy.SparkSubmit$.org$apache
at org.apache.spark.deploy.SparkSubmit$.doRunMain$
at org.apache.spark.deploy.SparkSubmit$.submit(Spa
at org.apache.spark.deploy.SparkSubmit$.main(Spark
at org.apache.spark.deploy.SparkSubmit.main(SparkS

java.lang.NullPointerException
at org.apache.spark.sql.SQLContext$.createListener
at org.apache.spark.sql.hive.HiveContext.<init>(Hi
at sun.reflect.NativeConstructorAccessorImpl.newIn
at sun.reflect.NativeConstructorAccessorImpl.newIn
at sun.reflect.DelegatingConstructorAccessorImpl.n
at java.lang.reflect.Constructor.newInstance(Const
at org.apache.spark.repl.SparkILoop.createSQLConte
at $iwC$$iwC.<init>(<console>:15)
at $iwC.<init>(<console>:24)
at <init>(<console>:26)
at .<init>(<console>:30)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Na
at sun.reflect.NativeMethodAccessorImpl.invoke(Nat
at sun.reflect.DelegatingMethodAccessorImpl.invoke
at java.lang.reflect.Method.invoke(Method.java:498
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.
at org.apache.spark.repl.SparkIMain$Request.loadAn
at org.apache.spark.repl.SparkIMain.loadAndRunReq$
at org.apache.spark.repl.SparkIMain.interpret(Spar
at org.apache.spark.repl.SparkIMain.interpret(Spar
at org.apache.spark.repl.SparkILoop.reallyInterpre
at org.apache.spark.repl.SparkILoop.interpretStart
at org.apache.spark.repl.SparkILoop.command(SparkI
at org.apache.spark.repl.SparkILoopInit$$anonfun$i
at org.apache.spark.repl.SparkILoopInit$$anonfun$i
at org.apache.spark.repl.SparkIMain.beQuietDuring(
at org.apache.spark.repl.SparkILoopInit$class.init
at org.apache.spark.repl.SparkILoop.initializeSpar
at org.apache.spark.repl.SparkILoop$$anonfun$org$aSparkILoop.scala:974)
at org.apache.spark.repl.SparkILoopInit$class.runT
at org.apache.spark.repl.SparkILoop.runThunks(Spar
at org.apache.spark.repl.SparkILoopInit$class.post
at org.apache.spark.repl.SparkILoop.postInitializa
at org.apache.spark.repl.SparkILoop$$anonfun$org$a
at org.apache.spark.repl.SparkILoop$$anonfun$org$a
at org.apache.spark.repl.SparkILoop$$anonfun$org$a
at scala.tools.nsc.util.ScalaClassLoader$.savingCo
at org.apache.spark.repl.SparkILoop.org$apache$spa
at org.apache.spark.repl.SparkILoop.process(SparkI
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Na
at sun.reflect.NativeMethodAccessorImpl.invoke(Nat
at sun.reflect.DelegatingMethodAccessorImpl.invoke
at java.lang.reflect.Method.invoke(Method.java:498
at org.apache.spark.deploy.SparkSubmit$.org$apache
at org.apache.spark.deploy.SparkSubmit$.doRunMain$
at org.apache.spark.deploy.SparkSubmit$.submit(Spa
at org.apache.spark.deploy.SparkSubmit$.main(Spark
at org.apache.spark.deploy.SparkSubmit.main(SparkS

<console>:16: error: not found: value sqlContext
import sqlContext.implicits._
^
<console>:16: error: not found: value sqlContext
import sqlContext.sql
^
Snow_Face
关注
【spark-shell本地启动报java错误】本地windows安装spark在控制台输入spark-shell出错：StringIndexOutOf Bounds Exception
gdhbc的博客
05-28 838
前提：本地java、hadoop已经安装好，但本地cmd启动spark-shell报出下面错误，我的配置是hadoop2.7、spark-2.3.1。 提示:没有配置好hadoop、spark(两者版本有联系，有对应)请先在本地安装好hadoop、spark。 出错原因：本地java版本过高. 处理方法：降低java版本到java8。如果不想影响你之前的java环境配置，你可以在本地多安一个java，想用哪个就到环境变量里面修改路径即可。 java8百度云资源： 链接: https://pan.b
windows安装spark后spark-shell启动报错
木凡_Denius
05-17 6218
Hadoop lib下已添加对应版本的winutils cmd输入D:\peizhi\hadoop\bin\winutils.exe chmod 777 /tmp/hive 不能成功启动spark-shell <console>:16: error: not found: value sqlContext import sqlContext.implicits._
SPARK 启动spark-shell报错The specified datastore driver (“com.mysql.jdbc.Driver“) was not found
80后大叔爱学习
07-25 1429
错误提示：Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "DBCP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was...
Spark-遇到的坑-启动spark报错
热门推荐
a1032818891的博客
07-26 1万+
1.解决措施： Please instead use:  - ./spark-submit with --num-executors to specify the number of executors  - Or set SPARK_EXECUTOR_INSTANCES  - spark.executor.instances to configure the number of instanc...
spark-shell启动报错解决办法
hsg77的专栏
01-03 4914
spark-shell启动报错解决办法: scala版本不兼容问题 这是因为加入了项目依赖库到/usr/cwgis/app/spark/jars/lib/中 删除相关的scala开头的jar文件即可启动spark-shell [root@node111 ~]# runCmd.sh &amp;quot;rm /usr/cwgis/app/spark/jars/lib/scala*.jar&amp;quot; all ...
spark Shell启动的时候报错：ERROR spark.SparkContext: Error initializing SparkContext.ConnectException
技术探求
07-18 5453
spark Shell启动的时候报错：ERROR spark.SparkContext: Error initializing SparkContext.ConnectException: Call From hadoop202/192.168.1.202 to hadoop202:9000 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: spark连接HDFS拒绝连接，其实是犯了一
kylin启动成功，登陆不了网页！！!!!
xiaobaicxx的博客
05-17 5295
kylin-1.5.2.1搭建hadoop-2.6.0habse-1.2.0hive-1.1.0jdk-1.7.0_80zookeeper-3.4.5已确认所有组件都能正常运行，然后kylin启动成功登陆网页报错，求解，报错如下：五月 17, 2018 10:57:15 上午 org.apache.catalina.core.AprLifecycleListener lifecycleEvent信...
Spark遇到的问题
chenjieying12321的博客
10-27 3534
本人spark小白一枚，最开始使用spark时遇到过许多问题，现在把它们都记录下来，希望给后人一个参考。大神不要嘲笑哦~~~ 1.清除mac缓存：rm -rf /etc/udev/rules.d/70-persistent-net.rules 2.spark-submit \ --class main.scala.SparkWordCount \ --master sp
Hbase启动后Hmaster自动消失，求大神教导
xiaobaicxx的博客
05-11 2478
日志报错 2018年 05月 11日 星期五 10:47:03 CST Starting master on Hbase.Testcore file size          (blocks, -c) 0data seg size           (kbytes, -d) unlimitedscheduling priority             (-e) 0file size    ...
spark面试题
Summer2381的博客
09-19 1447
一、简答题 1.Spark master使用zookeeper进行HA的，有哪些元数据保存在Zookeeper？ 答：spark通过这个参数spark.deploy.zookeeper.dir指定master元数据在zookeeper中保存的位置，包括Worker，Driver和Application以及Executors。standby节点要从zk中，获得元数据信息，恢复集群运行状态，才能对外...
spark-shell启动spark报错
zhang__rong的博客
11-05 1801
spark-shell启动spark报错 前言 　　离线安装好CDH、Coudera Manager之后，通过Coudera Manager安装所有自带的应用，包括hdfs、hive、yarn、spark、hbase等应用，过程很是波折，此处就不抱怨了，直接进入主题。 描述 　　在安装有spark的节点上，通过spark-shell启动spark，满怀期待的启动spark，but，来了个晴...
spark-shell启动错误
weixin_30709061的博客
06-24 228
18/06/24 16:41:40 ERROR spark.SparkContext: Error initializing SparkContext.java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed after 16 retries (starting from 0...
启动spark-shell时报错java.lang.NumberFormatException: For input string: “0x100“
最新发布
哇咔咔哇咔的博客
02-26 711
启动spark-shell时报错java.lang.NumberFormatException: For input string: "0x100"
进入spark-shell报错
习惯一个人面对所有
04-10 1073
进入spark-shell总是报一大堆错才能进去 Unable to load native-hadoop library for your platform… using builtin-java classes where applicable To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLo...
Spark 启动Spark Shell环境出错
m0_65893781的博客
10-31 1804
Spark 启动Spark Shell环境出错 Exception in thread "main" java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
spark-shell启动报错--无法连接master
weixin_43079984的博客
02-24 1907
①spark standalone模式无法启动，连接master失败，但是集群用jps查看时，master明明已经启动。 ②spark-shell on yarn启动时，无法创建sc。 原因：通过hadoop104:8080查看master节点时，master节点的状态为standby，说明集群中目前没有master提供服务。 ...
spark-shell on yarn 出错解决【启动命令bin/spark-shell --master yarn-client出现错误，类ExecutorLauncher 找不到】
Thinking
12-01 1万+
文章来源：http://www.dataguru.cn/thread-331456-1-1.html 今天想要将spark-shell 在yarn-client的状态下 结果出错： [python] view plaincopy [hadoop@localhost spark-1.0.1-bin-hadoop2]$ bin/spark-shell
spark-1.6.3源码---启动spark shell报错
Ms_WangH
07-25 196
启动spark-shell报错: log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http:/...
ERROR Utils: Uncaught exception in thread stdout writer for python
weixin_34349320的博客
07-28 751
大数据量的情况下用pyspark对数据进行处理的时候 pycharm运行会出现ERROR Utils: Uncaught exception in thread stdout writer for python错误 需要修改JVM参数 打开pycharm的安装目录 C:\Program Files (x86)\JetBrains\PyCharm 2016.2.3\bin下文件pycharm....
warn no appenders
05-01
"warn no appenders"是一个常见的日志错误，通常在尝试记录日志数据时出现。它表示应用程序已调用日志API，但没有配置适当的“appender”（其作用是将日志记录输出到文件、控制台等目标中）。 这个错误通常出现在以下几种情况下： 1. 缺少日志配置文件或配置不正确，没有定义appender。 2. 日志库的配置被修改或更改文件名，但程序重启之前没有重置或重新加载。 3. 程序依赖的库或模块没有正确初始化或注入配置。 要解决这个错误，需要检查程序的日志配置文件，并确保定义了至少一个appender，以将日志信息输出到目标。确保配置文件的路径和名称以及日志库的版本与程序的依赖列表中的版本匹配。如果仍然无法解决，请尝试重新加载配置或重新启动程序。 另外，如果应用程序旨在在不同的环境中运行，例如开发、测试和生产环境，请确保每个环境都有自己的日志配置文件，并按照需要进行调整。
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值