启动Spark-Shell出现hdfs报错

最新推荐文章于 2023-03-13 18:28:41 发布
最新推荐文章于 2023-03-13 18:28:41 发布
阅读量666 收藏
点赞数 1
分类专栏： Bug排查 文章标签： spark hdfs hadoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_48238787/article/details/121870722
版权
在尝试使用Spark Shell学习Spark SQL时遇到错误，错误信息显示为`NoSuchMethodException`，根源在于HDFS的RPC响应超出了最大长度。问题的解决是由于在`spark-defaults.xml`配置文件中，将HDFS端口误配置为访问端口而非内部通信的8020端口。修正配置后，成功启动Spark Shell。
摘要由CSDN通过智能技术生成

今天在学习Spark-Sql的时候，想要在Spark-Shell来学习语法，启动的时候却报了如下错误：

java.lang.NoSuchMethodException: org.apache.hadoop.ipc.RpcException.<init>(java.lang.String)
        at java.lang.Class.getConstructor0(Class.java:3082)
        at java.lang.Class.getConstructor(Class.java:1825)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:830)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806)
        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1515)
        at org.apache.hadoop.ipc.Client.call(Client.java:1457)
        at org.apache.hadoop.ipc.Client.call(Client.java:1367)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
        at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:903)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
        at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1665)
        at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1582)
        at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1594)
        at org.apache.spark.deploy.history.EventLogFileWriter.requireLogBaseDirAsDirectory(EventLogFileWriters.scala:77)
        at org.apache.spark.deploy.history.SingleEventLogFileWriter.start(EventLogFileWriters.scala:221)
        at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:81)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:572)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2555)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$1(SparkSession.scala:930)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
        at org.apache.spark.repl.Main$.createSparkSession(Main.scala:106)
        at $line3.$read$$iw$$iw.<init>(<console>:15)
        at $line3.$read$$iw.<init>(<console>:42)
        at $line3.$read.<init>(<console>:44)
        at $line3.$read$.<init>(<console>:48)
        at $line3.$read$.<clinit>(<console>)
        at $line3.$eval$.$print$lzycompute(<console>:7)
        at $line3.$eval$.$print(<console>:6)
        at $line3.$eval.$print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:745)
        at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1021)
        at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:574)
        at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:41)
        at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:37)
        at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
        at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:600)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:570)
        at scala.tools.nsc.interpreter.IMain.$anonfun$quietRun$1(IMain.scala:224)
        at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
        at scala.tools.nsc.interpreter.IMain.quietRun(IMain.scala:224)
        at org.apache.spark.repl.SparkILoop.$anonfun$initializeSpark$2(SparkILoop.scala:83)
        at scala.collection.immutable.List.foreach(List.scala:392)
        at org.apache.spark.repl.SparkILoop.$anonfun$initializeSpark$1(SparkILoop.scala:83)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.tools.nsc.interpreter.ILoop.savingReplayStack(ILoop.scala:99)
        at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:83)
        at org.apache.spark.repl.SparkILoop.$anonfun$process$4(SparkILoop.scala:165)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.tools.nsc.interpreter.ILoop.$anonfun$mumly$1(ILoop.scala:168)
        at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
        at scala.tools.nsc.interpreter.ILoop.mumly(ILoop.scala:165)
        at org.apache.spark.repl.SparkILoop.loopPostInit$1(SparkILoop.scala:153)
        at org.apache.spark.repl.SparkILoop.$anonfun$process$10(SparkILoop.scala:221)
        at org.apache.spark.repl.SparkILoop.withSuppressedSettings$1(SparkILoop.scala:189)
        at org.apache.spark.repl.SparkILoop.startup$1(SparkILoop.scala:201)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:236)
        at org.apache.spark.repl.Main$.doMain(Main.scala:78)
        at org.apache.spark.repl.Main$.main(Main.scala:58)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
21/12/11 11:06:30 ERROR SparkContext: Error initializing SparkContext.
java.io.IOException: Failed on local exception: org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; Host Details : local host is: "hadoop104/192.168.10.104"; destination host is: "hadoop104":9870;

最后一句非常重要，说明我这里的错误是连接不上HDFS，因为我之前在 spark-defaults.xml中配置hdfs的端口，我一不小心将端口号写成了访问端口号，而不是内部接受信号的8020端口，当我修改后，就能顺利启动了。

孤独的偷学者
关注
专栏目录
spark-shell启动spark报错
zhang__rong的博客
11-05 1804
spark-shell启动spark报错 前言 　　离线安装好CDH、Coudera Manager之后，通过Coudera Manager安装所有自带的应用，包括hdfs、hive、yarn、spark、hbase等应用，过程很是波折，此处就不抱怨了，直接进入主题。 描述 　　在安装有spark的节点上，通过spark-shell启动spark，满怀期待的启动spark，but，来了个晴...
spark on yarn java8_spark on yarn 错误
weixin_30954817的博客
02-26 262
hduser@xxx:/data1/hadoop/spark/bin$ ./spark-shell --master yarn --deploy-mode client2020-04-13 10:04:25 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using buil...
问题记录：spark读取hdfs文件出错
weixin_30411239的博客
01-03 263
错误信息： scala> val file = sc.textFile("hdfs://kit-b5:9000/input/README.txt") 13/10/29 16:59:45 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.secur...
Spark-遇到的坑-启动spark报错
热门推荐
a1032818891的博客
07-26 1万+
1.解决措施： Please instead use:  - ./spark-submit with --num-executors to specify the number of executors  - Or set SPARK_EXECUTOR_INSTANCES  - spark.executor.instances to configure the number of instanc...
spark Shell启动的时候报错：ERROR spark.SparkContext: Error initializing SparkContext.ConnectException
技术探求
07-18 5463
spark Shell启动的时候报错：ERROR spark.SparkContext: Error initializing SparkContext.ConnectException: Call From hadoop202/192.168.1.202 to hadoop202:9000 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: spark连接HDFS拒绝连接，其实是犯了一
spark-shell
lkx626859的博客
03-13 128
spark-shell
安装CDH后启动spark-shell报错解决方案
jaysen1005的博客
02-28 2302
描述 安装好CDH、Coudera Manager之后，通过Coudera Manager安装所有自带的应用，包括hdfs、hive、yarn、spark、hbase等。利用远程客户端在安装有spark的节点上，通过spark-shell启动spark，满怀期待的启动spark，but，来了个晴天霹雳，报错了，报错了！错误信息如下： 19/02/28 13:45:06 ERROR spark....
启动spark-shell出现问题，解决办法
jiaofengaoxue的博客
12-29 7621
在启动spark-shell出现以下问题： 15/12/30 10:05:29 ERROR SparkContext: Errorinitializing SparkContext. java.lang.IllegalArgumentException: WrongFS:file://219.226.86.159:~/home/iespark/hadoop_program_files/spar
spark-shell 启动报错org.apache.hadoop.security.AccessControlException: Permission denied: user=root, acc
蘑菇丁的专栏
05-16 2154
[root@cm3 ~]# spark-shell Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 19/05/16 17:08:23 ERROR spark.SparkContext:...
执行./pyspark,./spark-shell报错拒绝链接，Error initializing SparkContext. java.net.ConnectException:
m0_50641264的博客
01-12 3557
1.错误描述,执行./pyspark,./spark-shell报错拒绝链接 22/01/12 03:49:27 ERROR SparkContext: Error initializing SparkContext. java.net.ConnectException: Call From 6274master/192.168.47.10 to 6274master:8020 failed on connection exception: java.net.ConnectException: Conne
Spark启动过程中遇到的错误 ./spark-shell –master spark://node001:7077
01-20
20/02/20 19:52:17 ERROR spark.SparkContext: Error initializing SparkContext. java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.at.org.apache.spark.memory.Uni
Spark 连接hive，启动spark-shell报错：Error creating transactional connection factory
qq_32783151的博客
05-09 691
在使用Spark连接HIve，启动spark-shell的时候，报错： Error creating transactional connection factory 输入命令为：./spark-shell --master spark://nodes03:7077 此时已经将$HIVE_HOME/conf/hive-site.xml，$HADOOP_HOME/etc/hadoop/core-site.xml hdfs-site.xml拷贝到$SPARK_HOME/conf下，但是依然报错。 解决办法如下
解决spark-shell启动，hdfs权限问题
小谷的博客
10-26 1226
解决spark-shell启动，hdfs权限问题环境问题尝试创建创建文件夹 环境 cdh6.3.2 问题 在启动spark-shell的时候，出现 /user/spark/driverLogs does not exist. Please create this dir in order to persist driver logs错误 虽然后面正常启动了，但是心中还是觉得创建一下这个日志路径比较好 尝试创建 hadoop dfs -mkdir -p /user/spark/driverLogs 可
Spark 启动Spark Shell环境出错
m0_65893781的博客
10-31 1812
Spark 启动Spark Shell环境出错 Exception in thread "main" java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
SPARK 启动spark-shell报错The specified datastore driver (“com.mysql.jdbc.Driver“) was not found
80后大叔爱学习
07-25 1430
错误提示：Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "DBCP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was...
Spark2.4读取HDFS失败
weixin_42529806的博客
08-13 833
报错：Exception in thread “main” java.lang.IllegalAccessError: class org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface org.apache.hadoop.hdfs.web.TokenAspect$TokenManagementDel...
Spark/HDFS上传文件错误：Permission denied
虾哔哔的博客
09-13 2235
问题描述 今天用spark将RDD结果输出到HDFS时，因为CDH集群重新配置的原因，权限没有设置好。导致写文件出bug。 错误情况 org.apache.hadoop.security.AccessControlException: Permission denied: user=xiet, access=WRITE, inode="/":hdfs:supergroup:drwxr-...
spark-shell启动报错解决办法
hsg77的专栏
01-03 4916
spark-shell启动报错解决办法: scala版本不兼容问题 这是因为加入了项目依赖库到/usr/cwgis/app/spark/jars/lib/中 删除相关的scala开头的jar文件即可启动spark-shell [root@node111 ~]# runCmd.sh &amp;quot;rm /usr/cwgis/app/spark/jars/lib/scala*.jar&amp;quot; all ...
Spark 远程读写 Hive (HDFS) 失败
L, there!
03-02 2790
[WARN] - I/O error constructing remote block reader. java.net.ConnectException: Connection timed out: no further information ... [WARN] - Connection failure: Failed to connect to /10.0.0.24...
在yarn中启动spark-shell客户端提交任务？
最新发布
05-30
要在YARN中启动Spark-shell客户端并提交任务，可以按照以下步骤操作： 1. 首先确保您已经正确安装和配置了Spark，并且Spark环境变量已经设置好。 2. 然后，在终端中输入以下命令来启动Spark-shell： ``` spark-...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

孤独的偷学者

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值