进入spark-shell报错

最新推荐文章于 2024-06-19 14:17:35 发布
最新推荐文章于 2024-06-19 14:17:35 发布
阅读量1k 收藏
点赞数
分类专栏： spark 文章标签： spark
by 习惯于一个人面对所有
本文链接：https://blog.csdn.net/qiaoenshi/article/details/105435121
版权

进入spark-shell总是报一大堆错才能进去
Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
20/04/10 15:35:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/04/10 15:36:15 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master node200:7077
org.apache.spark.SparkException: Exception thrown in awaitResult: 
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
	at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
	at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:100)
	at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:108)
	at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1$$anon$1.run(StandaloneAppClient.scala:106)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to connect to node200/192.168.128.200:7077
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:232)
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:182)
	at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197)
	at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
	at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
	... 4 more
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: node200/192.168.128.200:7077
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:631)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
	... 1 more
20/04/10 15:36:33 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master node200:7077
org.apache.spark.SparkException: Exception thrown in awaitResult: 
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
	at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
	at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:100)
	at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:108)
	at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1$$anon$1.run(StandaloneAppClient.scala:106)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to connect to node200/192.168.128.200:7077
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:232)
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:182)
	at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197)
	at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
	at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
	... 4 more
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: node200/192.168.128.200:7077
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:631)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
	... 1 more
20/04/10 15:36:53 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master node200:7077
org.apache.spark.SparkException: Exception thrown in awaitResult: 
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
	at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
	at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:100)
	at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:108)
	at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1$$anon$1.run(StandaloneAppClient.scala:106)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to connect to node200/192.168.128.200:7077
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:232)
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:182)
	at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197)
	at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
	at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
	... 4 more
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: node200/192.168.128.200:7077
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:631)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
	... 1 more
20/04/10 15:37:13 ERROR StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
20/04/10 15:37:13 WARN StandaloneSchedulerBackend: Application ID is not initialized yet.
20/04/10 15:37:15 WARN StandaloneAppClient$ClientEndpoint: Drop UnregisterApplication(null) because has not yet connected to master
20/04/10 15:37:17 WARN MetricsSystem: Stopping a MetricsSystem that is not running
20/04/10 15:41:10 ERROR SparkContext: Error initializing SparkContext.
java.lang.NullPointerException
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:567)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910)
	at org.apache.spark.repl.Main$.createSparkSession(Main.scala:101)
	at $line3.$read$$iw$$iw.<init>(<console>:15)
	at $line3.$read$$iw.<init>(<console>:42)
	at $line3.$read.<init>(<console>:44)
	at $line3.$read$.<init>(<console>:48)
	at $line3.$read$.<clinit>(<console>)
	at $line3.$eval$.$print$lzycompute(<console>:7)
	at $line3.$eval$.$print(<console>:6)
	at $line3.$eval.$print(<console>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
	at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
	at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
	at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
	at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
	at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
	at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
	at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
	at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
	at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
	at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:38)
	at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
	at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
	at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
	at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:37)
	at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:98)
	at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)
	at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
	at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
	at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
	at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
	at org.apache.spark.repl.Main$.doMain(Main.scala:74)
	at org.apache.spark.repl.Main$.main(Main.scala:54)
	at org.apache.spark.repl.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
java.lang.NullPointerException
  at org.apache.spark.SparkContext.<init>(SparkContext.scala:567)
  at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910)
  at org.apache.spark.repl.Main$.createSparkSession(Main.scala:101)
  ... 47 elided
<console>:14: error: not found: value spark
       import spark.implicits._
              ^
<console>:14: error: not found: value spark
       import spark.sql
              ^
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.2
      /_/
         
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_51)
Type in expressions to have them evaluated.
Type :help for more information.

scala> 20/04/10 15:45:55 WARN DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /eventLog/spark-application-1586504172398.inprogress could only be replicated to 0 nodes instead of minReplication (=1).  There are 3 datanode(s) running and 3 node(s) are excluded in this operation.
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1559)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3245)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:663)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)

在这里插入图片描述

最后，我找到了原因，是由于我的疏忽，忘记启动spark了才导致报错。

解决办法就是退出shell，启动spark，然后再进入spark-shell

如果还不行，就关闭虚拟机，重新启动hadoop，再启动spark

这是我的亲身经验。

习惯了一个人面对所有
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
spark history web访问报错java.lang.NoSuchMethodError: javax.ws.rs.core.Application.getProperties()
qq_23112369的博客
09-18 906
版本：spark2.4.8 模式：spark on yarn 具体错误：访问history-web的时候，一片空白，查不出配置在hdfs的log，仔细看了conf下的配置没问题，然后浏览器F12发现接口方法报错，信息为java.lang.NoSuchMethodError: javax.ws.rs.core.Application.getProperties()Ljava/util/Map; 搜索了这个方法来源于jersey-core。由于是spark on yarn，yarn的jersy-cor..
启动spark-shell报错
Please_Forgive的博客
07-09 1101
错误信息： [root@HadoopStudy bin]# ./spark-shell WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.Kerbero...
Spark 启动Spark Shell环境出错
m0_65893781的博客
10-31 1775
Spark 启动Spark Shell环境出错 Exception in thread "main" java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
Spark三种运行模式（SparkShell使用）
m0_71010347的博客
06-19 1750
唯一不同的是，Spark Shell本身为集群的client提交方式运行，不支持cluster提交方式，即使用Spark Shell时，Driver运行于本地客户端，而不能运行于集群中。​ Driver为主控进程，负责执行应用程序的main()方法，创建SparkContext对象（负责与Spark集群进行交互），提交Spark作业，并将作业转化为Task（一个作业由多个Task任务组成），然后在各个Executor进程间对Task进行调度和监控。通常用SparkContext代表Driver。
spark-shell启动报错解决办法
hsg77的专栏
01-03 4894
spark-shell启动报错解决办法: scala版本不兼容问题 这是因为加入了项目依赖库到/usr/cwgis/app/spark/jars/lib/中 删除相关的scala开头的jar文件即可启动spark-shell [root@node111 ~]# runCmd.sh &amp;quot;rm /usr/cwgis/app/spark/jars/lib/scala*.jar&amp;quot; all ...
在本地cmd中执行Spark-shell报错：Exception in thread “main“ java.lang.ExceptionInInitializerError
qq_64078377的博客
10-10 1091
本地执行spark-shell报错
java.lang.NoSuchMethodError: javax.ws.rs.core.Application.getProperties()Ljava/util/Map
skayliu的专栏
03-29 5723
Spring Boot整合Camunda，使用feigin调用Camunda REST API 时失败 首先，pom.xml中有下面三个引用 <dependency> <groupId>org.camunda.bpm.springboot</groupId> <artifactId&gt...
启动spark shell 报错
justinzwd的博客
01-19 934
启动 bin/spark-shell的时候出现如下报错信息 Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 21/01/19 10:45:5
Win10下spark-shell报错处理-附件资源
03-02
Win10下spark-shell报错处理-附件资源
Spark启动过程中遇到的错误 ./spark-shell –master spark://node001:7077
01-20
- 首先，你需要进入Spark的配置目录，通常是`$SPARK_HOME/conf`。 - 编辑`spark-env.sh`文件，这是Spark环境变量的配置文件。如果你的系统是Linux，可以使用`vi`或`nano`编辑器。 - 在文件中添加或修改以下两行，...
Spark在windows运行报错-ERROR Shell Failed to locate the winutils binary in the hadoop binary path java.io-附件资源
03-02
Spark在windows运行报错-ERROR Shell Failed to locate the winutils binary in the hadoop binary path java.io-附件资源
Spark/Hadoop开发缺失插件-winutils.exe
04-25
本地开发Spark/Hadoop报错“ERROR Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.” ...
Spark写本地异常修复包
最新发布
07-10
运行时写本地window磁盘报错：Exception in thread "main" ExitCodeException exitCode=-1073741701: at org.apache.hadoop.util.Shell.runCommand(Shell.java:1009) msvcr120.dll文件缺失 方法一： 现在大部分...
基于spark mllib库的垃圾短信识别1
08-03
- 打开命令行工具，输入`spark-shell`命令，如果成功启动Spark Shell，则表明配置正确。 #### 二、程序运行过程中遇到的问题 在实际操作过程中，可能会遇到一些编码和类型转换方面的问题。 ##### 1. 在【txt2csv....
Hadoop 启动WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using
01-09
搭建hadoop 环境时遇到启动异常告警问题 “WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable” 上来不多说，百度收集些相关...
Jupyter notebook运行Spark+Scala教程
12-20
4. 如果遇到版本不兼容的问题，如报错`NoSuchMethodError: scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet;`，那可能是因为Toree默认使用了不匹配的Scala版本。这时你需要指定...
执行spark-shell报错：
weixin_30359021的博客
11-24 289
执行spark-shell报错： [ERROR] Terminal initialization failed; falling back to unsupportedjava.lang.NoClassDefFoundError: Could not initialize class scala.tools.fusesource_embedded.jansi.internal.Kernel32 ...
spark Shell启动的时候报错：ERROR spark.SparkContext: Error initializing SparkContext.ConnectException
技术探求
07-18 5403
spark Shell启动的时候报错：ERROR spark.SparkContext: Error initializing SparkContext.ConnectException: Call From hadoop202/192.168.1.202 to hadoop202:9000 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: spark连接HDFS拒绝连接，其实是犯了一
spark启动spark-shell报错
04-29
当启动Spark Shell时，可能会遇到各种错误。以下是一些常见的错误和解决方法： 1. 错误：`java.lang.OutOfMemoryError: Java heap space` 解决方法：增加堆内存大小。可以通过设置`spark.driver.memory`和`spark.executor.memory`来增加内存大小。例如，可以使用以下命令启动Spark Shell： ``` spark-shell --driver-memory 4g --executor-memory 4g ``` 2. 错误：`java.lang.ClassNotFoundException: org.apache.spark.sql.hive.HiveContext` 解决方法：这个错误通常是由于缺少Hive相关的依赖库引起的。请确保在启动Spark Shell时包含了正确的Hive依赖。例如，可以使用以下命令启动Spark Shell： ``` spark-shell --packages org.apache.spark:spark-hive_2.12:3.1.2 ``` 3. 错误：`java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem` 解决方法：这个错误通常是由于Spark配置文件中的一些错误引起的。请检查`spark-defaults.conf`文件中的配置项是否正确，并确保Spark的配置文件没有任何语法错误。 4. 错误：`java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream` 解决方法：这个错误通常是由于缺少Hadoop相关的依赖库引起的。请确保在启动Spark Shell时包含了正确的Hadoop依赖。例如，可以使用以下命令启动Spark Shell： ``` spark-shell --packages org.apache.hadoop:hadoop-client:3.2.0 ```
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值