Spark执行spark-shell 报错 Cannot create directory /tmp/hive/root/xxx. Name node is in safe mode

最新推荐文章于 2023-04-26 19:57:10 发布
最新推荐文章于 2023-04-26 19:57:10 发布
阅读量861 收藏 1
点赞数 1
分类专栏： Linux
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_41986096/article/details/109525886
版权

1. 问题：

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
20/11/05 21:03:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':
  at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1053)
  at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:130)
  at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:130)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:129)
  at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:126)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:938)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:938)
  at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
  at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
  at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
  at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
  at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
  at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:938)
  at org.apache.spark.repl.Main$.createSparkSession(Main.scala:97)
  ... 47 elided
Caused by: org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create directory /tmp/hive/root/1af1f42a-4233-4abd-b335-62a580f200e5. Name node is in safe mode.
The reported blocks 122 needs additional 5 blocks to reach the threshold 0.9990 of total blocks 127.
The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1366)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4258)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4233)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:853)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:600)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:975)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2036)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2034)
;
  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
  at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:193)
  at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:105)
  at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:93)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:39)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog$lzycompute(HiveSessionStateBuilder.scala:54)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:52)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:35)
  at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:289)
  at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1050)
  ... 61 more
Caused by: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create directory /tmp/hive/root/1af1f42a-4233-4abd-b335-62a580f200e5. Name node is in safe mode.
The reported blocks 122 needs additional 5 blocks to reach the threshold 0.9990 of total blocks 127.
The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1366)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4258)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4233)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:853)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:600)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:975)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2036)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2034)

  at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
  at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:191)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264)
  at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:362)
  at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:266)
  at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66)
  at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65)
  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:194)
  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:194)
  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:194)
  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
  ... 70 more
Caused by: org.apache.hadoop.ipc.RemoteException: Cannot create directory /tmp/hive/root/1af1f42a-4233-4abd-b335-62a580f200e5. Name node is in safe mode.
The reported blocks 122 needs additional 5 blocks to reach the threshold 0.9990 of total blocks 127.
The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1366)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4258)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4233)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:853)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:600)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:975)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2036)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2034)

  at org.apache.hadoop.ipc.Client.call(Client.java:1470)
  at org.apache.hadoop.ipc.Client.call(Client.java:1401)
  at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
  at com.sun.proxy.$Proxy21.mkdirs(Unknown Source)
  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:539)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
  at com.sun.proxy.$Proxy22.mkdirs(Unknown Source)
  at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2742)
  at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2713)
  at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:870)
  at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:866)
  at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
  at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:866)
  at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:859)
  at org.apache.hadoop.hive.ql.session.SessionState.createPath(SessionState.java:639)
  at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:574)
  at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
  ... 84 more
<console>:14: error: not found: value spark
       import spark.implicits._
              ^
<console>:14: error: not found: value spark
       import spark.sql
              ^
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.0
      /_/
         
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_202)
Type in expressions to have them evaluated.
Type :help for more information.

 2. 执行：

NameNode启动时会进入该模式进行检测，检查数据块的完整性，处于该模式下的集群无法对HDFS进行操作，可以手动离开安全模式

hdfs dfsadmin -safemode leave

而进入安全模式是

# hadoop dfsadmin -saftmode enter

获取当前安全模式的状态

hdfs dfsadmin -safemode get

墨迹嘿嘿
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
用spark-shell命令时候报错了——aboutyun
火麒马
01-02 652
[aboutyun@master ~]$ spark-shell Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). 20/01/02 10:42:19 WARN util.NativeCodeLoader: Unable to load native-hadoop l...
cdh5.13启动spark-shell报错
weixin_37796929的博客
05-19 1556
[root@master01 ~]# spark-shell Setting default log level to “WARN”. To adjust logging level use sc.setLogLevel(newLevel). Welcome to ____ __ / / ___ / / \ / _ / _ `/ __/ '/ // .__/_,// ...
Spark-遇到的坑-启动spark报错
热门推荐
a1032818891的博客
07-26 1万+
1.解决措施： Please instead use:  - ./spark-submit with --num-executors to specify the number of executors  - Or set SPARK_EXECUTOR_INSTANCES  - spark.executor.instances to configure the number of instanc...
spark启动时Cannot assign requested address: Service 'sparkDriver'问题分析
weixin_43267910的博客
12-24 5184
问题经过 当使用bin/spark-shell命令或使用spark-submit提交任务时，会遇到如下报错。 Setting default log level to “WARN”. To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 19/12/24 09:58...
spark环境部署(local、standalone、yarn)
clear-ninghongkang的博客
04-20 1000
spark环境的部署
基于Shell脚本，通过简单配置后，可以自动安装Hadoop、Hive、Spark等大数据组件.zip
最新发布
04-19
2.下载完成之后，安装包默认保存在下载文件夹中，解压安装包apache-hive-1.2.1-bin.tar.gz至路径 /usr/local，命令如下： 3.然后切换至目录 /usr/local，将文件夹名改为hive（为了后续的方便），并且修改文件权限给...
编译的spark-hive_2.11-2.3.0和 spark-hive-thriftserver_2.11-2.3.0.jar
06-23
spark-hive_2.11-2.3.0...spark-hive-thriftserver_2.11-2.3.0.jar log4j-2.15.0.jar slf4j-api-1.7.7.jar slf4j-log4j12-1.7.25.jar curator-client-2.4.0.jar curator-framework-2.4.0.jar curator-recipes-2.4.0.jar
spark-3.2.4-bin-hadoop3.2-scala2.13 安装包
08-28
5. **运行Spark**: Spark可以通过命令行工具启动，例如`spark-shell`（交互式Spark会话）或`pyspark`（Python版本的交互式会话）。对于应用程序开发，可以使用Scala、Java、Python或R编写代码，然后通过`spark-...
spark--bin-hadoop3-without-hive.tgz
09-17
安装完成后，你可以通过`./bin/spark-shell`启动Spark的交互式Shell，或者使用`./bin/pyspark`启动Python版本的Shell，开始进行数据处理。 总的来说，"spark--bin-hadoop3-without-hive.tgz"提供了一个在CentOS 8和...
hive-spark-client-3.1.2.jar
09-17
hive-on-spark客户端
spark-shell数据处理
风
04-26 513
【代码】spark（实验课）
Spark：报错- Cannot create directory /tmp/hive/hadoop/. Name node is in safe mode.
BigData_Mining的博客
06-18 2190
java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':  at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSessi...
Spark：01基础环境
momotouguang的博客
12-25 787
Spark：01基础环境一、Spark 框架概述 官网 http://spark.apache.org/ https://databricks.com/spark/about 一、Spark 框架概述 官方对Spark的一个介绍 At a high level, every Spark application consists of a driver program #每个Spark程序都会包含一个驱动程序：Driver that runs the user’s main functio
mkdir: Cannot create directory /usr/spark. Name node is in safe mode.
qq_42321843的博客
03-07 1276
使用如下命令在HDFS文件系统中创建本实验的工作目录： /usr/local/hadoop/bin/hadoop fs -mkdir -p /user/spark 时遇到问题无法创建目录 mkdir: Cannot create directory /usr/spark. Name node is in safe mode. 这是因为该节点处于安全模式 可以采用强制退出的安全模式命令 找到ha...
Hadoop "Cannot create directory .Name node is in safe mode."解决方案
chuowen140381的博客
07-21 1579
转载自：http://www.waitig.com/hadoop-name-node-is-in-safe-mode.html 在使用Hadoop建立文件的时候，出现“Cannot create directory /user/hadoop/input. Name node is i...
在集群上运行python编写的spark应用程序（过程记录）
superce的博客
07-05 1万+
启动hadooproot@master:/usr/local/hadoop-2.7.5/sbin#./start-all.sh This script is Deprecated. Instead use start-dfs.shand start-yarn.shStarting namenodes on [master]master: starting namenode, logging to/...
Error :spark-shell模式报错：java.sql.SQLException: A read-only user or a user in a read-only database
小蚯蚓的博客
11-03 1533
1.问题描述： 启动spark-shell local的模式 bin/spark-shell --master local[2] 报错： [hadoop@hadoop spark-2.1.0-bin-hadoop2.6]$ bin/spark-shell --master local[2] Setting default log level to "WARN". To adjust l...
spark启动报错
m0_45502588的博客
01-02 694
今天在启动spark集群时遇到了一处错误： [root@bigdata01 spark]# bin/spark-shell --master spark://bigdata01:7070 Setting default log level to “WARN”. To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use...
Cannot create directory /tmp/hive/root/f57d4abf-40a5-48b1-9c78-d480c3447073. Name node is in safe mode.
07-27
引用\[1\]和\[2\]中提到的错误信息是由于Hadoop的NameNode处于安全模式导致的。安全模式是Hadoop的一种保护机制，它会限制对HDFS的写操作，以确保数据的完整性和一致性。当NameNode处于安全模式时，无法创建新的目录或文件。 解决这个问题的方法是强制关闭安全模式。根据引用\[3\]中的代码块，可以使用以下命令来离开安全模式： ``` hadoop dfsadmin -safemode leave ``` 执行这个命令后，NameNode将退出安全模式，并且你应该能够成功创建目录/tmp/hive/root/f57d4abf-40a5-48b1-9c78-d480c3447073。 #### 引用[.reference_title] - *1* *3* [hive客户端无法进入（Cannot create directory /tmp/hive/root/a0ec1f00-cd32-41ef-814d-b6aa3d669d77）](https://blog.csdn.net/bg_xyjn/article/details/112132518)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v91^insert_down1,239^v3^insert_chatgpt"}} ] [.reference_item] - *2* [问题随记 —— Cannot create directory /tmp/hive. Name node is in safe mode.](https://blog.csdn.net/qq_21484461/article/details/126678668)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v91^insert_down1,239^v3^insert_chatgpt"}} ] [.reference_item] [ .reference_list ]

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

墨迹嘿嘿

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值