1、spark-shell --master local[2]启动出现如下报错:
[hadoop@hadoop002 spark-2.4.2-bin-2.6.0-cdh5.7.0]$ spark-shell --master local[2]
19/07/23 00:52:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
19/07/23 00:52:32 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create file/g6_directory/local-1563814350999.inprogress. Name node is in safe mode.
The reported blocks 25 needs additional 10 blocks to reach the threshold 0.9990 of total blocks 35.
The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
19/07/23 00:52:33 ERROR repl.Main: Failed to initialize Spark session.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create file/g6_directory/local-1563814350999.inprogress. Name node is in safe mode.
The reported blocks 25 needs additional 10 blocks to reach the threshold 0.9990 of total blocks 35.
The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
定位ERROR:Name node is in safe,那么NameNode为什么在安全模式下?
NameNode发现集群中DataNode丢失达到一定比例(0.01%)时会进入安全模式,此时只允许查看数据不允许对数据进行任何操作。
使用命令:hdfs dfsadmin -safemode leave 离开安全模式
hdfs dfsadmin -safemode enter 进入安全模式
退出安全模式即可重新进入。
2、PK哥上课过程中重启spark-shell报错:
https://blog.csdn.net/zhikanjiani/article/details/97833470
问题:Failed to start database ‘metastore_db’ with class loader org.apache.spark.sql.hive.client
解决:rm -rf metastore_db/
3、更新:2019-08-16
暂时使用hdfs dfsadmin -safemode leave命令离开了安全模式。
每次启动spark-shell都会报错:2019-08-24