HDFS block丢失过多进入安全模式（safe mode）的解决方法

HDFS block丢失过多进入安全模式（Safe mode）的解决方法

背景及现象描述(Background and Symptom)

 

 因磁盘空间不足，内存不足，系统掉电等其他原因导致dataNode datablock丢失，出现如下类似日志：

 

The number of live datanodes 3 has reached the minimum number 0.

Safe mode will be turned off automatically once the thresholds have been reached.

Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Log not rolled.

 Name node is in safe mode.

The reported blocks 632758 needs additional 5114 blocks to reach the threshold 0.9990

of total blocks 638510.

The number of live datanodes 3 has reached the minimum number 0.

Safe mode will be turned off automatically once the thresholds have been reached.

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode

(FSNamesystem.java:1209)

    ... 12 more

 

 

原因分析(Cause Analysis)*

由于系统断电，内存不足等原因导致dataNode丢失超过设置的丢失百分比，系统自动进入安全模式

解决办法(Solution)*

安装HDFS客户端，并执行如下命令：

                                步骤 1     执行命令退出安全模式：hadoop dfsadmin -safemode leave

                                步骤 2     执行健康检查，删除损坏掉的block。  hdfs fsck  /  -delete

 

注意: 这种方式会出现数据丢失，损坏的block会被删掉