检查HDFS块状态

hadoop集群运行过程中，节点的块状态或者上下线节点时集群都会受影响。

如何查看当前的hdfs的块的状态

hadoop1.x时候的命令，hadoop2.x也可使用：

hadoop fsck /

在hadoop2.0之后，可以使用新命令：

hdfs fsck /

[root@node01 ~]# su hdfs -c "hadoop fsck /"
WARNING: Use of this script to execute fsck is deprecated.
WARNING: Attempting to execute replacement "hdfs fsck" instead.

Connecting to namenode via http://node01:9870/fsck?ugi=hdfs&path=%2F
FSCK started by hdfs (auth:SIMPLE) from /192.168.1.81 for path / at Wed Apr 29 09:19:27 CST 2020

Status: HEALTHY
 Number of data-nodes:	3
 Number of racks:		1
 Total dirs:			1884
 Total symlinks:		0

Replicated Blocks:
 Total size:	387368629 B
 Total files:	85
 Total blocks (validated):	85 (avg. block size 4557277 B)
 Minimally replicated blocks:	85 (100.0 %)
 Over-replicated blocks:	0 (0.0 %)
 Under-replicated blocks:	0 (0.0 %)
 Mis-replicated blocks:		0 (0.0 %)
 Default replication factor:	3
 Average block replication:	3.0
 Missing blocks:		0
 Corrupt blocks:		0
 Missing replicas:		0 (0.0 %)
 Blocks queued for replication:	0

Erasure Coded Block Groups:
 Total size:	0 B
 Total files:	0
 Total block groups (validated):	0
 Minimally erasure-coded block groups:	0
 Over-erasure-coded block groups:	0
 Under-erasure-coded block groups:	0
 Unsatisfactory placement block groups:	0
 Average block group size:	0.0
 Missing block groups:		0
 Corrupt block groups:		0
 Missing internal blocks:	0
 Blocks queued for replication:	0
FSCK ended at Wed Apr 29 09:19:28 CST 2020 in 542 milliseconds


The filesystem under path '/' is HEALTHY

参数说明：

• Total size ： hdfs集群存储大小，不包括复本大小。如：75423236058649 B （字节）。（字节->KB->m->G->TB，75423236058649/1024/1024/1024/1024=68.59703358591014TB）

• Total blocks (validated) : 总共的块数量，不包括复本。（5363690 (avg. block size 14061818 B) (Total open file blocks (not validated): 148)，计算： 14061818 *5363690=75423232588420 集群的容量大小，不包括复本的）

• Number of data-nodes ： datanode的节点数量

• Number of racks ： 机架数量

• Default replication factor ： 默认的复制因子

• Average block replication ： 当前块的平均复制数，如果小 default replication factor，则有块丢失

• Under-replicated blocks ： 正在复制块数量，可采用 hadoop fsck -blocks 解决问题

• Mis-replicated blocks ： 正复制的缺少复制块的数量

• Missing replicas ： 缺少复制块的数量，通常情况下Under-replicated blocks\Mis-- replicated blocks\Missing replicas 都为0，则集群健康，如果不为0，则缺失块了

• Corrupt blocks ： 坏块的数量，这个值不为0，则说明当前集群有不可恢复的块，即数据有丢失了

当下架节点时Under-replicated blocks\Mis-replicated blocks\Missing replicas，这三个参数会显示当前，需要补的块的数量，集群会自动补全，当三个参数都为0时，则集群块的复制块完全了。

• 检测缺失块

[root@node01 ~]# su hdfs -c "hdfs fsck -list-corruptfileblocks"
Connecting to namenode via http://node01:9870/fsck?ugi=hdfs&listcorruptfileblocks=1&path=%2F
The filesystem under path '/' has 0 CORRUPT files

[root@node01 ~]# su hdfs -c "hdfs fsck / | egrep -v '^\.+$' | grep -v eplica"
Connecting to namenode via http://node01:9870/fsck?ugi=hdfs&path=%2F
FSCK started by hdfs (auth:SIMPLE) from /192.168.1.81 for path / at Wed Apr 29 09:27:20 CST 2020

Status: HEALTHY
 Number of data-nodes:	3
 Number of racks:		1
 Total dirs:			1890
 Total symlinks:		0

 Total size:	387368629 B
 Total files:	85
 Total blocks (validated):	85 (avg. block size 4557277 B)
 Missing blocks:		0
 Corrupt blocks:		0

Erasure Coded Block Groups:
 Total size:	0 B
 Total files:	0
 Total block groups (validated):	0
 Minimally erasure-coded block groups:	0
 Over-erasure-coded block groups:	0
 Under-erasure-coded block groups:	0
 Unsatisfactory placement block groups:	0
 Average block group size:	0.0
 Missing block groups:		0
 Corrupt block groups:		0
 Missing internal blocks:	0
FSCK ended at Wed Apr 29 09:27:20 CST 2020 in 96 milliseconds


The filesystem under path '/' is HEALTHY