hadoop fsck命令详解

HDFS健康的标准：

如果所有的文件满足最小副本的要求，那么就认为文件系统是健康的。
（HDFS is considered healthy if—and only if—all files have a minimum number of replicas available）

检查HDFS的健康情况：

hadoop提供了fsck tool来对整个文件系统或者单独的文件、目录来进行健康状态的检查。

hdfs fsck命令：

   hdfs fsck <path>
          [-list-corruptfileblocks |
          [-move | -delete | -openforwrite]
          [-files [-blocks [-locations | -racks]]]
          [-includeSnapshots]
          [-storagepolicies] [-blockId <blk_Id>]

参数	描述
<path>	指定诊断的路径（既可以是目录也可以是文件）
-delete	删除损坏的文件
-files	打印出被诊断的文件
-files -blocks	打印被诊断的文件的块信息
-files -blocks -locations	打印每个块的位置
-files -blocks -racks	打印数据块的网络拓扑结构
-includeSnapshots	如果指定路径包含快照的路径或者快照在该路径下，则包含快照数据
-list-corruptfileblocks	打印出丢失的块列表以及块所属的文件
-move	将损坏的文件移动至/lost+found目录
-openforwrite	打印出正在被写入的文件
-storagepolicies	打印块的存储策略摘要
-blockId	打印出有关该块的信息

Please Note:
1. By default fsck ignores files opened for write, use -openforwrite to report such files. They are usually tagged CORRUPT or HEALTHY depending on their block allocation status
2. Option -includeSnapshots should not be used for comparing stats, should be used only for HEALTH check, as this may contain duplicates if the same file present in both original fs tree and inside snapshots.

hadoop fsck / 这个命令可以检查整个文件系统的健康状况，但是要注意它不会主动恢复备份缺失的block，这个是由NameNode单独的线程异步处理的。

使用hdfs fsck / -openforwrite命令检查HDFS健康状态，输出如图。

参数描述：

• Total blocks：区块总个数
• Average block replication：平均副本数 MISSING
• Corrupt blocks：块损坏数
• Missing replicas：丢失副本数
• Over-replicated blocks：一些文件块副本数超出了它所属文件的限定
• Under-replicated blocks :文件块数未达到所属文件要求的副本数量
• Mis-replicated blocks ：指出不满足块副本存储位置策略的块
• Status：块状态（HEALTHY/CORRUPT）

下面介绍每一个选项的含义和用法。

1.查看文件中损坏的块（-list-corruptfileblocks）

[hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ -list-corruptfileblocks
The filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/' has 0 CORRUPT files

2.将损坏的文件移动至/lost+found目录（-move）

[hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 -move
FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 at Thu Aug 13 09:36:35 CST 2015
.Status: HEALTHY
 Total size:    13497058 B
 Total dirs:    0
 Total files:   1
 Total symlinks:                0
 Total blocks (validated):      1 (avg. block size 13497058 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    2
 Average block replication:     2.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          15
 Number of racks: