hdfs如何查找指定目录是否文件_hdfs fsck命令查看HDFS文件对应的文件块信息(Block)和位置信息(Locations)...

关键字：hdfs fsck、block、locations

在HDFS中，提供了fsck命令，用于检查HDFS上文件和目录的健康状态、获取文件的block信息和位置信息等。

fsck命令必须由HDFS超级用户来执行，普通用户无权限。

[hadoop@dev ~]$ hdfs fsck

Usage: DFSck [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]]

start checking from this path

-move move corrupted files to /lost+found

-delete delete corrupted files

-files print out files being checked

-openforwrite print out files opened for write

-includeSnapshots include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it

-list-corruptfileblocks print out list of missing blocks and files they belong to

-blocks print out block report

-locations print out locations for every block

-racks print out network topology for data-node locations

下面介绍每一个选项的含义及用法。

查看文件中损坏的块(-list-corruptfileblocks)

[hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ -list-corruptfileblocks

The filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/' has 0 CORRUPT files

将损坏的文件移动至/lost+found目录(-move)

[hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 -move

FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 at Thu Aug 13 09:36:35 CST 2015

.Status: HEALTHY

Total size: 13497058 B

Total dirs: 0

Total files: 1

Total symlinks: 0

Total blocks (validated): 1 (avg. block size 13497058 B)

Minimally replicated blocks: 1 (100.0 %)

Over-replicated blocks: 0 (0.0 %)

Under-replicated blocks: 0 (0.0 %)

Mis-replicated blocks: 0 (0.0 %)

Default replication factor: 2

Average block replication: 2.0

Corrupt blocks: 0

Missing replicas: 0 (0.0 %)

Number of data-nodes: 15

Number of racks: 1

FSCK ended at Thu Aug 13 09:36:35 CST 2015 in 1 milliseconds

The filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168' is HEALTHY

删除损坏的文件(-delete)

[hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liu