参考链接:http://lxw1234.com/archives/2015/08/452.htm
在HDFS中,提供了fsck命令,用于检查HDFS上文件和目录的健康状态、获取文件的block信息和位置信息等。
fsck命令必须由HDFS超级用户来执行,普通用户无权限。
- [hadoop@dev~]$ hdfs fsck
- Usage:DFSck[-list-corruptfileblocks|[-move|-delete|-openforwrite][-files[-blocks[-locations|-racks]]]]
- start checkingfromthispath
- -move move corrupted files to/lost+found
- -deletedeletecorrupted files
- -filesprintoutfiles beingchecked
- -openforwriteprintoutfiles openedforwrite
- -includeSnapshots include snapshot dataifthe given path indicates a snapshottable directoryorthere are snapshottable directories under it
- -list-corruptfileblocksprintoutlist of missing blocksandfiles they belong to
- -blocksprintoutblock report
- -locationsprintoutlocationsforevery block
- -racksprintoutnetwork topologyfordata-node locations
下面介绍每一个选项的含义及用法。
查看文件中损坏的块(-list-corruptfileblocks)
- [hadoop@dev~]$ hdfs fsck/hivedata/warehouse/liuxiaowen.db/lxw_product_names/-list-corruptfileblocks
- Thefilesystem under path'/hivedata/warehouse/liuxiaowen.db/lxw_product_names/'has0CORRUPT files
将损坏的文件移动至/lost+found目录(-move)
- [hadoop@dev~]$ hdfs fsck/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168-move
- FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168atThuAug1309:36:35CST2015
- .Status:HEALTHY
- Totalsize:13497058B
- Totaldirs:0
- Totalfiles:1
- Totalsymlinks:0
- Totalblocks(validated):1(avg.block size13497058B)
- Minimallyreplicated blocks:1(100.0%)
- Over-replicated blocks:0(0.0%)
- Under-replicated blocks:0(0.0%)
- Mis-replicated blocks:0(0.0%)
- Defaultreplication factor:2
- Averageblock replication:2.0
- Corruptblocks:0
- Missingreplicas:0(0.0%)
- Numberof data-nodes:15
- Numberof racks:1
- FSCK ended atThuAug1309:36:35CST2015in1milliseconds
- Thefilesystem under path'/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168'isHEALTHY
删除损坏的文件(-delete)
- [hadoop@dev~]$ hdfs fsck/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168-delete
- FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168atThuAug1309:37:58CST2015
- .Status:HEALTHY
- Totalsize:13497058B
- Totaldirs:0
- Totalfiles:1
- Totalsymlinks:0
- Totalblocks(validated):1(avg.block size13497058B)
- Minimallyreplicated blocks:1(100.0%)
- Over-replicated blocks:0(0.0%)
- Under-replicated blocks:0(0.0%)
- Mis-replicated blocks:0(0.0%)
- Defaultreplication factor:2
- Averageblock replication:2.0
- Corruptblocks:0
- Missingreplicas:0(0.0%)
- Numberof data-nodes:15
- Numberof racks:1
- FSCK ended atThuAug1309:37:58CST2015in1milliseconds
- Thefilesystem under path'/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168'isHEALTHY
检查并列出所有文件状态(-files)
- [hadoop@dev~]$ hdfs fsck/hivedata/warehouse/liuxiaowen.db/lxw_product_names/-files
- FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/hivedata/warehouse/liuxiaowen.db/lxw_product_names/atThuAug1309:39:38CST2015
- /hivedata/warehouse/liuxiaowen.db/lxw_product_names/dir
- /hivedata/warehouse/liuxiaowen.db/lxw_product_names/_SUCCESS0bytes,0block(s):OK
- /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000013583807bytes,1block(s):OK
- /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000113577427bytes,1block(s):OK
- /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000213588601bytes,1block(s):OK
- /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000313479213bytes,1block(s):OK
- /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000413497012bytes,1block(s):OK
- /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000513557451bytes,1block(s):OK
- /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000613580267bytes,1block(s):OK
- /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000713486035bytes,1block(s):OK
- /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000813481498bytes,1block(s):OK
- ...
检查并打印正在被打开执行写操作的文件(-openforwrite)
- [hadoop@dev~]$ hdfs fsck/hivedata/warehouse/liuxiaowen.db/lxw_product_names/-openforwrite
- FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/hivedata/warehouse/liuxiaowen.db/lxw_product_names/atThuAug1309:41:28CST2015
- ....................................................................................................
- ....................................................................................................
- .Status:HEALTHY
- Totalsize:2704782548B
- Totaldirs:1
- Totalfiles:201
- Totalsymlinks:0
- Totalblocks(validated):200(avg.block size13523912B)
- Minimallyreplicated blocks:200(100.0%)
- Over-replicated blocks:0(0.0%)
- Under-replicated blocks:0(0.0%)
- Mis-replicated blocks:0(0.0%)
- Defaultreplication factor:2
- Averageblock replication:2.0
- Corruptblocks:0
- Missingreplicas:0(0.0%)
- Numberof data-nodes:15
- Numberof racks:1
- FSCK ended atThuAug1309:41:28CST2015in10milliseconds
- Thefilesystem under path'/hivedata/warehouse/liuxiaowen.db/lxw_product_names/'isHEALTHY
打印文件的Block报告(-blocks)
需要和-files一起使用。
- [hadoop@dev~]$ hdfs fsck/logs/site/2015-08-08/lxw1234.log-files-blocks
- FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/logs/site/2015-08-08/lxw1234.log atThuAug1309:45:59CST2015
- /logs/site/2015-08-08/lxw1234.log7408754725bytes,56block(s):OK
- 0.BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728repl=2
- 1.BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728repl=2
- 2.BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728repl=2
- 3.BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728repl=2
- 4.BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728repl=2
- 5.BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728repl=2
- 6.BP-1034052771-172.16.212.130-1405595752491:blk_1075892999_2152398 len=134217728repl=2
- 7.BP-1034052771-172.16.212.130-1405595752491:blk_1075893000_2152399 len=134217728repl=2
- 8.BP-1034052771-172.16.212.130-1405595752491:blk_1075893001_2152400 len=134217728repl=2
- 9.BP-1034052771-172.16.212.130-1405595752491:blk_1075893002_2152401 len=134217728repl=2
- 10.BP-1034052771-172.16.212.130-1405595752491:blk_1075893003_2152402 len=134217728repl=2
- 11.BP-1034052771-172.16.212.130-1405595752491:blk_1075893004_2152403 len=134217728repl=2
- 12.BP-1034052771-172.16.212.130-1405595752491:blk_1075893005_2152404 len=134217728repl=2
- 13.BP-1034052771-172.16.212.130-1405595752491:blk_1075893006_2152405 len=134217728repl=2
- 14.BP-1034052771-172.16.212.130-1405595752491:blk_1075893007_2152406 len=134217728repl=2
- ...
其中,/logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s): 表示文件的总大小和block数;
0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2
1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2
2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2
前面的0. 1. 2.代表该文件的block索引,56的文件块,就从0-55;
BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381表示block id;
len=134217728 表示该文件块大小;
repl=2 表示该文件块副本数;
打印文件块的位置信息(-locations)
需要和-files -blocks一起使用。
- [hadoop@dev~]$ hdfs fsck/logs/site/2015-08-08/lxw1234.log-files-blocks-locations
- FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/logs/site/2015-08-08/lxw1234.log atThuAug1309:45:59CST2015
- /logs/site/2015-08-08/lxw1234.log7408754725bytes,56block(s):OK
- 0.BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728repl=2[172.16.212.139:50010,172.16.212.135:50010]
- 1.BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728repl=2[172.16.212.140:50010,172.16.212.133:50010]
- 2.BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728repl=2[172.16.212.136:50010,172.16.212.141:50010]
- 3.BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728repl=2[172.16.212.133:50010,172.16.212.135:50010]
- 4.BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728repl=2[172.16.212.142:50010,172.16.212.139:50010]
- 5.BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728repl=2[172.16.212.133:50010,172.16.212.139:50010]
- 6.BP-1034052771-172.16.212.130-1405595752491:blk_1075892999_2152398 len=134217728repl=2[172.16.212.141:50010,172.16.212.135:50010]
- 7.BP-1034052771-172.16.212.130-1405595752491:blk_1075893000_2152399 len=134217728repl=2[172.16.212.144:50010,172.16.212.142:50010]
- 8.BP-1034052771-172.16.212.130-1405595752491:blk_1075893001_2152400 len=134217728repl=2[172.16.212.133:50010,172.16.212.138:50010]
- 9.BP-1034052771-172.16.212.130-1405595752491:blk_1075893002_2152401 len=134217728repl=2[172.16.212.140:50010,172.16.212.134:50010]
- ...
和打印出的文件块信息相比,多了一个文件块的位置信息:[172.16.212.139:50010, 172.16.212.135:50010]
打印文件块位置所在的机架信息(-racks)
- [hadoop@dev~]$ hdfs fsck/logs/site/2015-08-08/lxw1234.log-files-blocks-locations-racks
- FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/logs/site/2015-08-08/lxw1234.log atThuAug1309:45:59CST2015
- /logs/site/2015-08-08/lxw1234.log7408754725bytes,56block(s):OK
- 0.BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728repl=2[/default-rack/172.16.212.139:50010,/default-rack/172.16.212.135:50010]
- 1.BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728repl=2[/default-rack/172.16.212.140:50010,/default-rack/172.16.212.133:50010]
- 2.BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728repl=2[/default-rack/172.16.212.136:50010,/default-rack/172.16.212.141:50010]
- 3.BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728repl=2[/default-rack/172.16.212.133:50010,/default-rack/172.16.212.135:50010]
- 4.BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728repl=2[/default-rack/172.16.212.142:50010,/default-rack/172.16.212.139:50010]
- 5.BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728repl=2[/default-rack/172.16.212.133:50010,/default-rack/172.16.212.139:50010]
- ...
和前面打印出的信息相比,多了机架信息:[/default-rack/172.16.212.139:50010, /default-rack/172.16.212.135:50010]
来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/31347383/viewspace-2119093/,如需转载,请注明出处,否则将追究法律责任。
转载于:http://blog.itpub.net/31347383/viewspace-2119093/