一、HDFS
基本的检查操作如下:
部分举例如下:
1、查看空间使用情况:hdfs dfs -df -h
Filesystem Size Used Available Use%
hdfs://mycluster 288.8 G 34.4 M 269.2 G 0%
2、查看文件列表:hdfs dfs -ls /
Found 4 items
drwxr-xr-x - root supergroup 0 2018-01-28 21:09 /000
drwx—— - root supergroup 0 2018-01-26 01:02 /tmp
drwxr-xr-x - root supergroup 0 2018-01-28 21:33 /user
drwxr-xr-x - root supergroup 0 2018-02-27 18:19 /wc
3、查看目录下空间占用情况:hdfs dfs -du -h /
1.4 M /000
10.8 M /tmp
1.4 M /user
3.0 M /wc
4、hdfs块检查:hdfs fsck /
Connecting to namenode via http://node02:50070
FSCK started by root (auth:SIMPLE) from /192.168.159.36 for path / at Tue Feb 27 22:36:42 CST 2018
……………………………………………….Status: HEALTHY
Total size: 17471820 B
Total dirs: 19
Total files: 55
Total symlinks: 0
Total blocks (validated): 52 (avg. block size 335996 B)
Minimally replicated blocks: 52 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Tue Feb 27 22:36:42 CST 2018 in 24 milliseconds
The filesystem under path ‘/’ is HEALTHY
处理HDFS上的文件:
5、创建和推送文件:
hdfs dfs -mkdir -p /tq/input
hdfs dfs -put tq.txt /tq/input
6、删除hdfs上的文件:
hdfs dfs -rm 文件对应的路径
7、运行JAR程序包shell:
hadoop jar hadoop-mapreduce-examples-2.6.5.jar wordcount /user/root/ccc.txt /wc/out
(XXX.jar是程序目录,wordcount是程序入口,user/root/ccc.txt是文件输入源,wc/out是文件输出源)
7、查看HDFS状态:
hadoop dfsadmin -report
查看的内容为,每个节点的信息,还有包含的datanode,以及每个datanode的情况等。
8、离开安全模式:
hadoop dfsadmin -safemode leave
9、进入安全模式:
hadoop dfsadmin -safemode enter