1、基础命令
# 版本查看
hadoop version
# 启动
cd /opt/homebrew/sbin
start-all.sh
# 查看进程
jps
# 关闭
stop-all.sh
2、HDFS 命令
# 创建目录
hdfs dfs -mkdir -p /user/xiaokkk/documents
# 删除目录
hdfs dfs -rm -r /user/xiaokkk/documents
# 上传文件
hdfs dfs -put /Users/xiaokkk/Desktop/test1.txt /data
# 下载文件
hdfs dfs -get /data/test.txt test2.txt
# 显示文件内容
hdfs dfs -cat /data/test.txt
hdfs dfs -tail /data/test.txt
3、hdfs dfsadmin -report:
(base) xiaokkk@xiaokkkdeMacBook-Pro Desktop % hdfs dfsadmin -report
2024-09-02 14:49:46,687 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Configured Capacity: 245107195904 (228.27 GB)
Present Capacity: 15063244977 (14.03 GB)
DFS Remaining: 15063072768 (14.03 GB)
DFS Used: 172209 (168.17 KB)
DFS Used%: 0.00%
Replicated Blocks:
Under replicated blocks: 3
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Low redundancy blocks with highest priority to recover: 3
Pending deletion blocks: 0
Erasure Coded Block Groups:
Low redundancy block groups: 0
Block groups with corrupt internal blocks: 0
Missing block groups: 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
-------------------------------------------------
Live datanodes (1):
Name: 127.0.0.1:9866 (localhost)
Hostname: 192.168.82.162
Decommission Status : Normal
Configured Capacity: 245107195904 (228.27 GB)
DFS Used: 172209 (168.17 KB)
Non DFS Used: 230043950927 (214.25 GB)
DFS Remaining: 15063072768 (14.03 GB)
DFS Used%: 0.00%
DFS Remaining%: 6.15%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Mon Sep 02 14:49:45 CST 2024
Last Block Report: Mon Sep 02 14:22:18 CST 2024
Num of Blocks: 3
从您提供的hdfs dfsadmin -report
输出来看,您的Hadoop集群配置和运行状态如下:
-
总体容量:
- 配置容量:228.27 GB
- 当前可用容量:14.03 GB
- 已用容量:168.17 KB
- 使用率:0.00%
-
DataNode状态:
- 活动DataNode数量:1
- 主机名:192.168.82.162
- 配置容量:228.27 GB
- 已用容量:168.17 KB
- 剩余容量:14.03 GB
- 使用率:0.00%
- 剩余比例:6.15%
-
NameNode状态:
- 存在3个未完全复制的块(Under replicated blocks)
- 没有损坏的块(Blocks with corrupt replicas)
- 没有丢失的块(Missing blocks)
4、附录
Hadoop 管理命令
-
hdfs dfsadmin -report
- 显示HDFS集群的状态报告。
- 示例:
hdfs dfsadmin -report
-
hdfs dfsadmin -safemode [enter|leave|get]
- 控制HDFS的安全模式。
- 示例:
hdfs dfsadmin -safemode leave
-
hdfs fsck / [options]
- 检查文件系统的完整性。
- 示例:
hdfs fsck / -delete
-
hdfs balancer -threshold [percentage]
- 平衡DataNode之间的数据分布。
- 示例:
hdfs balancer -threshold 10
-
hdfs namenode -format
- 格式化NameNode。
- 示例:
hdfs namenode -format
-
hdfs secondarynamenode -start
- 启动SecondaryNameNode。
- 示例:
hdfs secondarynamenode -start
YARN 管理命令
-
yarn application -list
- 列出所有活动的应用程序。
- 示例:
yarn application -list
-
yarn application -kill [appId]
- 终止一个应用程序。
- 示例:
yarn application -kill application_1234567890123456
-
yarn node -list
- 列出所有节点的状态。
- 示例:
yarn node -list
-
yarn cluster -metrics
- 显示集群的度量信息。
- 示例:
yarn cluster -metrics
-
yarn jar [jarFile] [mainClass] [args]
- 提交一个MapReduce作业。
- 示例:
yarn jar myjob.jar com.example.MyJob
其他命令
-
hadoop version
- 显示Hadoop的版本信息。
- 示例:
hadoop version
-
hadoop classpath
- 显示Hadoop的类路径。
- 示例:
hadoop classpath