hadoop dfs 也能使用、但不推荐,执行会提示:
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
常看hdfs系统使用情况
hadoop fs -du -s -h /
Hive LOAD 语法
LOAD DATA [ LOCAL ] INPATH {file_path} [ OVERWRITE ] INTO TABLE { table_name } [ PARTITION(partition_colname1=“val1”, partition_colname2=“val2”,)… ];
LOAD 示例:
– 从本地OS系统目录加载文件到Hive表
LOAD DATA LOCAL INPATH ‘/opt/frank/tb_test03.txt’ INTO TABLE tb_test03;
LOAD DATA LOCAL INPATH ‘/opt/frank/tb_test03.txt’ OVERWRITE INTO TABLE tb_test03;
LOAD DATA LOCAL INPATH ‘/opt/frank/tb_test04_pt.txt’ OVERWRITE INTO TABLE tb_test04_pt PARTITION(pt=“20240101”);
– 从HDFS系统目录加载文件到Hive表
hadoop fs -put /opt/frank/tb_test03.txt /test/
LOAD DATA INPATH ‘/test/tb_test03.txt’ INTO TABLE tb_test03;
slave 节点查看 datanode 服务状态:
$ jps -v |grep DataNode
$ hadoop dfsadmin -report
slave 节点重启(停止、启动) datanode 服务:
$ ./sbin/hadoop-daemon.sh stop datanode
$ ./sbin/hadoop-daemon.sh start datanode
查看 HDFS 文件中系统的DFS使用情况:
$ hadoop fs -du -s -h /
垃圾清理(多次执行):
$ hadoop fs -expunge
datanode 使用率占满&配置容量显示为0的问题【DFS Used%: 100.00% & Configured Capacity: 0 (0 B)】
$ hadoop dfsadmin -report
查看到使用率占满,显示:
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Configured Capacity: 0 (0 B)
Present Capacity: 0 (0 B)
DFS Remaining: 0 (0 B)
DFS Used: 0 (0 B)
DFS Used%: NaN%
Under replicated blocks: 76125
Blocks with corrupt replicas: 0
Missing blocks: 76125
Missing blocks (with replication factor 1): 21993
Live datanodes (1):
Name: 192.168.1.188:50010 (hadoop01)
Hostname: hadoop01
Decommission Status : Normal
Configured Capacity: 0 (0 B)
DFS Used: 0 (0 B)
Non DFS Used: 0 (0 B)
DFS Remaining: 0 (0 B)
DFS Used%: 100.00%
DFS Remaining%: 0.00%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Mon Mar 25 17:02:43 CST 2024
配置容量显示为0了:
Configured Capacity: 0 (0 B)
尝试多种方法、最终原因是 slave 中配置的 datanode 节点 hostname 配置成了 localhost 而导致的问题。
vi salve 把配置的 datanode 主机名配置为 hadoop01 后,重启 datanode 服务,问题就解决了。
$ ./sbin/hadoop-daemon.sh stop datanode
$ ./sbin/hadoop-daemon.sh start datanode
$ hadoop dfsadmin -report
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Configured Capacity: 98337751040 (91.58 GB)
Present Capacity: 65340043264 (60.85 GB)
DFS Remaining: 61911707648 (57.66 GB)
DFS Used: 3428335616 (3.19 GB)
DFS Used%: 5.25%
Under replicated blocks: 73720
Blocks with corrupt replicas: 0
Missing blocks: 82
Missing blocks (with replication factor 1): 21993
Live datanodes (1):
Name: 192.168.1.188:50010 (hadoop01)
Hostname: hadoop01
Decommission Status : Normal
Configured Capacity: 98337751040 (91.58 GB)
DFS Used: 3428335616 (3.19 GB)
Non DFS Used: 32997707776 (30.73 GB)
DFS Remaining: 61911707648 (57.66 GB)
DFS Used%: 3.49%
DFS Remaining%: 62.96%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Mar 25 17:27:05 CST 2024
Hive建表和LOAD数据:
– 普通表(TextFile存储格式)
drop table if exists testdb.tb_test03;
create table testdb.tb_test03 (
id int,
info string,
cnt bigint)
– partitioned by (pt_sheng string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘,’ LINES TERMINATED BY ‘\n’
STORED AS TextFile
– STORED AS INPUTFORMAT
– ‘org.apache.hadoop.mapred.TextInputFormat’
– OUTPUTFORMAT
– ‘org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat’
LOCATION ‘hdfs://192.168.1.188:9000/user/hive/warehouse/testdb.db/tb_test03’
;
– hadoop fs -ls /user/hive/warehouse/testdb.db/tb_test03.txt
show tables;
show create table testdb.tb_test03;
select * from testdb.tb_test03;
– 方法1:
$ echo ‘1,jack,95
2,frank,96
3,lucy,97
4,hack,99’ > /opt/frank/tb_test03.txt
– hiveSQL: load from Local OS dir
LOAD DATA LOCAL INPATH ‘/opt/frank/tb_test03.txt’ OVERWRITE INTO TABLE tb_test03;
– 方法2:
– shell_cmd: 先拷贝到 hdfs, 再从 hdfs 路径 load
– $ hadoop fs -rm /frank/tb_test03.txt
– $ hadoop fs -put /opt/frank/tb_test03.txt /frank/
– $ hadoop fs -cat /frank/tb_test03.txt
一个人可以走的很快,但一群人才能走的更远!不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人,都欢迎加入我们的的圈子(技术交流、学习资源、职场吐槽、大厂内推、面试辅导),让我们一起学习成长!
[外链图片转存中…(img-g3l5CZ8U-4701984712580)]
[外链图片转存中…(img-RPzLuUK4-4701984712581)]
一个人可以走的很快,但一群人才能走的更远!不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人,都欢迎加入我们的的圈子(技术交流、学习资源、职场吐槽、大厂内推、面试辅导),让我们一起学习成长!