Hadoop学习第四课（HDFS常用命令+Mapreduce-架构设计 ）

1.HDFS命令与使用

hdfs dfs -Linux，-后面使用的命令和Linux的命令操作是一样的

1.1 Hadoop常用命令-----只放常用的

[hadoop@hadoop001 ~]$ hadoop
    Admin Commands:            #admin命令

    Client Commands:             #客户端命令
checknative   check native Hadoop and compression libraries availability
---->查看hadoop的版本，类库，支不支持压缩
classpath       prints the class path needed to get the Hadoop jar and the required libraries
---->打印当前大数据环境的类的路径（用的场景比较少，但是必须要知道）
fs                    run a generic filesystem user client
---->运行一个文件系统的客户端
jar <jar>         run a jar file. NOTE: please use "yarn jar" to launch YARN applications, not this command. 
---->提交jar包到yarn上

    Daemon Commands:         #进程命令

1.2 hdfs常用命令-----只放常用的

hadoop@hadoop001 ~]$ hdfs
    Admin Commands:           #admin命令
debug            run a Debug Admin to execute HDFS debug commands
---->修复作用
dfsadmin       run a DFS admin client
---->管理员命令 
fsck                run a DFS filesystem checking utility
---->对hdfs做一个系统的check检查
haadmin        run a DFS HA admin client
---->后面HA的时候重点讲

    Client Commands:            #客户端命令
getconf         get config values from configuration
---->xml文件里面的配置参数和对应的值是个kv结构（kv结构--通过参数，传入k，就能得到value值）

    Daemon Commands:       #进程命令

balancer       run a cluster balancing utility
---->集群的平衡

1.3 查看当前版本压缩情况---checknat

[hadoop@hadoop001 ~]$ hadoop checknative
2021-12-03 14:20:38,347 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2021-12-03 14:20:38,395 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2021-12-03 14:20:38,420 WARN zstd.ZStandardCompressor: Error loading zstandard native libraries: java.lang.InternalError: Cannot load libzstd.so.1 (libzstd.so.1: cannot open shared object file: No such file or directory)!
2021-12-03 14:20:38,533 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
2021-12-03 14:20:39,093 INFO nativeio.NativeIO: The native code was built without PMDK support.
Native library checking:
hadoop:  true /home/hadoop/app/hadoop-3.2.2/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  false 
snappy:  true /lib64/libsnappy.so.1
lz4:     true revision:10301
bzip2:   true /lib64/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   false libhadoop was built without ISA-L support
PMDK:    false The native code was built without PMDK support.

snappy、lz、bzip三者都是true，表示支持该压缩

如果压缩都是false ，一般需要自己编译压缩，但是使用CDH系列的就不用担心。

hadoop2.6.0-cdh5.15.1编译源码支持压缩_阿顾的博客-CSDN博客---附送编译源码支持压缩

未来：在生产上，你一定要注意压缩的版本-->hadoop软件包是否支持压缩

1.4 打印当前类路径---classpath

[hadoop@hado