大数据技术之_04_Hadoop学习_01_HDFS
第1章 HDFS概述
1.1 HDFS产出背景及定义
[外链图片转存失败(img-QbuwkvX1-1562298793078)(https://s2.ax1x.com/2019/02/05/kY1wKx.png)]
其他文件管理系统:
[外链图片转存失败(img-PN5GO8tY-1562298793078)(https://s2.ax1x.com/2019/02/05/kY1GaF.png)]
1.2 HDFS优缺点
优点
[外链图片转存失败(img-CtR0jTbe-1562298793079)(https://s2.ax1x.com/2019/02/05/kY11bT.png)]
缺点
[外链图片转存失败(img-ldqwbcUz-1562298793079)(https://s2.ax1x.com/2019/02/05/kY18VU.png)]
1.3 HDFS组成架构
1)NameNode(nn)和DataNode(dn)
[外链图片转存失败(img-Oyf4KcYr-1562298793079)(https://s2.ax1x.com/2019/02/05/kY1lrV.png)]
2)Client和Secondary NameNode(2nn)
[外链图片转存失败(img-YweL61h4-1562298793079)(https://s2.ax1x.com/2019/02/05/kY1J54.png)]
1.4 HDFS文件块大小(面试重点)
[外链图片转存失败(img-wzfiMDct-1562298793079)(https://s2.ax1x.com/2019/02/05/kY1NG9.png)]
思考:为什么块的大小不能设置太小,也不能设置太大?
[外链图片转存失败(img-KxKDsycU-1562298793080)(https://s2.ax1x.com/2019/02/05/kY10r6.png)]
传统硬盘HDD(Hard Disk Drive)传输速率:100MB/s
固态硬盘SSD(Solid State Drive)传输速率:500MB/s
混合硬盘HHD(Hybrid Harddrive)传输速率:300MB/s
PCIe固态硬盘SSD(Solid State Drive)传输速率:1500MB/s
第2章 HDFS的Shell操作(开发重点)
1、基本语法
bin/hadoop fs 具体命令 OR bin/hdfs dfs 具体命令
dfs是fs的实现类。
2、命令大全
[[atguigu@hadoop102 hadoop-2.7.2]$ bin/hadoop fs
Usage: hadoop fs [generic options]
[-appendToFile <localsrc> ... <dst>]
[-cat [-ignoreCrc] <src> ...]
[-checksum <src> ...]
[-chgrp [-R] GROUP PATH...]
[-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
[-chown [-R] [OWNER][:[GROUP]] PATH...]
[-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>]
[-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
[-count [-q] [-h] <path> ...]
[-cp [-f] [-p | -p[topax]] <src> ... <dst>]
[-createSnapshot <snapshotDir> [<snapshotName>]]
[-deleteSnapshot <snapshotDir> <snapshotName>]
[-df [-h] [<path> ...]]
[-du [-s] [-h] <path> ...]
[-expunge]
[-find <path> ... <expression> ...]
[-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
[-getfacl [-R] <path>]
[-getfattr [-R] {-n name | -d} [-e en] <path>]
[-getmerge [-nl] <src> <localdst>]
[-help [cmd ...]]
[-ls [-d] [-h] [-R] [<path> ...]]
[-mkdir [-p] <path> ...]
[-moveFromLocal <localsrc> ... <dst>]
[-moveToLocal <src> <localdst>]
[-mv <src> ... <dst>]
[-put [-f] [-p] [-l] <localsrc> ... <dst>]
[-renameSnapshot <snapshotDir> <oldName> <newName>]
[-rm [-f] [-r|-R] [-skipTrash] <src> ...]
[-rmdir [--ignore-fail-on-non-empty] <dir> ...]
[-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
[-setfattr {-n name [-v value] | -x name} <path>]
[-setrep [-R] [-w] <rep> <path> ...]
[-stat [format] <path> ...]
[-tail [-f] <file>]
[-test -[defsz] <path>]
[-text [-ignoreCrc] <src> ...]
[-touchz <path> ...]
[-truncate [-w] <length> <path> ...]
[-usage [cmd ...]]
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|resourcemanager:port> specify a ResourceManager
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
[atguigu@hadoop102 hadoop-2.7.2]$
3.常用命令实操
(0)启动Hadoop集群(方便后续的测试)
[atguigu@hadoop102 hadoop-2.7.2]$ sbin/start-dfs.sh
[atguigu@hadoop103 hadoop-2.7.2]$ sbin/start-yarn.sh
(1)-help:输出这个命令参数
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -help rm
(2)-ls: 显示目录信息
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -ls /
(3)-mkdir:在HDFS上创建目录
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -mkdir -p /sanguo/shuguo/
(4)-moveFromLocal:从本地系统中剪切粘贴文件到HDFS上(本地系统中不存在该文件)
[atguigu@hadoop102 hadoop-2.7.2]$ touch kongming.txt
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -moveFromLocal ./kongming.txt /sanguo/shuguo/
(5)-appendToFile:追加一个文件到已经存在的文件末尾
[atguigu@hadoop102 hadoop-2.7.2]$ touch liubei.txt
[atguigu@hadoop102 hadoop-2.7.2]$ vim liubei.txt
输入
san gu mao lu
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -appendToFile liubei.txt /sanguo/shuguo/kongming.txt
(6)-cat:显示文件内容
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -cat /sanguo/shuguo/kongming.txt
(7)-chgrp 、-chmod、-chown:跟Linux文件系统中的用法一样,修改文件所属权限
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -chmod 666 /sanguo/shuguo/kongming.txt
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -chown atguigu:atguigu /sanguo/shuguo/kongming.txt
(8)-copyFromLocal:从本地文件系统中拷贝文件到HDFS路径中去(本地系统中还存在该文件)
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -copyFromLocal README.txt /
(9)-copyToLocal:从HDFS拷贝到本地系统中
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -copyToLocal /sanguo/shuguo/kongming.txt ./
(10)-cp:从HDFS的一个路径拷贝到HDFS的另一个路径(文件还在旧的HDFS中)
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -cp /sanguo/shuguo/kongming.txt /zhuge.txt
(11)-mv:在HDFS目录中移动文件(文件不在旧的HDFS中,在新的HDFS中)
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -mv /zhuge.txt /sanguo/shuguo/
(12)-get:等同于copyToLocal,就是从HDFS下载文件到本地系统中
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -get /sanguo/shuguo/kongming.txt ./
(13)-getmerge:合并下载多个文件,比如HDFS的目录 /user/atguigu/test/ 下有多个文件:log.1,log.2,log.3,…
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -getmerge /sanguo/shuguo/* ./zaiyiqi.txt
(14)-put:等同于copyFromLocal,就是从本地文件系统中拷贝文件到HDFS路径中去
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -put ./zaiyiqi.txt /sanguo/shuguo/
(15)-tail:显示一个文件的末尾几行(因为日志文件一般是在文件的末尾不断地追加,即监控新产生的数据变化)
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -tail /sanguo/shuguo/kongming.txt
(16)-rm:删除文件或文件夹
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -rm /sanguo/shuguo/zaiyiqi.txt
(17)-rmdir:删除空目录
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -mkdir /test
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -rmdir /test
(18)-du:统计文件夹的大小信息
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -du -s -h /user/atguigu/test/
2.7 K /user/atguigu/test
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -du -h /user/atguigu/test/
1.3 K /user/atguigu/test/README.txt
1.4 K /user/atguigu/test/zaiyiqi.txt
(19)-setrep:设置HDFS中文件的副本数量
[atguigu@hadoop102 hadoop-2.7.2]$ hadoop fs -setrep 10 /kongming.txt
HDFS副本数量
[外链图片转存失败(img-rIwQwWDM-1562298793080)(https://s2.ax1x.com/2019/02/05/kY1tPJ.png)]
这里设置的副本数只是记录在NameNode的元数据中,是否真的会有这么多副本,还得看DataNode的数量。因为目前只有3台设备,最多也就3个副本,只有节点数的增加到10台时,副本数才能达到10。
第3章 HDFS客户端操作(开发重点)
3.1 HDFS客户端环境准备
1、根据自己电脑的操作系统拷贝对应的编译后的hadoop jar包到非中文路径(例如:D:\Develop\hadoop-2.7.2),如下图所示。
[外链图片转存失败(img-i8NeZabH-1562298793080)(https://s2.ax1x.com/2019/02/05/kY1U2R.png)]
2、配置HADOOP_HOME环境变量,如下图所示:
[外链图片转存失败(img-JO9RrlL6-1562298793080)(https://s2.ax1x.com/2019/02/05/kY1ax1.png)]
3、配置Path环境变量,如下图所示:
[外链图片转存失败(img-YbC47SXf-1562298793081)(https://s2.ax1x.com/2019/02/05/kY1rVO.png)]
4、创建一个Maven工程HdfsClientDemo
5、导入相应的依赖坐标+日志添加
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>RELEASE</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.8.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.7.2</version>
</dependency>
<dependency>
<groupId>jdk.tools</groupId>
<artifactId>jdk.tools</artifactId>
<version>1.8</version>
<scope>system</scope>
<systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
</dependency>
</dependencies>
注意:如果Eclipse/Idea打印不出日志,在控制台上只显示
1.log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
2.log4j:WARN Please initialize the log4j system properly.
3.log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
需要在项目的src/main/resources目录下,新建一个文件,命名为“log4j.properties”,在文件中填入以下内容:
log4j.rootLogger=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - %m%n
log4j.appender.logfile=org.apache.log4j.FileAppender
log4j.appender.logfile.File=target/spring.log
log4j.appender.logfile.layout=org.apache.log4j.PatternLayout
log4j.appender.logfile.layout.ConversionPattern=%d %p [%c] - %m%n
6、创建包名:com.atguigu.hdfs
7、创建HdfsClient类
public class HdfsClient {
/**
* 测试创建多级文件夹:在hdfs系统上创建多级文件夹
*/
@Test
public void testMkdirs() throws IOException, InterruptedException, URISyntaxException {
// 1、获取hdfs文件系统对象
Configuration configuration = new Configuration();
// 配置在集群上运行
configuration.set("fs.defaultFS", "hdfs://hadoop102:9000");
FileSystem fs = FileSystem.get(configuration);
// 2 、创建目录
fs.mkdirs(new Path("/0529/dashen"));
// 3 、关闭资源
fs.close();
}
}
8、执行程序
运行时需要配置用户名称,右键 -> Run Configurations,如下图所示:
[外链图片转存失败(img-hCArV16k-1562298793081)(https://s2.ax1x.com/2019/02/05/kY1saD.png)]
客户端去操作HDFS时,是有一个用户身份的。默认情况下,HDFS客户端API会从JVM中获取一个参数来作为自己的用户身份:-DHADOOP_USER_NAME=atguigu,atguigu为用户名称。
9、另一种【配置在集群上运行】的方式,可以不用手动配置用户名称
public class HdfsClient {
/**
* 测试创建多级文件夹:在hdfs系统上创建多级文件夹
*/
@Test
public void testMkdirs() throws IOException, InterruptedException, URISyntaxException {
// 1、 获取hdfs文件系统对象
Configuration configuration = new Configuration();
// 配置在集群上运行
// configuration.set("fs.defaultFS", "hdfs://hadoop102:9000");
// FileSystem fs = FileSystem.get(configuration);
FileSystem fs = FileSystem.get(new URI("hdfs://hadoop102:9000"), configuration, "atguigu");
// 2 、创建目录
fs.mkdirs(new Path("/0529/dashen"));
// 3 、关闭资源
fs.close();
}
}
3.2 HDFS的API操作
3.2.1 HDFS文件上传(测试参数优先级)
1、编写源代码
/**
* 测试文件上传:从本地系统上传文件至hdfs文件系统上
*/
@Test
public void testCopyFromLocalFile() throws IOException, InterruptedException, URISyntaxException {
Configuration configuration = new Configuration();
// 1、获取hdfs文件系统
FileSystem fs = FileSystem.get(new URI("hdfs://hadoop102:9000"), configuration, "atguigu");
configuration.set("dfs.replication", "2");
// 2、本地系统执行上传文件操作
fs.copyFromLocalFile(new Path("D:/temp/atguigu/0529/banzhang.txt"), new Path("/banzhang.txt"));
// 3、关闭资源
fs.close();
}
2、在/HdfsClientDemo/src/main/r