HDFS的访问

1 通过Shell

官网api传送门
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/FileSystemShell.html

1.1 创建目录

hdfs dfs -mkdir 文件夹名字

hdfs dfs -mkdir /installpkgs

1.2 上传文件到hdfs

hdfs dfs -put 文件名 新文件名

hdfs dfs -put jdk-7u79-linux-x64.rpm /installpkgs/jdk-7u79-linux-x64.rpm

或者使用
hdfs dfs -copyFromLocal 本地文件 远程文件

hdfs dfs -copyFromLocal /opt/software/hadoop-2.7.0.tar.gz /installpkgs/hadoop-2.7.0.tar.gz

put跟copyFromLocal区别
官网解释

1.3 列出HDFS上的文件

hdfs dfs -ls 目录名

hdfs dfs -ls /
hdfs dfs -ls /installpkgs 

1.4 查看HDFS下某个文件的内容

hdfs dfs -cat 文件名

hdfs dfs -cat /docs/test.txt

创建个测试文件

1.5 将hdfs中的文件复制到本地系统

hdfs dfs -get 文件名 新文件名

hdfs dfs -get /docs/test.txt ~/test.txt

或者使用
hdfs dfs -copyToLocal 远程文件 本地文件

hdfs dfs -copyToLocal /docs/test.txt ~/test2.txt

与get的区别
官网解释

1.6 删除hdfs下的文档

hdfs dfs -rmr 文件

hdfs dfs -rmr 

1.7 格式化hdfs

hdfs namenode -format

1.8 关闭HDFS

stop-dfs.sh

1.9 启动HDFS

start-dfs.sh

确认启动成功

2 通过Java api操作

官网api传送门
https://hadoop.apache.org/docs/stable/api/index.html

前置工作
a) 创建一个Java项目，用maven来管理

导入依赖jar包

   <properties>
        <hadoop.version>2.7.0</hadoop.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
        </dependency>
    </dependencies>

没有用maven的自己得去手动加jar包，不推荐

b) 拷贝配置文件core-site.xml,hdfs-site.xml到resource目录下

2.1 创建目录

    @Test
    public void testMkdir() throws IOException {
        Configuration conf = new Configuration();
        FileSystem fs = FileSystem.get(conf);
        Path path = new Path("/app");
        fs.mkdirs(path);
    }

2.1.1 异常报错UnknownHostException: masternode

解决
把master换成具体的ip地址 或者修改主机的host文件

2.1.2 异常报错权限不够

解决
修改目录的访问权限

hdfs dfs -chmod 777 /

注：生产环境不要把根目录权限搞这么大
或者代码中加入

System.setProperty("HADOOP_USER_NAME", "root");

修改后测试通过

2.2 上传文件到hdfs

    @Test
    public void testCopyFromLocal() throws IOException {
	    Configuration conf = new Configuration();
        System.setProperty("HADOOP_USER_NAME", "root");
        FileSystem fs = FileSystem.get(conf);
        Path srcPath = new Path("c://testCopyFromLocal.txt");
        Path destPath = new Path("/docs/testCopyFromLocal.txt");
        fs.copyFromLocalFile(srcPath, destPath);
    }

创建一个文件准备把它上传到hdfs


查看结果
成功上传到hdfs上

2.3 列出HDFS上的文件

    @Test
    public void testLs() throws IOException {
    	Configuration conf = new Configuration();
        System.setProperty("HADOOP_USER_NAME", "root");
        FileSystem fs = FileSystem.get(conf);
        Path path = new Path("/");
        FileStatus[] fileStatuses = fs.listStatus(path);
        for (FileStatus fileStatus : fileStatuses) {
            System.out.println(fileStatus.getPath());
        }
    }

2.3.1 拓展递归显示文件

    @Test
    public void testListFiles() throws IOException {
        System.setProperty("HADOOP_USER_NAME", "root");
        Path path = new Path("/");
        recursiveListFiles(path);
    }

    public void recursiveListFiles(Path path)throws IOException {
        FileSystem fs = FileSystem.get(conf);
        FileStatus[] fileStatuses = fs.listStatus(path);
        for (FileStatus fileStatus : fileStatuses) {
            if (fileStatus.isDirectory()) {
                System.out.println(fileStatus.getPath());
                recursiveListFiles(fileStatus.getPath());
            } else {
                System.out.println(fileStatus.getPath());
            }
        }
    }

2.4 查看HDFS下某个文件的内容

    @Test
    public void testCat() throws IOException {
    	Configuration conf = new Configuration();
        System.setProperty("HADOOP_USER_NAME", "root");
        Path path = new Path("/docs/test.txt");
        FileSystem fs = FileSystem.get(conf);
        FSDataInputStream in = fs.open(path);
        int i = 0;
        while ((i = in.read()) != -1) {
            System.out.print(Character.toChars(i));
        }
    }

2.5 将hdfs中的文件复制到本地系统

    @Test
    public void testCopyToLocal() throws IOException {
    	Configuration conf = new Configuration();
        System.setProperty("HADOOP_USER_NAME", "root");
        Path src = new Path("/docs/test.txt");
        Path dst = new Path("f://test.txt");
        FileSystem fs = FileSystem.get(conf);
        fs.copyToLocalFile(false,src,dst,true);
    }

2.6 删除hdfs下的文档

    @Test
    public void testRm() throws IOException {
        Configuration conf = new Configuration();
        System.setProperty("HADOOP_USER_NAME", "root");
        Path path = new Path("/installpkgs/test.txt");
        FileSystem fs = FileSystem.get(conf);
        boolean isDeleted = fs.delete(path, true);
        if (isDeleted) {
            System.out.println("删除成功");
        }
    }

准备一个替死鬼


送他去见佛祖

删除成功

2.7 格式化hdfs

暂时没找到，老老实实用shell吧

hdfs namenode -format

2.8 关闭hdfs

暂时没找到，老老实实用shell吧

stop-dfs.sh

2.9 启动hdfs

暂时没找到，老老实实用shell吧

start-dfs.sh