HDFS上传本地的整个文件夹

最新推荐文章于 2024-06-05 08:04:47 发布

w_l666

最新推荐文章于 2024-06-05 08:04:47 发布

阅读量177

点赞数 1

文章标签： hdfs hadoop 大数据 java 数据库

本文链接：https://blog.csdn.net/w_l666/article/details/133273993

版权

需要引入的maven依赖如下：

 <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>3.2.4</version>
            <exclusions>
                <exclusion>
                    <artifactId>servlet-api</artifactId>
                    <groupId>javax.servlet</groupId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.logging.log4j</groupId>
                    <artifactId>log4j-slf4j-impl</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.log4j</groupId>
                    <artifactId>org.apache.hive.org.apache.log4j</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <!-- hadoop -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>3.2.4</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>servlet-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>3.2.4</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>servlet-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

代码如下：

  public  void uploadFolder(String localFolderPath, String hdfsFolderPath){
        Configuration conf = new Configuration();
        //设置hdfs的访问路径
        conf.set("fs.defaultFS", "hdfs://192.168.2.20:9010");
        Path localPath = new Path(localFolderPath);
        Path hdfsPath = new Path(hdfsFolderPath);
        try (FileSystem fs = FileSystem.get(conf)) {
            fs.copyFromLocalFile(false, true, localPath, hdfsPath);
            log.info("向hdfs上传文件夹成功："+localPath.toString() + " -> " + hdfsPath.toString());
        }catch (IOException e) {
            log.error("向hdfs上传文件夹失败: "+localPath.toString() + " -> " + hdfsPath.toString());
            log.error(e.getMessage(),e);
        }
    }

注意：本地的localPath文件夹路径，最后一定要写一个斜杠（例如：/home/project/）,project后面的斜杠一定要加。如果不加，会将这个东西当成一个文件，而不是一个文件夹。