windows搭建HDFS客户端

1、去Apache官网下载hadoop解压包(官网)

2、找到你需要的安装包下载

3、下载后解压，配置环境变量

3、下载尽量对应版本的 hadoop.dll 和 winutils.exe ，或者2.x的用2.x，3.x用3.x的（下载地址1，下载地址2）

4、然后将下载后的 winutils.exe 放入解压后的 %HADOOP_HOME%\bin 目录下，将hadoop.dll 放入 C:\Windows\System32 目录下

5、新建Maven项目测试

 <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <hadoop.version>3.1.0</hadoop.version>
    </properties>


    <dependencies>
        <!-- https://mvnrepository.com/artifact/junit/junit -->
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>RELEASE</version>
        </dependency>

        <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>${hadoop.version}</version>
    </dependency>


        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
        </dependency>


        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

    </dependencies>

6、写代码

    @Test
    public void testConnection() throws Exception {
        //如果当前用户没有写hdfs权限的话，需要指定hdfs用户或者其他具有hdfs写权限的用户
        System.setProperty("HADOOP_USER_NAME","hdfs");

        /**如果不指定你本地hadoop安装路径的话就会报以下错误 :
         * 2021-02-14 15:59:31,262 WARN org.apache.hadoop.util.Shell Did not find winutils.exe: {}
         * java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
         */
        System.setProperty("hadoop.home.dir", "D:\\study\\Software\\hadoop-3.1.0");

        /**
         * 如果在这里指定要连接的集群地址的话就不需要把core-site.xml文件放入Resources目录下
         * 否则就将core-site.xml文件放入Resources目录下，并添加以下内容
         *      <property>
         *         <name>fs.default.name</name>
         *         <value>hdfs://hadoop000:8020</value>
         *      </property>
         */
//        FileSystem fileSystem = FileSystem.get(URI.create("hdfs://hadoop000:8020"), new Configuration());

        FileSystem fileSystem = FileSystem.get(new Configuration());
        FileStatus[] fileStatuses = fileSystem.listStatus(new Path("/"));

        for (FileStatus fileStatus : fileStatuses) {
            System.out.println(fileStatus.getPath().toString());
        }


    }