hadoop的java实现

1、新建maven项目

注意修改maven下载位置

2、再建一个子maven项目

3、设置依赖

• 父maven（官网—搜索hadoop—前三个依赖—版本2.7.6）

      <dependencyManagement>
      <dependencies>
          <dependency>
              <groupId>org.apache.hadoop</groupId>
              <artifactId>hadoop-common</artifactId>
              <version>2.7.6</version>
          </dependency>
          <dependency>
              <groupId>org.apache.hadoop</groupId>
              <artifactId>hadoop-client</artifactId>
              <version>2.7.6</version>
          </dependency>
          <dependency>
              <groupId>org.apache.hadoop</groupId>
              <artifactId>hadoop-hdfs</artifactId>
              <version>2.7.6</version>
          </dependency>
      </dependencies>
  
      </dependencyManagement>
  

• 子maven

      <dependencies>
          <dependency>
              <groupId>org.apache.hadoop</groupId>
              <artifactId>hadoop-client</artifactId>
          </dependency>
          <dependency>
              <groupId>org.apache.hadoop</groupId>
              <artifactId>hadoop-hdfs</artifactId>
          </dependency>
          <dependency>
              <groupId>org.apache.hadoop</groupId>
              <artifactId>hadoop-common</artifactId>
          </dependency>
      </dependencies>
  

4、编写程序

快照保存

集群打开

http://master:50070/

• 前奏准备

          //获取HDFS的文件系统
          Configuration conf = new Configuration();
          //指定连接集群的主节点，以及端口号
          conf.set("fs.defaultFS","hdfs://master:9000");
          //如果想要操作HDFS。就必须获取到hdfs提供的抽象父类的实现，FileSystem
          FileSystem fs = FileSystem.get(conf);
  

• 利用fs进行文件操作

          //创建文件夹
          boolean mkdirs = fs.mkdirs(new Path("/test"));
          System.out.println(mkdirs);
  
          //文件上传
          fs.copyFromLocalFile(new Path("D:\\1\\数加大数据\\hadoop\\students.txt"),
                  new Path("/test/"));
  
          //文件下载
          fs.copyToLocalFile(new Path("/test/students.txt"),
                  new Path("D:\\stu"));//文件夹
  
          //文件的删除
          boolean delete = fs.delete(new Path("/test/students.txt"),true);
          System.out.println(delete);
  
          //文件改名
          boolean rename = fs.rename(new Path("/test/students.txt"), new Path("/test/s.txt"));
          System.out.println(rename);
  
          //查看文件的描述
          RemoteIterator<LocatedFileStatus> listFiles = fs.listFiles(new Path("/test"), true);
          System.out.println("/test目录下文件信息：");
          while (listFiles.hasNext()){
              System.out.println("===========文件查看=============");
              LocatedFileStatus next = listFiles.next();
              System.out.println(next);
  
              String name = next.getPath().getName();
              System.out.println("文件名："+name);
              long len = next.getLen();
              System.out.println("文件字节数："+len);
              FsPermission permission = next.getPermission();
              System.out.println("权限："+permission);
  
              //查看存储块信息
              BlockLocation[] locations = next.getBlockLocations();
              for (BlockLocation location : locations) {
                  //获取块对应存储主机
                  String[] hosts = location.getHosts();
                  for (String host : hosts) {
                      System.out.println(host);
                  }
              }
          }
  

  • 使用IO流

           //创建输入流
            FileInputStream fileInputStream = new FileInputStream(new File("D:\\1\\bigDataProjects\\hadoopStudy\\hadoopHDFS\\data\\love.txt"));
    
            //创建输出流
            FSDataOutputStream fsDataOutputStream = fs.create(new Path("/test/love.txt"));
            //使用流进行拷贝
            int copy = IOUtils.copy(fileInputStream, fsDataOutputStream);
            System.out.println("使用流进行拷贝:"+copy);
            IOUtils.closeQuietly(fileInputStream);
            IOUtils.closeQuietly(fsDataOutputStream);
    

• 关闭

          fs.close();
  

所有对文件的上传下载，都要对路径写详细，写出文件名.txt