Java: Hadoop文件系统的读写操作

版权声明:本文为博主原创文章,欢迎转载,请注明出处 https://blog.csdn.net/mouday/article/details/84405252

所需jar包路径:

hadoop-2.8.5/share/hadoop/common
hadoop-2.8.5/share/hadoop/common/bin

hadoop-2.8.5/share/hadoop/hdfs
hadoop-2.8.5/share/hadoop/hdfs/bin

java代码实例

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

import java.io.*;
import java.net.URI;
import java.net.URISyntaxException;

/**
 * HDFS操作,需要导入common和hdsf文件夹下的jar
 */
public class HDSFDemo {

    private FileSystem fs = null;

    HDSFDemo() throws IOException, URISyntaxException, InterruptedException {
        Configuration conf = new Configuration();

        fs = FileSystem.get(new URI("hdfs://master:9000"), conf, "root");

    }

    /**
     * 下载文件
     *
     * @throws IOException
     */
    public void download() throws IOException {
        InputStream in = fs.open(new Path("/java"));
        OutputStream out = new FileOutputStream("/Users/Desktop");
        IOUtils.copyBytes(in, out, 4096, true);
    }

    /**
     * 上传文件
     *
     * @throws IOException
     */
    public void upload() throws IOException {
        System.out.println("开始上传...");
        InputStream in = new FileInputStream("/Users/Desktop/compare.py");
        OutputStream out = this.fs.create(new Path("/compare.py"));
        IOUtils.copyBytes(in, out, 4096, true);
    }

    /**
     * 删除文件
     *
     * @throws IOException
     */
    public void delete() throws IOException {
        boolean ret = this.fs.delete(new Path("/java"), true);
        System.out.println("文件删除结果:" + ret);
    }

    /**
     * 创建文件夹
     *
     * @throws IOException
     */
    public void mkdir() throws IOException {
        boolean ret = this.fs.mkdirs(new Path("/temp"));
        System.out.println("文件创建成功:" + ret);
    }
    

    public static void main(String[] args) throws URISyntaxException, IOException, InterruptedException {
        HDSFDemo hdfs = new HDSFDemo();
        hdfs.upload();

    }
}

备注:
URI的端口号一定要和配置文件中的端口号一致

问题:

log4j:WARN Please initialize the log4j system properly

可参考:log4j:WARN Please initialize the log4j system properly 问题解决

阅读更多

扫码向博主提问

彭世瑜

非学,无以致疑;非问,无以广识
  • 擅长领域:
  • python
  • python爬虫
  • python全栈
去开通我的Chat快问

没有更多推荐了,返回首页