(6)Java 读写 hdfs文件或者目录

最新推荐文章于 2024-07-25 16:52:43 发布

GE12

最新推荐文章于 2024-07-25 16:52:43 发布

阅读量4.2k

点赞数

分类专栏： hadoop

本文链接：https://blog.csdn.net/ZhongGuoZhiChuang/article/details/54944817

版权

hadoop 专栏收录该内容

22 篇文章 0 订阅

订阅专栏

1.读取单个文件

			Date date = DateUtil.getSpecifiedDayBefore();
			String yesterday = DateUtil.dateToStr(date, "yyyy-MM-dd");
			String path = "hdfs://ip:9000/output_log/output_log_click" + yesterday;
			
		  Configuration conf = new Configuration();
			FileSystem fs = FileSystem.get(URI.create(path), conf);
			FSDataInputStream hdfsInStream = fs.open(new Path(path));
			InputStreamReader isr = new InputStreamReader(hdfsInStream, "utf-8");
			BufferedReader br = new BufferedReader(isr);
			String line;
			// int k = 0;
			while ((line = br.readLine()) != null) {
				System.out.println(line);
			}

2.读取文件夹

		Date date = DateUtil.getSpecifiedDayBefore();
		String yesterday = DateUtil.dateToStr(date, "yyyy-MM-dd");
		String path = "hdfs://ip:9000/output_log/output_log_click" + yesterday;

		Configuration conf = new Configuration();
		FileSystem fs = FileSystem.get(URI.create(path), conf);
		FileStatus[] status = fs.listStatus(new Path(path));
		for (FileStatus file : status) {
			if (!file.getPath().getName().startsWith("newsMap")) {
                continue;
            }
			FSDataInputStream hdfsInStream = fs.open(file.getPath());
			InputStreamReader isr = new InputStreamReader(hdfsInStream, "utf-8");
			BufferedReader br = new BufferedReader(isr);
			String line;
			// int k = 0;
			while ((line = br.readLine()) != null) {
				System.out.println(line);
			}
			
		}