HDFS与Java

已于 2022-07-03 10:50:48 修改
阅读量232 收藏
点赞数
分类专栏： Hadoop 文章标签： HDFS API
于 2022-06-14 22:59:33 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_51699336/article/details/125238316
版权

引入jar包

<properties>
    <hadoop.version>2.7.1</hadoop.version>
</properties>

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>${hadoop.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>${hadoop.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs</artifactId>
    <version>${hadoop.version}</version>
</dependency>

连接Hadoop客户端，去下面的API工具类修改对象位置

1.SNN配置连接方式

1.application.yml
	hdfs:
	  path: hdfs://node1:9000
	  username: root
2.Java API
	@Value("${hdfs.path}")
    private String path;
    @Value("${hdfs.username}")
    private String username;

    //获取HDFS配置信息
    public Configuration getConf() {
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", path);
        return conf;
    }

2.HA配置连接方式

1.application.yml
	hdfs:
	  path: hdfs://yc #hdfs://node1:9000
	  username: root
	  nameservices: yc
	  namenodes: nn1,nn2
	  node1: node1:8020
	  node2: node2:8020
	  provider: org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
2.Java API
	@Value("${hdfs.path}")
    private String path;
    @Value("${hdfs.username}")
    private String username;
    @Value("${hdfs.nameservices}")
    private String nameservices;
    @Value("${hdfs.namenodes}")
    private String namenodes;
    @Value("${hdfs.node1}")
    private String node1;
    @Value("${hdfs.node2}")
    private String node2;
    @Value("${hdfs.provider}")
    private String provider;

    //获取HDFS配置信息
    public Configuration getConf() {
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", path);
        conf.set("dfs.nameservices", nameservices);
        conf.set("dfs.ha.namenodes.yc", namenodes);
        conf.set("dfs.namenode.rpc-address.yc.nn1", node1);
        conf.set("dfs.namenode.rpc-address.yc.nn2", node2);
        conf.set("dfs.client.failover.proxy.provider.yc", provider);
        return conf;
    }

API工具类

1.HdfsService

package com.yc.T1_API.service;

import org.apache.hadoop.fs.BlockLocation;
import org.springframework.core.io.InputStreamResource;
import org.springframework.http.ResponseEntity;
import org.springframework.web.multipart.MultipartFile;

import java.io.IOException;
import java.net.URISyntaxException;
import java.util.List;
import java.util.Map;

public interface HdfsService {
    //获取配置文件,以map形式返回
    public Map<String, String> getConfInfoAsMap() throws InterruptedException, IOException, URISyntaxException;

    //在HDFS新建文件夹
    public boolean mkdir(String path) throws InterruptedException, IOException, URISyntaxException;

    //判断HDFS文件是否存在
    public boolean existFile(String path) throws InterruptedException, IOException, URISyntaxException;

    //读取HDFS目录信息
    public List<Map<String, Object>> readPathInfo(String path) throws InterruptedException, IOException, URISyntaxException;

    //HDFS创建文件夹
    public void createFile(String path, MultipartFile file) throws IOException, URISyntaxException, InterruptedException;

    //读取HDFS文件内容
    public String readFile(String path) throws InterruptedException, IOException, URISyntaxException;

    //在整个磁盘搜索所有的某类型的文件
    public List<Map<String, String>> listStatus(int type) throws InterruptedException, IOException, URISyntaxException;

    public List<Map<String, String>> listStatus(String type) throws InterruptedException, IOException, URISyntaxException;

    //获取某个文件的信息
    public Map<String, String> getFileInfo(String path) throws InterruptedException, IOException, URISyntaxException;

    //读取HDFS文件列表
    public List<Map<String, String>> listFile(String path) throws InterruptedException, IOException, URISyntaxException;

    //HDFS重命名文件
    public boolean renameFile(String oldName, String newName) throws InterruptedException, IOException, URISyntaxException;

    //删除HDFS文件
    public boolean deleteFile(String path) throws InterruptedException, IOException, URISyntaxException;

    //上传HDFS文件
    public void uploadFile(String path, String uploadpath) throws InterruptedException, IOException, URISyntaxException;

    //下载HDFS文件
    public ResponseEntity<InputStreamResource> downloadFile(String path, String fileName) throws InterruptedException, IOException, URISyntaxException;

    //压缩打包目录
    public ResponseEntity<byte[]> downloadDirectory(String path, String fileName) throws IOException;

    //HDFS文件复制
    public void copyFile(String sourcePath, String targetPath) throws InterruptedException, IOException, URISyntaxException;

    //打开HDFS上的文件并返回byte数组
    public byte[] openFileToBytes(String path) throws InterruptedException, IOException, URISyntaxException;

    //获取某个文件在HDFS集群的位置
    public BlockLocation[] getFileBlockLocations(String path) throws InterruptedException, IOException, URISyntaxException;
}

2.HdfsServiceImpl

package com.yc.service;

import com.yc.util.MyFileUtil;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.*;
import org.apache.zookeeper.common.IOUtils;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.io.InputStreamResource;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Service;
import org.springframework.util.StringUtils;
import org.springframework.web.multipart.MultipartFile;

import java.io.*;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.*;
import java.util.regex.Pattern;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;

/**
 * @program: Hadoop_yunpan
 * @description:
 * @author: 作者
 * @create: 2022-06-14 19:27
 */
@Service
public class HdfsServiceImpl implements HdfsService {
    @Value("${hdfs.path}")
    private String path;
    @Value("${hdfs.username}")
    private String username;
    @Value("${hdfs.nameservices}")
    private String nameservices;
    @Value("${hdfs.namenodes}")
    private String namenodes;
    @Value("${hdfs.node1}")
    private String node1;
    @Value("${hdfs.node2}")
    private String node2;
    @Value("${hdfs.provider}")
    private String provider;

    private final int bufferSize = 1024 * 1024 * 64;

    //获取HDFS配置信息
    public Configuration getConf() {
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", path);
        conf.set("dfs.nameservices", nameservices);
        conf.set("dfs.ha.namenodes.yc", namenodes);
        conf.set("dfs.namenode.rpc-address.yc.nn1", node1);
        conf.set("dfs.namenode.rpc-address.yc.nn2", node2);
        conf.set("dfs.client.failover.proxy.provider.yc", provider);
        return conf;
    }

    //获取HDFS文件系统对象
    public FileSystem getFileSystem() throws URISyntaxException, IOException, InterruptedException {
        //客户端操作hdfs需要一个用户身份，默认hdfs客户端API会从JVM中获取一个参数作为自己的用户身份
        //1.在Windows环境变量中添加一个变量 HADOOP_USER_NAME=root
        //2.创建fs对象时，通过参数传递
        FileSystem fs = FileSystem.get(new URI(path), getConf(), username);
        return fs;
    }

    //获取配置文件,以map形式返回
    @Override
    public Map<String, String> getConfInfoAsMap() throws InterruptedException, IOException, URISyntaxException {
        FileSystem fs = getFileSystem();
        Configuration conf = getConf();
        Iterator<Map.Entry<String, String>> ite = conf.iterator();

        Map<String, String> map = new HashMap<>();
        while (ite.hasNext()) {
            Map.Entry<String, String> entry = ite.next();
            map.put(entry.getKey(), entry.getValue());
        }
        return map;
    }

    //判断HDFS文件是否存在
    @Override
    public boolean existFile(String path) throws InterruptedException, IOException, URISyntaxException {
        if (StringUtils.isEmpty(path)) {
            return false;
        }
        FileSystem fs = getFileSystem();
        Path srcpath = new Path(path);
        return fs.exists(srcpath);
    }

    //新建HDFS目录
    @Override
    public boolean mkdir(String path) throws InterruptedException, IOException, URISyntaxException {
        if (StringUtils.isEmpty(path)) {
            return false;
        }
        if (existFile(path)) {//文件已经存在，返回true
            return true;
        }
        FileSystem fs = getFileSystem();
        Path srcpath = new Path(path);
        return fs.mkdirs(srcpath);
    }

    //读取HDFS目录信息
    @Override
    public List<Map<String, Object>> readPathInfo(String path) throws InterruptedException, IOException, URISyntaxException {
        if (StringUtils.isEmpty(path)) {
            return null;
        }
        if (!existFile(path)) {
            return null;
        }

        FileSystem fs = getFileSystem();
        Path srcpath = new Path(path);
        FileStatus[] statusList = fs.listStatus(srcpath);//读取出来的状态信息
        List<Map<String, Object>> res = new ArrayList<>();
        if (statusList != null && statusList.length > 0) {
            for (FileStatus status : statusList) {
                Map<String, Object> map = new HashMap<>();
                map.put("filePath", status.getPath());
                map.put("fileStatus", status.toString());
                res.add(map);
            }
            return res;
        }
        return null;
    }

    //创建HDFS文件
    @Override
    public void createFile(String path, MultipartFile file) throws IOException, URISyntaxException, InterruptedException {
        if (StringUtils.isEmpty(path) || file.getBytes() == null) {
            return;
        }
        FileSystem fs = getFileSystem();
        String fileName = file.getOriginalFilename();
        Path newpath = null;
        if ("/".equals(path)) {
            newpath = new Path(path + fileName);
        } else {
            newpath = new Path(path + "/" + fileName);
        }
        //打开一个输出流
        FSDataOutputStream outputStream = fs.create(newpath);
        outputStream.write(file.getBytes());
        outputStream.close();
        fs.close();
    }

    //读取HDFS文件内容
    @Override
    public String readFile(String path) throws InterruptedException, IOException, URISyntaxException {
        if (StringUtils.isEmpty(path)) {
            return null;
        }
        if (!existFile(path)) {
            return null;
        }
        FileSystem fs = getFileSystem();
        Path srcpath = new Path(path);
        //打开一个输入流
        FSDataInputStream inputStream = fs.open(srcpath);
        //防止中文乱码
        BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));

        String lineTxt = "";
        StringBuffer sb = new StringBuffer();
        while ((lineTxt = reader.readLine()) != null) {
            sb.append(lineTxt);
        }
        return sb.toString();
    }

    //读取HDFS文件列表
    @Override
    public List<Map<String, String>> listFile(String path) throws InterruptedException, IOException, URISyntaxException {
        if (StringUtils.isEmpty(path)) {
            return null;
        }
        if (!existFile(path)) {
            return null;
        }
        FileSystem fs = getFileSystem();
        Path srcpath = new Path(path);

        //递归找到所有文件
        RemoteIterator<LocatedFileStatus> filesList = fs.listFiles(srcpath, true);
        List<Map<String, String>> resList = new ArrayList<>();
        while (filesList.hasNext()) {
            LocatedFileStatus next = filesList.next();
            Map<String, String> map = new HashMap<>();
            map.put("fileName", next.getPath().getName());
            map.put("filePath", next.getPath().toString());
            resList.add(map);
        }
        fs.close();
        return resList;
    }

    //获取某个文件的信息
    @Override
    public Map<String, String> getFileInfo(String path) throws InterruptedException, IOException, URISyntaxException {
        if (StringUtils.isEmpty(path)) {
            return null;
        }
        if (!existFile(path)) {
            return null;
        }
        FileSystem fs = getFileSystem();
        Path srcpath = new Path(path);
        FileStatus fileStatus = fs.getFileStatus(srcpath);
        return fileStatusToMap(fileStatus);
    }

    //HDFS重命名文件
    @Override
    public boolean renameFile(String oldName, String newName) throws InterruptedException, IOException, URISyntaxException {
        if (StringUtils.isEmpty(oldName) || StringUtils.isEmpty(newName)) {
            return false;
        }
        FileSystem fs = getFileSystem();
        Path oldpath = new Path(oldName);
        Path newpath = new Path(newName);
        return fs.rename(oldpath, newpath);
    }

    //删除HDFS文件
    @Override
    public boolean deleteFile(String path) throws InterruptedException, IOException, URISyntaxException {
        if (StringUtils.isEmpty(path)) {
            return false;
        }
        if (!existFile(path)) {
            return false;
        }
        FileSystem fs = getFileSystem();
        Path srcpath = new Path(path);
        return fs.delete(srcpath, true);
    }

    //上传HDFS文件
    @Override
    public void uploadFile(String path, String uploadpath) throws InterruptedException, IOException, URISyntaxException {
        if (StringUtils.isEmpty(path) || StringUtils.isEmpty(uploadpath)) {
            return;
        }
        FileSystem fs = getFileSystem();
        Path oldpath = new Path(path);
        Path newpath = new Path(uploadpath);
        fs.copyFromLocalFile(false, oldpath, newpath);
        fs.close();
    }

    //下载HDFS文件
    @Override
    public ResponseEntity<InputStreamResource> downloadFile(String path, String fileName) throws InterruptedException, IOException, URISyntaxException {
        FileSystem fs = getFileSystem();
        Path srcpath = new Path(path);
        FSDataInputStream in = fs.open(srcpath);
        return MyFileUtil.downloadFile(in,fileName);
    }

    //将fileStatus转为map
    public Map<String, String> fileStatusToMap(FileStatus fileStatus) {
        Map<String, String> map = new HashMap<>();
        Path path = fileStatus.getPath();
        map.put("fileName", path.getName());//a
        map.put("filePath", path.toUri().toString());// hdfs://node1:9000/idea/a.txt
        map.put("relativePath", path.toUri().toString().substring(this.path.length()));// /idea/a.txt
        map.put("parenPath", path.getParent().toUri().toString().substring(this.path.length())); // /idea
        map.put("owner", fileStatus.getOwner());
        map.put("group", fileStatus.getGroup());
        map.put("isFile", fileStatus.isFile() + "");
        map.put("duplicates", fileStatus.getReplication() + "");
        map.put("size", MyFileUtil.formatFileSize(fileStatus.getLen()));
        map.put("rights", fileStatus.getPermission().toString());
        map.put("modifyTime", MyFileUtil.formatTime(fileStatus.getModificationTime()));
        return map;
    }

    public void search(Path srcpath, List<Map<String, String>> list, Pattern pattern) throws InterruptedException, IOException, URISyntaxException {
        FileSystem fs = getFileSystem();
        FileStatus[] fileStatuses = fs.listStatus(srcpath);
        if (fileStatuses != null && fileStatuses.length > 0) {
            for (FileStatus fileStatus : fileStatuses) {
                boolean res = fileStatus.isFile();
                if (!res) {//是目录则递归
                    search(fileStatus.getPath(), list, pattern);
                } else {
                    boolean bool = pattern.matcher(fileStatus.getPath().getName()).find();
                    if (bool) {
                        Map<String, String> map = this.fileStatusToMap(fileStatus);
                        list.add(map);
                    }
                }
            }
        }
    }

    //在整个磁盘搜索所有的某类型的文件
    @Override
    public List<Map<String, String>> listStatus(int type) throws InterruptedException, IOException, URISyntaxException {
        String path = "/";//查找全盘所有文件
        Path srcpath = new Path(path);//目标路径
        List<Map<String, String>> resList = new ArrayList<>();
        String reg = null;
        if (type == 1) {
            reg = ".+(.jpeg|.jpg|.png|.bmp|.gif)$";
        } else if (type == 2) {
            reg = ".+(.txt|.rtf|.doc|.docx|.xls|.xlsx|.html|.htm|.xml)$";
        } else if (type == 3) {
            reg = ".+(.mp4|.avi|.wmv)$";
        } else if (type == 4) {
            reg = ".+(.mp3|.wav)$";
        } else if (type == 5) {
            reg = "^\\S+\\.*$";
        }
        Pattern pattern = Pattern.compile(reg, Pattern.CASE_INSENSITIVE);
        search(srcpath, resList, pattern);
        return resList;
    }

    //在给定路径下搜索文件status，并统计在list
    @Override
    public List<Map<String, String>> listStatus(String path) throws InterruptedException, IOException, URISyntaxException {
        if (StringUtils.isEmpty(path)) {
            return null;
        }
        if (!existFile(path)) {
            return null;
        }
        FileSystem fs = getFileSystem();
        Path srcpath = new Path(path);

        FileStatus[] fileStatuses = fs.listStatus(srcpath);
        if (fileStatuses == null || fileStatuses.length <= 0) {
            return null;
        }

        List<Map<String, String>> resList = new ArrayList<>();
        for (FileStatus fileStatus : fileStatuses) {
            Map<String, String> map = fileStatusToMap(fileStatus);
            resList.add(map);
        }
        fs.close();
        return resList;
    }

    public void compress(String baseDir, ZipOutputStream zipOutputStream, FileSystem fs) {
        try {
            FileStatus[] fileStatuses = fs.listStatus(new Path(baseDir));
            String[] strs = baseDir.split("/");
            //lastName代表路径最后的单词
            String lastName = strs[strs.length - 1];
            for (int i = 0; i < fileStatuses.length; i++) {//循环baseDir路径下每一个文件
                String name = fileStatuses[i].getPath().toString();//文件path
                name = name.substring(name.indexOf("/" + lastName));//文件名
                if (fileStatuses[i].isFile()) {
                    Path path = fileStatuses[i].getPath();
                    FSDataInputStream in = fs.open(path);
                    zipOutputStream.putNextEntry(new ZipEntry(name.substring(1)));
                    IOUtils.copyBytes(in, zipOutputStream, this.bufferSize);
                    in.close();
                } else {
                    zipOutputStream.putNextEntry(new ZipEntry(fileStatuses[i].getPath().getName() + "/"));
                    compress(fileStatuses[i].getPath().toString(), zipOutputStream, fs);
                }
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    //压缩打包目录
    @Override
    public ResponseEntity<byte[]> downloadDirectory(String path, String fileName) throws IOException {
        //1.获取对象
        ByteArrayOutputStream out = null;
        try {
            FileSystem fs = this.getFileSystem();
            out = new ByteArrayOutputStream();
            ZipOutputStream zos = new ZipOutputStream(out);
            compress(path, zos, fs);
            zos.close();
        } catch (Exception e) {
            e.printStackTrace();
        }
        byte[] bs = out.toByteArray();
        out.close();
        return MyFileUtil.downloadFile(in, fileName);
    }

    //HDFS文件复制
    @Override
    public void copyFile(String sourcePath, String targetPath) throws InterruptedException, IOException, URISyntaxException {
        if (StringUtils.isEmpty(sourcePath) || StringUtils.isEmpty(targetPath)) {
            return;
        }
        FileSystem fs = getFileSystem();
        Path oldpath = new Path(sourcePath);
        Path newpath = new Path(targetPath);

        FSDataInputStream in = null;
        FSDataOutputStream out = null;
        try {
            in = fs.open(oldpath);
            out = fs.create(newpath);
            IOUtils.copyBytes(in, out, bufferSize, false);
        } finally {
            in.close();
            out.close();
            fs.close();
        }
    }

    //打开HDFS上的文件并返回byte数组
    @Override
    public byte[] openFileToBytes(String path) throws InterruptedException, IOException, URISyntaxException {
        if (StringUtils.isEmpty(path)) {
            return null;
        }
        if (!existFile(path)) {
            return null;
        }
        FileSystem fs = getFileSystem();
        //目标路径
        Path srcpath = new Path(path);
        byte[] res = null;
        try {
            FSDataInputStream in = fs.open(srcpath);
            InputStream iss = in.getWrappedStream();
            ByteArrayOutputStream baos = new ByteArrayOutputStream();
            byte[] bs = new byte[10 * 1024];
            int len = 0;
            while ((len = iss.read(bs, 0, bs.length)) != -1) {
                baos.write(bs, 0, len);
            }
            baos.flush();
            res = baos.toByteArray();
        } finally {
            fs.close();
        }
        return res;
    }

    //获取某个文件在HDFS集群的位置
    @Override
    public BlockLocation[] getFileBlockLocations(String path) throws InterruptedException, IOException, URISyntaxException {
        if (StringUtils.isEmpty(path)) {
            return null;
        }
        if (!existFile(path)) {
            return null;
        }
        FileSystem fs = getFileSystem();
        //目标路径
        Path srcpath = new Path(path);
        FileStatus fileStatus = fs.getFileStatus(srcpath);
        return fs.getFileBlockLocations(fileStatus, 0, fileStatus.getLen());
    }
}

3.MyFileUtil

import org.springframework.core.io.InputStreamResource;
import org.springframework.http.HttpHeaders;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;

import java.io.InputStream;
import java.text.DateFormat;
import java.text.DecimalFormat;
import java.text.SimpleDateFormat;
import java.util.Date;

/**
 * @program: Hadoop_yunpan
 * @description:
 * @author: 作者
 * @create: 2022-06-14 20:29
 */
public class MyFileUtil {
    public static String formatTime(long time) {
        SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd hh:mm:sss");
        Date date = new Date(time);
        return simpleDateFormat.format(date);
    }

    public static String formatFileSize(long fileS) {
        DecimalFormat df = new DecimalFormat("#.00");
        String fileSizeString = "";
        String wrongSize = "0B";
        if (fileS == 0) {
            return wrongSize;
        }
        if (fileS < 1024) {
            fileSizeString = df.format((double) fileS) + "B";
        } else if (fileS < 1048576) {
            fileSizeString = df.format((double) fileS / 1024) + "KB";
        } else if (fileS < 1073741824) {
            fileSizeString = df.format((double) fileS / 1048576) + "MB";
        } else {
            fileSizeString = df.format((double) fileS / 1073741824) + "GB";
        }
        return fileSizeString;
    }

    public static ResponseEntity<InputStreamResource> downloadFile(InputStream in, String fileName) {
        try {
            byte[] bytes = new byte[in.available()];
            //http响应协议
            HttpHeaders httpHeaders = new HttpHeaders();
            httpHeaders.add("Cache-Control", "no-cache, no-store, must-revalidate");
            //下载文件的方式
            httpHeaders.add("Content-Disposition", String.format("attachment; filename=\"%s\"", fileName));
            httpHeaders.add("Pragma", "no-cache");
            httpHeaders.add("Expires", "0");
            httpHeaders.add("Content-Language", "UTF-8");
            //文件内容以流的形式输出
            return ResponseEntity.ok().headers(httpHeaders).contentLength(bytes.length)
                    .contentType(MediaType.parseMediaType("application/octet-stream")).body(new InputStreamResource(in));
        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }

    public static ResponseEntity<byte[]> downloadDirectory(byte[] bytes, String fileName) {
        try {
            HttpHeaders httpHeaders = new HttpHeaders();
            httpHeaders.add("Cache-Control", "no-cache, no-store, must-revalidate");
            //下载文件的方式
            httpHeaders.add("Content-Disposition", String.format("attachment; filename=\"%s\"", fileName));
            httpHeaders.add("Pragma", "no-cache");
            httpHeaders.add("Expires", "0");
            httpHeaders.add("Content-Language", "UTF-8");
            //文件内容以流的形式输出
            return ResponseEntity.ok().headers(httpHeaders).contentLength(bytes.length)
                    .contentType(MediaType.parseMediaType("application/octet-stream")).body(bytes);
        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }

    public static String getFileName() {
        Date d = new Date();
        DateFormat format = new SimpleDateFormat("yyyyMMddHHmmss");
        return format.format(d) + ".zip";
    }
}
ChenJieYaYa
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Hadoop——HDFS文件系统的Java API操作（上传、下载、查看、删除、创建文件）详细教学
2401_84415708的博客
04-18 952
分享一套我整理的面试干货，这份文档结合了我多年的面试官经验，站在面试官的角度来告诉你，面试官提的那些问题他最想听到你给他的回答是什么，分享出来帮助那些对前途感到迷茫的朋友。
java 读取hadoop_使用HDFS客户端java api读取hadoop集群上的信息
weixin_39795284的博客
02-16 517
本文介绍使用hdfs java api的配置方法。1、先解决依赖，pomorg.apache.hadoophadoop-client2.7.2provided2、配置文件，存放hdfs集群配置信息，基本都是来源于core-site.xml和hdfs-site.xml，可以根据hdfs集群client端配置文件里的信息进行填写#============== hadoop ==============...
JAVA操作HDFS集群
m0_56369204的博客
06-25 2040
本文介绍了使用Java操作HDFS集群的方法，具体的操作包括创建目录、删除文件、移动与重命名文件、查询文件信息以及文件的上传和下载等。通过示例代码和截图展示了每个操作的实际效果，帮助读者理解和使用HDFS安全模式和相关操作。
HDFS之Java客户端操作
CSDN2022博客之星Top39；华为云博主7&，CSDN、稀土掘金、微信公众号、阿里云开发者社区、腾讯云社区、博客园、知乎、51CTO同名博主WHYBIGDATA；
06-27 2052
HDFS之Java客户端操作
HDFS Java API 基本操作实验
最新发布
bjyxszd_YYDS的博客
12-08 406
本实验已经搭建好了Hadoop HA的完全分布式集群。
Java操作HDFS
HelloWorld
01-11 2362
使用Java操作HDFS
hdfs-java-api
10-17
hdfs-java-api
HDFS-Java-API.md
08-07
HDFS-Java-API，基础篇
java-Hdfs体系结构与基本概念
06-26
Java-HDFS 体系结构与基本概念 HDFS（Hadoop Distributed File System）是一种分布式文件系统，主要用于存储和管理大规模数据。HDFS 的设计初衷是为了满足高性能、高可靠性和高可扩展性的需求。 HDFS 体系结构 ...
webhdfs-java-client
04-30
webhdfs-java-client 带有Kerberos身份验证的Hadoop WebHDFS REST API的Java客户端代码。
HdfsClient.java
07-23
一个hdfsclient的操作类，如果有报错，在idea开发工具中File -> Project Struct... -> Libraies -> 点击绿色的加号 -> Java -> 找到 IDEA 安装路径下的 Lib 中的junit-4.12 -> 确定完就行了，点击OK就出去了
JAVA操作HDFS案例
hzqing的博客
07-03 1万+
将本地文件上传到hdfs服务器: /** * 上传文件到hdfs上 */ @Test public void upload() throws IOException { Configuration conf = new Configuration(); conf.set("fs.defaultFS","hdfs://hz
Hadoop权威指南---HDFS简介和java API操作
l1394049664的博客
09-16 954
目录 1、HDFS的设计 2、 HDFS的概念  2.1 数据块  2.2 namenode和datanode 概述 2.3 块缓存  ​2.4 联邦HDFS  2.5 HDFS的高可用性  3、 命令行接口 4、 Hadoop文件系统分类 5、Java接口API 5.1、增：创建文件create和write、文件夹mkdirs 5.2、删：删除文件夹和文件delete ...
java操作hdfs文件、文件夹
热门推荐
daxiang的专栏
10-01 2万+
代码参考网上其他博文，在其基础上作部分优化。以下代码在win7 + jdk1.7 + eclipse3.7 + centos 7 + hadoop2.6条件下运行通过。 java工程需添加hadoop根目录下/share/hadoop/下所有jar包到编译路径中。 工具类HdfsUtils.java，及测试用例代码如下： HdfsUtils.java package com.xy6
大数据——熟悉常用的HDFS操作（命令行和JAVA实现）
万万VV
03-02 1万+
实验目的 理解 HDFS在 Hadoop体系结构中的角色； 熟练使用 HDFS操作常用的 Shell 命令； 熟悉 HDFS操作常用的 JavaAPI。 实验平台 操作系统：Linux Hadoop 版本：2.6.0 或以上版本JDK 版本：1.6 或以上版本Java IDE：Eclipse 实验内容和要求 编程实现以下指定功能，并利用 Hadoop提供的 Shel...
HDFS命令和java API
Dear__JIA的博客
05-26 302
HDFS命令和java API 实验步骤 搭建Hadoop伪分布式模式、或者完全分布式模式 Windows里安装配置JDK+Eclipse+Maven 在Eclipse里新建Maven Project，新建包，新建class 编程上述4个Java程序 虚拟机的namenode主机上，启动Hadoop 在eclipse里运行上述4个Java程序 使用hdfs的shell命令查看运行结果 使用web...
大数据学习第三课之hdfs的java客户端IO流和简单Util的文件操作（以idea展示，eclipse为辅）
qq_41915149的博客
09-25 306
导包（在java中的jar包）流程如此下： 导入这两个包,这里就演示的包，common相同。 先导入这第三个还有它lib第三方依赖包的全部： 点击打开就导入了，下面也是。 这里就是导入成功了（hdfs和hdfsbao包一样的，举个例子，我这里是hdfs，名字随便取）： ...
hdfs Java使用_java使用hdfs
weixin_33998152的博客
02-12 224
1、mavenorg.apache.hadoophadoop-client2.7.3org.slf4jslf4j-log4j12org.slf4jlog4j-over-slf4j2、resource放置core-site.xml/hdfs-site.xml3、工具类packagecom.test;importorg.apache.commons.lang.StringUtils;import...
JAVA 进行hdfs 文件操作
CARROTHWQ的专栏
05-07 5516
1 文件上传2 文件下载3 新建文件夹4查看文件  “listFiles“列出的是hdfs上所有文件的路径，不包括文件夹。根据你的设置，支持递归查找。   ”listStatus“列出的是所有的文件和文件夹，不支持递归查找。如许递归，需要自己实现。5 删除文件fs.delete(hdfsPath， true); //设置成true 表示如果目标路径是一个文件夹，则递归删除。PS：HDFS 路径的书...
hdfs的javaAPI操作
04-24
HDFS的Java API操作可以通过创建Hadoop Configuration对象来实现。可以使用以下代码来创建Configuration对象： ```java Configuration config = new Configuration(); config.set("fs.defaultFS", "hdfs://...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值