hadoop：hdfs：HdfsUtil

最新推荐文章于 2023-04-10 15:05:45 发布

花和尚也有春天

最新推荐文章于 2023-04-10 15:05:45 发布

阅读量982

点赞数

分类专栏： hadoop 文章标签： hdfs

本文链接：https://blog.csdn.net/weixin_38750084/article/details/107630955

版权

遇到了几种报错：

hadoop运行报错Wrong FS: hdfs:/, expected: file:///

Class org.apache.hadoop.hdfs.DistributedFileSystem not found


java.io.IOException: No FileSystem for scheme: hdfs

NoClassDefFoundError: org/apache/hadoop/crypto/key/KeyProviderTokenIssuer

代码：

application.properties

#hdfs参数
fs.defaultFS=nameservice1
dfs.blocksize=134217728
dfs.replication=3
HADOOP_USER_NAME=root

import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.net.URISyntaxException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.BlockLocation;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FileUtil;
import org.apache.hadoop.fs.Path;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;

@Component
public class HdfsUtil {

    @Value("${fs.defaultFS}")
    private String defaultFS;
    @Value("${dfs.blocksize}")
    private Integer blocksize;
    @Value("${dfs.replication}")
    private Integer replication;
    @Value("${HADOOP_USER_NAME}")
    private String HADOOP_USER_NAME;

    public FileSystem getFileSystem() throws IOException, URISyntaxException{
        Configuration conf = new Configuration();
        conf.set("fs.de