hadoop HA场景下 java客户端远程访问hdfs配置

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/wo198711203217/article/details/80528860

当hadoop namenode是HA集群时,客户端远程访问hdfs有两种实现方法:
方法1:将所有关于namenode的参数写入Configuration对象中。
代码:

package com.lx;

import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

public class Demo {

    private static void uploadToHdfs() throws Exception {
        //本地文件地址
        String localSrc = "d:/PowerDesigner15_Evaluation.rar";
        //存放在hdfs的目的地址
        String dest = "/user/PowerDesigner15_Evaluation.rar";
        InputStream in = new BufferedInputStream(new FileInputStream(localSrc));
        //得到配置对象
        Configuration conf = new Configuration();

        conf.set("fs.defaultFS", "hdfs://ns1");
        conf.set("dfs.nameservices", "ns1");
        conf.set("dfs.ha.namenodes.ns1", "nn1,nn2");
        conf.set("dfs.namenode.rpc-address.ns1.nn1", "hadoop1:9000");
        conf.set("dfs.namenode.rpc-address.ns1.nn2", "hadoop2:9000");
        conf.set("dfs.client.failover.proxy.provider.ns1", "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");



        //文件系统
        FileSystem fs = FileSystem.get(new URI("hdfs://ns1"), conf, "hadoop");
        //输出流
        OutputStream out = fs.create(new Path(dest));
        //连接两个流,形成通道,使输入流向输出流传输数据
        IOUtils.copyBytes(in, out, 4096, true);
    }

    public static void main(String[] args) throws Exception{
        // TODO Auto-generated method stub
        uploadToHdfs();
    }

}

方法2:将hadoop集群配置文件core-site.xml和hdfs-site.xml文件拷贝到项目的src目录下。
这里写图片描述
代码:

package com.lx;

import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

public class Demo2 {

    public static void main(String[] args) throws Exception{
        // TODO Auto-generated method stub
        String src="d:/RoseHA.rar";
        String dst="/user/RoseHA.rar";

        InputStream in=new BufferedInputStream(new FileInputStream(src));

        Configuration conf=new Configuration();

        FileSystem fs=FileSystem.get(new URI("hdfs://ns1"), conf, "hadoop");

        OutputStream out=fs.create(new Path(dst));

        IOUtils.copyBytes(in, out, 4096, true);
    }

}
阅读更多

没有更多推荐了,返回首页