记录window平台eclipse远程连接hadoop集群(linux)并执行mr任务(不需要安装eclipse插件及本地hadoop环境)

1 拷贝集群hosts主机目录及ip至window系统hosts文件中;

2 创建maven工程,pom文件添加:

<dependencies>

    <dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.7.1</version>

</dependency>

</dependencies>

3 下载远程集群core-site.xml,mapred-site.xml客户端配置文件,配置文件放到工程资源目录下;

3 修改core-site.xml文件中fs.defaultFS属性的值namenode地址,例如hdfs://host213.ehl.com;注释掉io.compression.codec.lzo.class及io.compression.codecs这两个属性;

4 创建hadoop-remote.xml文件并放到资源目录,文件内容如下:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>host211.ehl.com</value><!--修改RM节点名称-->
    </property>
    <property>
        <name>mapreduce.app-submission.cross-platform</name>
        <value>true</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>host212.ehl.com:10020</value><!--修改jobhistory服务地址-->
    </property>

</configuration>

5 编写map及reduce类,main方法示例代码如下:

public static void main(String[] args) throws Exception {
//连接hdfs集群,url为namenode地址

FileSystem fileSystem = FileSystem.get(new URI("hdfs://host213.ehl.com"), new Configuration(), "hdfs");

Path out = new Path("/sort/output");

                //删除输出目录

fileSystem.deleteOnExit(out);

                Path localJarPath = new Path("file:///D:/sort.jar");

Path jarPath = new Path("/sort/sort.jar");

                //将本工程打的jar包上传至hdfs集群

fileSystem.copyFromLocalFile(false, true, new Path[]{localJarPath}, jarPath);
fileSystem.close();

Configuration configuration = new Configuration();

                //添加相关配置文件

configuration.addResource("core-site.xml");
configuration.addResource("mapred-site.xml");

configuration.addResource("hadoop-remote.xml");

                //因本hadoop集群通过hdp平台搭建,所以配置文件中含有hdp.version属性

configuration.set("hdp.version", "2.3.4.0-3485");

Job job = Job.getInstance(configuration,"remoteSub");

                //将上传至集群的jar包添加至job的classpath

job.addFileToClassPath(jarPath);

FileInputFormat.addInputPath(job, new Path("/sort/input"));
FileOutputFormat.setOutputPath(job, out);

job.setJarByClass(SortMain.class);
job.setMapperClass(SortMapper.class);
job.setReducerClass(SortReducer.class);

job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(NullWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(NullWritable.class);


job.setNumReduceTasks(4);
job.setPartitionerClass(SortPartition.class);

System.exit(job.waitForCompletion(true) ? 0 : 1);

}

6 将本工程通过eclipse打成jar包,放到main方法中localJarPath指定的地址上;

7 main方法中右击选择Run As -> Run Configurations...

切换至Environment标签,添加环境变量HADOOP_USER_NAME,值为hadoop集群用户名,本例为hdfs;

8 点击Run,通过RM的web地址查看任务是否提交,如下为本例eclipse控制台部分相关输出:

运行成功了!!!!!!