Hadoop MapReduce Job 提交的多种方案

提交hadoop作业时我们遇到了许多的问题，在网上也查过许多的文章，有许多对hadoop提交作业原理进行分析的文章，却总看不到对具体操作过程讲解的文章，导致我们在eclipse提交的作业总是在eclipse虚拟的云环境中运行。慢慢摸索中，一个一个的作业提交方法被我们发现。

方案：

1. 用命令行方式提交。

2. 在Eclipse中提交作业, 并且读取的是本地文件系统的资源，Job的提交器为localcommiter。

在Linux的eclipse中直接启动Runner类的main方法，这种方式可以使job运行在本地，也可以运行在yarn集群。

究竟运行在本地还是在集群，取决于一个配置参数： mapreduce.framework.name == yarn (local)
如果确实需要在eclipse中提交到yarn执行，必须做好以下两个设置：
a、将mr工程打成jar包(wc.jar)，放在工程目录下
b、在工程的main方法中，加入一个配置参数   conf.set("mapreduce.job.jar","wc.jar");

方案一的讲解：

方案二的讲解：

Mapper类如下：

package com.npf.hadoop;

import java.io.IOException;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.util.StringUtils;

public class WordCountMapper extends Mapper<LongWritable, Text, Text, LongWritable> {

        @Override
        protected void setup(Mapper<LongWritable, Text, Text, LongWritable>.Context context)throws IOException, InterruptedException {
                System.out.println("WordCountMapper.setup()");
        }

        @Override
        protected void map(LongWritable key, Text value,Context context) throws IOException, InterruptedException {
                String[] words = StringUtils.split(value.toString(),' ');
                for (String word : words) {
                        context.write(new Text(word), new LongWritable(1L));
                }
        }

        @Override
        protected void cleanup(Mapper<LongWritable, Text, Text, LongWritable>.Context context)throws IOException, InterruptedException {
                System.out.println("WordCountMapper.cleanup()");
        }

}

Reducer类如下：

package com.npf.hadoop;

import java.io.IOException;
import java.util.Iterator;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class WordCountReducer extends Reducer<Text, LongWritable, Text, LongWritable>{

        @Override
        protected void setup(Reducer<Text, LongWritable, Text, LongWritable>.Context context)throws IOException, InterruptedException {
                System.out.println("WordCountReducer.setup()");
        }

        @Override
        protected void reduce(Text word, Iterable<LongWritable> counts,Context context)throws IOException, InterruptedException {
                Iterator<LongWritable> iterator = counts.iterator();
                long count = 0L;
                while (iterator.hasNext()) {
                        LongWritable element = iterator.next();
                        count = count + element.get();
                }
                context.write(word, new LongWritable(count));
        }

        @Override
        protected void cleanup(Reducer<Text, LongWritable, Text, LongWritable>.Context context)throws IOException, InterruptedException {
                System.out.println("WordCountReducer.cleanup()");
        }
}

Runner类如下：

/wordcount/srcdata ：这是linux下根目录下wordcount目录下的srcdata目录。

/wordcount/outputdata：  这是linux下根目录下wordcount目录下的outputdata目录。

package com.npf.hadoop;

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

/**
 *
 * @author root
 *
 */
public class WordCountRunner {

        public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {

                Configuration conf = new Configuration();

                Job job = Job.getInstance(conf);
                job.setJarByClass(WordCountRunner.class);

                //mappper
                job.setMapperClass(WordCountMapper.class);
                job.setMapOutputKeyClass(Text.class);
                job.setOutputValueClass(LongWritable.class);

                //reducer
                job.setReducerClass(WordCountReducer.class);
                job.setOutputKeyClass(Text.class);
                job.setOutputValueClass(LongWritable.class);


                FileInputFormat.setInputPaths(job, "/wordcount/srcdata");
                FileOutputFormat.setOutputPath(job, new Path("/wordcount/outputdata"));

                job.waitForCompletion(true);
        }

}

运行会得到如下结果：

方案三的讲解：