Hadoop基本的Map/Reduce

HelloHadoop.class

 

Mapper

static public class HelloMapper extends Mapper<LongWritable, Text, LongWritable, Text> {
    public void map(LongWritable key, Text value, Context context) {
        context.write((LongWritable)key, (Text)value);
    }
}

<LongWritable, Text, LongWritable, Text>中的第一个LongWritable和第一个Text为Mapper的输入数据类型，第二个LongWritable和Text为Mapper的输出数据类型。

 

Reducer

static public class HelloReducer extends Reducer<LongWritable, Text, L ongWritable, Text> {
    public void reduce(LongWritable key, Iterable<Text> values, Context context) {
        Text val = new Text();
        for (Text str : values) {
            val.set(str.toString());
        }
        context.write(key, val);
    }
}

<LongWritable, Text, LongWritable, Text>中的第一个LongWritable和第一个Text为Reducer的输入数据类型，第二个LongWritable和Text为Reducer的输出数据类型。

 

Main

public static void main(String[] args) {
    Configuration conf = new Configuration();
    Job job = new Job(conf, "Hadoop Hello World");
    job.setJarByClass(HelloHadoop.class);
    FileInputFormat.setInputPaths(job, "input");
    FileOutputFormat.setOutputPath(job, new Path("output"));
    job.setMapperClass(HelloMapper.class);
    job.setReducerClass(HelloReducer.class);
    job.waitForCompletion(true);
}