java用数组实现单词计数,使用MapReduce实现单词计数

描述

假定有一组文件，以空格为分隔符存放着数据。如

Hello World

Hello my love

Hello World , i love you

计算一组文件中字符所出现的次数。

实现思路

Map函数：

以字符作为key值，value为1，生成键值对。

Reduce函数：

获取Map输出的键值对，将各个键中的值相加，输出。

逻辑图：

?

代码实现：

class="java" name="code">import java.io.IOException;

import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

public static class TokenizerMapper extends

Mapper {

private final static IntWritable one = new IntWritable(1);

private Text word = new Text();

public void map(Object key, Text value, Context context)

throws IOException, InterruptedException {

StringTokenizer itr = new StringTokenizer(value.toString());

while (itr.hasMoreTokens()) {

word.set(itr.nextToken());

context.write(word, one);

}

}

}

public static class IntSumReducer extends

Reducer {

private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable values,

Context context) throws IOException, InterruptedException {

int sum = 0;

for (IntWritable val : values) {

sum += val.get();

}

result.set(sum);

context.write(key, result);

}

}

public static void main(String[] args) throws Exception {

Configuration conf = new Configuration();

Job job = new Job(conf, "word count11");

job.setJarByClass(WordCount.class);

job.setMapperClass(TokenizerMapper.class);

job.setCombinerClass(IntSumReducer.class);

job.setReducerClass(IntSumReducer.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(IntWritable.class);

// FileInputFormat.addInputPath(job, new Path(otherArgs[0]));

// FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));

FileInputFormat.addInputPath(job, new Path("input"));

FileOutputFormat.setOutputPath(job, new Path("output"));

System.exit(job.waitForCompletion(true) ? 0 : 1);

}

}

?

总结：

这个例子是Hadoop官网给出的例子，比较简单，是MapReduce的HelloWorld小程序。

?

大小: 37.7 KB

查看图片附件