import ......
/**单词计数Mapper类:**/
public class TokenizerMapper extends Mapper<LongWritable,Text,Text,IntWritable>{
//Mapper类的四个泛型分别表示map函数输入键值对的键的类、值的类、输出键值对的键的类、值的类
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key,Text value, Context context) throws IoException,
InterrupterException{
//key:键,代表行号。value:值,代表该行的内容
StringTokenizer itr = new StringTokenizer(value.toString());
while(itr.hasMoreTokens()){
word.set(itr.nextToken());//nextToken()将每行文本拆分成单个单词
context.write(word,one);//write()输出中间结果
}
}
}
/**
单词计数Reducer类:
Reducer接收Mapper输出的中间结果并执行reduce函数,reduce接收到的参数形如:<key,List<value>>,
这是因为map函数将key值相同的所有value都发送给reduce函数。在reduce函数中,完成对相同key值(同一
单词)的计数并将最后结果输出。Reducer类的泛型带包了reduce函数输入键值对的键的类、值的类,输出键
值对的键的类、值的类**/
public class IntSumReducer extends Reducer<Text,IntWritableText,Text,IntWritable>{
private IntWritable result= new IntWritable();
public void reduce(Text key,Interable<IntWritable> values, Context context) throws
IoException, InterrupterException{
Int sum = 0;
for(IntWritable val : values){
sum += val.get();
}
result.set(sum);
context.write(key,result);
}
}
/**
main函数:
对作业进行相关配置以及向Hadoop提交作业**/
public class WordCount{
public static void main(String[] args) throws IoException, ClassNotFoundException,InterruptException{
Configuration conf = new Configuration();
//Configuration类包含了对Hadoop的配置,代表了作业的配置,该类会加载mapred-site.xml
//hdfs-site.xml、core-site.xml、yarn-site.xml
if(args.length != 2){
System.err.println("Usage:wordcount <in> <out>");
System.exit(2);
}
Job job = new Job(conf,"word count");
job.setJarByClass(WordCount.class); //指定main函数所在类
job.setMapperClass(TokenizerMapper.class); //指定Mapper类
job.setReducerClass(IntSumReducer.class); //指定Reducer类
job.setOutputKeyClass(Text.class); //设置reduce函数输出key的类
job.setOutputValueClass(IntWritable.class); //设置reduce函数输出value的类
FileInputFormat.addInputPath(job,new Path(args[0])); //指定输入路径
FileOutputFormat.addOutputPath(job,new Path(args[1])); //指定输出路径
System.exit(job.waitForCompletion(true) ? 0 : 1); //提交任务
}
}
运行程序:
将代码打包成jar文件,上传至集群上任一节点,执行命令:
hadoop jar /home/hadoop/wordcount.jar com.hellohadoop.WordCount /user/test/wordcountinput /user/test/wordcountoutput
/user/test/wordcountinput为HDFS存放文本的目录,如果指定一个目录wei为MapReduce输入的路径,则MapReduce会将该路径xia下的所有文件作为sh输入,如果指定一个文件,则MapReduce只会将该文件作为输入。/user/test/wordcountoutput为作业sh输出路径,该路径在作业运行之前必须不存在,否则会报错。