运行WordCount—程序

一、实验题目

编写一个MapReduce程序WordCount

二、实验目的

该程序能够计算单词数以及每个单词出现的频率。

三、任务分析

计数很简单，mapper中将文档拆成n个单词，记为<word,1>的形式，reducer的时候将键值相同的相加就可以统计出每个单词的频率。

所有键值相加，就可以得出总数。

四、实验步骤

1.将Hadoop的安全模式关闭，命令为：

hadoop dfsadmin -safemode leave

2.将待处理文件导入到hdfs文件中，命令为：

bin/hadoop dfs -copyFromLocal 源文件位置 hdfs:/

3.启动eclipse，建立Java project，导入相关jar文件，开始编码。

mapper部分代码：

package com.bigdata;

import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

/**
 * 
 * @author training Class: WordCountMapper
 **/
public class WordCountMapper extends
		Mapper<LongWritable, Text, Text, IntWritable> {
	/**
	 * Optimization: Instead of creating the variables in the
	 */
	@Override
	public void map(LongWritable inputKey, Text inputVal, Context context)
			throws IOException, InterruptedException {
		String line = inputVal.toString();
		String[] splits = line.trim().split("\\W+");
		for (String outputKey : splits) {
			context.write(new Text(outputKey), new IntWritable(1));
		}
	}
}

首先用trim方法去除头尾的空格，然后使用\\W+按英文字符串分割。

接着将splits中的元素取出，输出格式为<word,1>

然后是reducer函数：

package com.bigdata;

import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class WordCountReducer extends
		Reducer<Text, IntWritable, Text, IntWritable> {
	@Override
	public void reduce(Text key, Iterable<IntWritable> listOfValues,
			Context context) throws IOException, InterruptedException {
		int sum = 0;
		for (IntWritable val : listOfValues) {
			sum = sum + val.get();
		}
		context.write(key, new IntWritable(sum));
	}
}

将所有key值相同的value相加，并输出<word,sum>

wordcount程序就写完了。

最后是driver函数：

package com.bigdata;

import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public class WordCountDriver extends Configured implements Tool {
	public static void main(String[] args) throws Exception {
		ToolRunner.run(new WordCountDriver(), args);
	}

	@Override
	public int run(String[] args) throws Exception {
		Job job = new Job(getConf(), "Basic Word Count Job");
		job.setJarByClass(WordCountDriver.class);
		// Map and Reduce
		job.setMapperClass(WordCountMapper.class);
		job.setReducerClass(WordCountReducer.class);
		job.setNumReduceTasks(1);
		job.setInputFormatClass(TextInputFormat.class);
		// the map output
		job.setMapOutputKeyClass(Text.class);
		job.setMapOutputValueClass(IntWritable.class);
		// the reduce output
		job.setOutputKeyClass(Text.class);
		job.setOutputValueClass(IntWritable.class);
		FileInputFormat.addInputPath(job, new Path(args[0]));
		FileOutputFormat.setOutputPath(job, new Path(args[1]));
		job.waitForCompletion(true);
		return 0;
	}
}

4.编码完毕后export成jar文件

5.执行mapreduce

6.查看结果