mappreduce词汇统计（小白教程）

mappreduce词汇统计（小白教程）

wordcount原理：

1. mapper阶段：
   从数据源读取一行数据传递给mapper函数的value
   处理数据并将处理结果输出到reduce中
2. reduce阶段：
   遍历values累加技术结果，并将数据输出
   
   一·复制jar包
   
   然后一直按住shift点击lib里面第一个jar包再点击最后一个jar包就可以全选了。再右键add to build path。
   二·代码示例
   mapper.Java

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class MRmapper extends Mapper<LongWritable,Text,Text,IntWritable>{

	@Override
	protected void map(LongWritable key,Text value,Mapper<LongWritable,Text,Text,IntWritable>.Context context) 
		throws IOException,InterruptedException{
			String formatValue=value.toString();
			String[] spiltWord=formatValue.split(" ");
			for(String wordString :spiltWord) {
				context.write(new Text(wordString),new IntWritable(1));
			}
		}
}

reduce.java

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class MRreduce extends Reducer<Text,IntWritable,Text,IntWritable> {
	
	protected void reduce(Text key3,Iterable<IntWritable> value3,
	 Reducer<Text,IntWritable,Text,IntWritable>.Context context) throws IOException,InterruptedException{
		
		int sum=0;
		for(IntWritable intWriable : value3) {
			sum += intWriable.get();
		}
		
		context.write(new Text(key3), new IntWritable(sum));
	}

}

job.java

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;


public class MRjobs {
	public static void main(String[] args)throws IOException,ClassNotFoundException,InterruptedException{
		
		Configuration configuration = new Configuration();
		Job job =Job.getInstance(configuration);
		job.setJarByClass(MRjobs.class);
		
		job.setMapperClass(MRmapper.class);
		job.setMapOutputKeyClass(Text.class);
		job.setMapOutputValueClass(IntWritable.class);
		
		job.setReducerClass(MRreduce.class);
		job.setOutputKeyClass(IntWritable.class);
		job.setOutputValueClass(IntWritable.class);
		
		FileInputFormat.setInputPaths(job, new Path(args[0]));
		FileOutputFormat.setOutputPath(job ,new Path(args[1]));
		
		job.waitForCompletion(true);
	}
}

三·expore出jar包

选择主类，出错会出现ClassNotFoundException

四·利用文件传输工具传入jar包

五·开始测试



刚入门，可能理解存在错误，希望路过大佬帮忙指点一下。