学习篇-Hadoop-MapReduce-词频统计

一、Hadoop-MapReduce-词频统计-Mapper

简要说明：Maps input key/value pairs to a set of intermediate key/value pairs.

释义：Mapper就是将输入的键/值对转换到一组中间键/值对

• Mapper中传入的泛型含义

  • KEYIN： Map任务读数据的key类型，offset，是每行数据起始位置的偏移量，LongWritable不再是Java中的Long
  • VALUEIN：Map任务读数据的value类型，其实就是一行行的字符串，Text不再是Java中的String
  • KEYOUT：map方法自定义实现输出的key的类型，例如：对于词频统计就是Text【注意不再是String】
  • VALUEOUT：map方法自定义实现输出的value的类型，例如：对于词频统计就是IntWritable【注意不能是Integer】

• 自定义词频Mapper：WordCountMapper

  /**
   * @ClassName WordCountMapper
   * @Description 词频统计mapper
   * @Author eastern
   * @Date 2020/4/29 下午2:15
   * @Version 1.0
   *
   * 对于词频统计：（word,1）KEYOUT就是String VALUEOUT就是Integer
   * LongWritable对应Long
   * Text对应String
   * IntWritable对应Integer
   **/
  public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
  	@Override
  	protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
  		// 通过分隔符分割单词
  		String[] words = value.toString().split("\t");
    		// 遍历单词  
  		for (String word: words) {
       		 // 写入到context中
  			context.write(new Text(word), new IntWritable(1));
  		}
  	}
  }
  

二、Hadoop-MapReduce-词频统计-Reducer

简要说明：Reduces a set of intermediate values which share a key to a smaller set of values.

释义：Reduce 将一组中间值转化成共享一个key，value合并成一组较小的值

比如：

 # 从文件中读取的单词
 (hello,1) (world,1)
 (hello,1) (world,1)
 (hello,1) (world,1)
 (welcome,1)
 # map的输出到reduce端，是按照相同的key分发到一个reduce上去执行
 reduce1:	(hello,1) (hello,1) (hello,1) ===> (hello, <1,1,1>)
 reduce2:	(world,1) (world,1) (world,1) ===> (world, <1,1,1>)
 reduce3:	(welcome,1) ===> (welcome, <1>)

• Reducer中传入的泛型含义

  • KEYIN： Map输出的Key的类型
  • VALUEIN：Map输出的Value的类型
  • KEYOUT：reduce方法自定义实现输出的key的类型，例如：对于词频统计就是Text【注意不再是String】
  • VALUEOUT：reduce方法自定义实现输出的value的类型，例如：对于词频统计就是IntWritable【注意不能是Integer】

• 自定义词频Reducer：WordCountReducer

  /**
   * @ClassName WordCountReducer
   * @Description 词频统计Reducer
   * @Author eastern
   * @Date 2020/4/29 下午3:18
   * @Version 1.0
   **/
  public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
  
  	@Override
  	protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException,
  			InterruptedException {
  		int count = 0;
  		Iterator<IntWritable> iterator = values.iterator();
  		// <1,1,1>
  		while (iterator.hasNext()) {
  			IntWritable value = iterator.next();
  			count += value.get();
  		}
  		context.write(key, new IntWritable(count));
  	}
  }
  

三、Hadoop-MapReduce-词频统计-Driver

/**
 * @ClassName WordCountApp
 * @Description Driver:配置Mapper Reducer的相关属性 提交到本地运行
 * @Author eastern
 * @Date 2020/4/29 下午4:35
 * @Version 1.0
 **/
public class WordCountApp {

	public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {

		System.setProperty("HADOOP_USER_NAME", "root");
		// 设置HDFS的Configuration
		Configuration configuration = new Configuration();
		configuration.set("fs.defaultFS", "hdfs://139.129.240.xxx:8020");
		configuration.set("dfs.client.use.datanode.hostname", "true");
		configuration.set("dfs.replication", "1");


		// 创建一个job
		Job job = Job.getInstance(configuration);

		// 设置Job对应的参数：主类
		job.setJarByClass(WordCountApp.class);

		// 设置Job对应的参数：设置自定义的Mapper和Reducer处理类
		job.setMapperClass(WordCountMapper.class);
		job.setReducerClass(WordCountReducer.class);

		// 设置Job对应的参数：Mapper输出key和value的类型
		job.setMapOutputKeyClass(Text.class);
		job.setMapOutputValueClass(IntWritable.class);

		// 设置Job对应的参数：Reducer输出key和value的类型
		job.setOutputKeyClass(Text.class);
		job.setOutputValueClass(IntWritable.class);

		// 设置Job对应的参数：设置输入/输出路径
		FileInputFormat.setInputPaths(job, new Path("/hdfsapi/test/second/words.txt"));
		FileOutputFormat.setOutputPath(job, new Path("/wordcount/output"));

		// 提交job
		job.waitForCompletion(true);
	}
}

四、Hadoop-MapReduce-词频统计-本地测试

• 去掉连接hdfs的配置

• 设置Job对应的参数：设置输入/输出路径，设置成本地路径即可。

  public class WordCountLocalFileApp {
  
  	public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
  
  		// 创建一个job
  		Job job = Job.getInstance();
  
  		// 设置Job对应的参数：主类
  		job.setJarByClass(WordCountLocalFileApp.class);
  
  		// 设置Job对应的参数：设置自定义的Mapper和Reducer处理类
  		job.setMapperClass(WordCountMapper.class);
  		job.setReducerClass(WordCountReducer.class);
  
  		// 设置Job对应的参数：Mapper输出key和value的类型
  		job.setMapOutputKeyClass(Text.class);
  		job.setMapOutputValueClass(IntWritable.class);
  
  		// 设置Job对应的参数：Reducer输出key和value的类型
  		job.setOutputKeyClass(Text.class);
  		job.setOutputValueClass(IntWritable.class);
  
  		// 设置Job对应的参数：设置输入/输出路径
  		FileInputFormat.setInputPaths(job, new Path("/Users/xxxx/IdeaProjects/bigdata/hadoop-mapreduce/src/main/resources/words.txt"));
  		FileOutputFormat.setOutputPath(job, new Path("/Users/xxxx/IdeaProjects/bigdata/hadoop-mapreduce/src/main/resources/output"));
  
  		// 提交job
  		job.waitForCompletion(true);
  	}
  }
  

五、Hadoop-MapReduce-词频统计-Combiner

• map端的聚合操作就叫combiner

• combiner的优点/局限

  • 减少IO，提升执行效率
  • 求除法运算时，不适合。

• 案例代码改造：将每个map的输出，先进行累加操作，再输出到reducer

  // 设置Combiner
  job.setCombinerClass(WordCountReducer.class);