MapReduce WordCount案例

需求

在给定的文本文件中统计输出每一个单词出现的总次数
（1）输入数据
Hello World
Hello Java
Hello Hadoop
Hello MR
(2)期望输出的数据
Hello 4
World 1
Java 1
Hadoop 1
MR 1

需求分析

按照MapReduce编程规范，分别编写Mapper，Reducer，Driver。
1）Mapper
将MapTask传过来的文本内容先转换为String
Hello World
根据空格将一行内容切分为单词
Hello
World
将单词输出为<单词，1>
Hello，1
World，1
2）Reducer
汇总各个key的个数
Hello，1
Hello，1
Hello，1
Hello，1
输出该key的总次数
Hello，4
3）Driver
1.获取配置信息，获取job对象实例
2.指定本程序的jar包所在的本地路径
3.关联Mapper/Reducer业务类
4.指定Mapper输出数据的kv类型
5.指定最终输出的数据的kv类型
6.指定job的输入原始文件所在目录
7.指定job的输出结果所在目录
8.提交作业

环境准备

（1）创建maven项目
（2）在pom.xml文件中添加如下依赖

<dependencies>
	 <dependency>
		 <groupId>org.apache.hadoop</groupId>
		 <artifactId>hadoop-client</artifactId>
		 <version>3.1.3</version>
	 </dependency>
	 <dependency>
		 <groupId>junit</groupId>
		 <artifactId>junit</artifactId>
		 <version>4.12</version>
	 </dependency>
	 <dependency>
		 <groupId>org.slf4j</groupId>
		 <artifactId>slf4j-log4j12</artifactId>
		 <version>1.7.30</version>
	 </dependency>
</dependencies>

编写程序

编写Mapper类

package com.atguigu.mapreduce.wordcount1;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

/**
 * @author:hjc
 * @create 2022/8/15 14:27:08
 */

public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {

    Text k = new Text();
    IntWritable v = new IntWritable(1);

    protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, IntWritable>.Context context) throws IOException, InterruptedException {

        // 1 获取一行
        String line = value.toString();

        // 2 切割
        String[] words = line.split(" ");

        // 3 输出
        for (String word : words) {
            k.set(word);
            context.write(k, v);
        }
    }

}

编写Reducer类

package com.atguigu.mapreduce.wordcount1;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;


/**
 * @author:hjc
 * @create 2022/8/15 14:34:08
 */

public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {

    int sum;
    IntWritable v = new IntWritable();

    protected void reduce(Text key, Iterable<IntWritable> values, Reducer<Text, IntWritable, Text, IntWritable>.Context context) throws IOException, InterruptedException {

        // 1 累加求和
        sum = 0;
        for (IntWritable count : values) {
            sum += count.get();
        }

        // 2 输出
        v.set(sum);
        context.write(key, v);
    }
}

编写Driver驱动类

package com.atguigu.mapreduce.wordcount1;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import java.io.IOException;

/**
 * @author:hjc
 * @create 2022/8/15 15:27:08
 */

public class WordCountDriver {

    public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {

        // 1 获取配置信息以及获取job对象
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf);

        // 2 关联本Driver程序的jar
        job.setJarByClass(WordCountDriver.class);

        // 3 关联Mapper和Reducer的jar
        job.setMapperClass(WordCountMapper.class);
        job.setReducerClass(WordCountReducer.class);

        // 4 设置Mapper输出的kv类型
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);

        // 5 设置Reducer输出的kv类型
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        // 6 设置输入和输出路径
        FileInputFormat.setInputPaths(job, new Path("src/main/java/com/atguigu/mapreduce/wordcount1/input/hello.txt"));
        FileOutputFormat.setOutputPath(job, new Path("src/main/java/com/atguigu/mapreduce/wordcount1/output"));

        // 7 提交job
        boolean result = job.waitForCompletion(true);
        System.exit(result ? 0 : 1);
    }

}

调整日志打印级别

在项目的 src/main/resources 目录下，新建一个文件，命名为“log4j.properties”，在
文件中填入。

log4j.rootLogger=INFO, stdout 
log4j.appender.stdout=org.apache.log4j.ConsoleAppender 
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout 
log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - %m%n 
log4j.appender.logfile=org.apache.log4j.FileAppender 
log4j.appender.logfile.File=target/spring.log 
log4j.appender.logfile.layout=org.apache.log4j.PatternLayout 
log4j.appender.logfile.layout.ConversionPattern=%d %p [%c] - %m%n

如果还是会打印详细日志，查看target/classes目录下，是否存在log4j.properties文件，没有则将resources目录下的log4j.properties文件复制过去

将MapReduce程序打包提交到集群测试

（1）用maven打jar包，需要添加的打包插件依赖

<build>
	 <plugins>
		 <plugin>
			 <artifactId>maven-compiler-plugin</artifactId>
			 <version>3.6.1</version>
			 <configuration>
				 <source>1.8</source>
				 <target>1.8</target>
			 </configuration>
		 </plugin>
		 <plugin>
			 <artifactId>maven-assembly-plugin</artifactId>
			 <configuration>
				 <descriptorRefs>
				 	<descriptorRef>jar-with-dependencies</descriptorRef>
				 </descriptorRefs>
			 </configuration>
			 <executions>
				 <execution>
					 <id>make-assembly</id>
					 <phase>package</phase>
					 <goals>
					 	<goal>single</goal>
					 </goals>
				 </execution>
			 </executions>
		 </plugin>
	 </plugins>
</build>

注意：如果工程上显示红叉。在项目上右键->maven->Reimport刷新即可。

（2）将程序打成jar包

（3）修改不带依赖的jar包名称为wc.jar，并拷贝该jar包到Hadoop集群的/opt/module/hadoop-3.1.3路径。
（4）启动Hadoop集群
/opt/module/hadoop-3.1.3/sbin/start-all.sh
（5）WordCount程序
hadoop jar wc.jar com.atguigu.mapreduce.wordcount1.WordCountDriver /user/atguigu/input /user/atguigu/output