大数据学习06_Hadoop: MapReduce概述

MapReduce概述

MapReduce核心

MapReduce运算程序分成至少2个阶段

1. 第一个阶段的MapTask并发实例完全并行运行,互不相干
2. 第二个阶段的ReduceTask并发实例完全并行运行,互不相干. 但他们的数据依赖于上一个阶段所有MapTask并发实例的输出。
3. MapReduce编程模型只能包含一个Map阶段和一个Reduce阶段,如果业务逻辑非常复杂,也只能多个MapReduce程序串行运行

一个完整的MapReduce程序在分布式运行时有三类实例进程:

1. MrAppMaster: 负责整个程序的过程调度及状态协调
2. MapTask: 负责Map阶段的整个数据处理流程
3. ReduceTask: 负责Reduce阶段的整个数据处理流程

MapReduce编程规范

1. Mapper类

   1. 用户自定义的Mapper类要继承jar包中的Mapper父类
   2. Mapper的输入数据是KV对的形式(KV的类型可自定义)
   3. Mapper中的业务逻辑写在map()方法中
   4. Mapper的输出数据是KV对的形式(KV的类型可自定义)
   5. MapTask进程对每一组<K,V>调用一次map()方法

2. Reducer类

   1. 用户自定义的Reducer类要继承jar包中的Reducer父类
   2. Reducer的输入数据类型对应Mapper的输出数据类型
   3. Reducer的业务逻辑写在reduce()方法中
   4. ReduceTask进程对每一组<k,v>调用一次reduce()方法

3. Driver类
   相当于YARN集群的客户端,用于提交我们整个程序到YARN集群,提交的是封装了MapReduce程序相关运行参数的job对象

4. MapReduce的数据序列化类型如下

   Java类型	Hadoop类型
   boolean	BooleanWritable
   byte	ByteWritable
   int	IntWritable
   float	FloatWritable
   long	LongWritable
   double	DoubleWritable
   String	Text
   map	MapWritable
   array	ArrayWritable

MapReduce案例实操1: WordCount

1. 环境准备:

   1. 将hadoop的jar包所在位置加入系统路径
   2. 新建Maven工程,添加如下依赖

   <dependencies>
   	<dependency>
   		<groupId>junit</groupId>
   		<artifactId>junit</artifactId>
   		<version>RELEASE</version>
   	</dependency>
   	<dependency>
   		<groupId>org.apache.logging.log4j</groupId>
   		<artifactId>log4j-core</artifactId>
   		<version>2.8.2</version>
   	</dependency>
   	<dependency>
   		<groupId>org.apache.hadoop</groupId>
   		<artifactId>hadoop-common</artifactId>
   		<version>2.7.7</version>
   	</dependency>
   	<dependency>
   		<groupId>org.apache.hadoop</groupId>
   		<artifactId>hadoop-client</artifactId>
   		<version>2.7.7</version>
   	</dependency>
   	<dependency>
   		<groupId>org.apache.hadoop</groupId>
   		<artifactId>hadoop-hdfs</artifactId>
   		<version>2.7.7</version>
   	</dependency>
   </dependencies>
   

2. 编写程序
   

   1. 编写Mapper类

      package cn.maoritian.mapreduce;
      
      import java.io.IOException;
      
      import org.apache.hadoop.io.IntWritable;
      import org.apache.hadoop.io.LongWritable;
      import org.apache.hadoop.io.Text;
      import org.apache.hadoop.mapreduce.Mapper;
      
      public class WordcountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
      	// Mapper对象的四个类型参数KEYIN, VALUEIN, KEYOUT, VALUEOUT
      	// 	KEYIN: 		输入key的类型: LongWritable(偏移量)
      	// 	VALUEIN: 	输入value的类型: Text(行内容)
      	// 	KEYOUT: 	输出key的类型: Text(单词字符串)
      	// 	VALUEOUT: 	输出value的类型: IntWritable(出现次数)
      
      	Text k = new Text();
      	IntWritable v = new IntWritable(1);
      	
      	@Override
      	protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
      		// 1 获取一行
      		String line = value.toString();
      		// 2 切割
      		String[] words = line.split(" ");
      		// 3 输出
      		for (String word : words) {
      			k.set(word);
      			context.write(k, v);
      		}
      	}
      }
      

   2. 编写Reducer类:

      package cn.maoritian.mapreduce;
      
      import java.io.IOException;
      
      import org.apache.hadoop.io.IntWritable;
      import org.apache.hadoop.io.Text;
      import org.apache.hadoop.mapreduce.Reducer;
      
      public class WordcountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
      	// Reducer对象的四个类型参数KEYIN, VALUEIN, KEYOUT, VALUEOUT
      	// 	KEYIN: 输入key的类型: Text(单词字符串)
      	// 	VALUEIN: 输入value的类型: IntWritable(出现次数)
      	// 	KEYOUT: 输出key的类型: Text(单词字符串)
      	// 	VALUEOUT: 输出value的类型: IntWritable(出现次数累加和)
      
      	int sum;
      	IntWritable v = new IntWritable();
      
      	@Override
      	protected void reduce(Text key, Iterable<IntWritable> values, Context context)
      			throws IOException, InterruptedException {
      
      		// 1. 累加求和
      		sum = 0;
      		for (IntWritable count : values) {
      			sum += count.get();
      		}
      
      		// 2. 输出
      		v.set(sum);
      		context.write(key, v);
      	}
      }
      

   3. 编写Driver类:

      package cn.maoritian.mapreduce;
      
      import java.io.IOException;
      
      import org.apache.hadoop.conf.Configuration;
      import org.apache.hadoop.fs.Path;
      import org.apache.hadoop.io.IntWritable;
      import org.apache.hadoop.io.Text;
      import org.apache.hadoop.mapreduce.Job;
      import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
      import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
      
      public class WordcountDriver {
      
      	public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
      		// 1. 获取配置信息以及封装任务
      		Configuration configuration = new Configuration();
      		Job job = Job.getInstance(configuration);
      
      		// 2.设置jar加载路径为驱动类的路径
      		job.setJarByClass(WordcountDriver.class);
      
      		// 3.设置map和reduce类
      		job.setMapperClass(WordcountMapper.class);
      		job.setReducerClass(WordcountReducer.class);
      
      		// 4.设置map输出
      		job.setMapOutputKeyClass(Text.class);
      		job.setMapOutputValueClass(IntWritable.class);
      
      		// 5.设置最终输出kv类型
      		job.setOutputKeyClass(Text.class);
      		job.setOutputValueClass(IntWritable.class);
      		https://github.com/srccodes/hadoop-common-2.2.0-bin/tree/master/
      		// 6 设置输入和输出路径
      		FileInputFormat.setInputPaths(job, new Path(args[0]));
      		FileOutputFormat.setOutputPath(job, new Path(args[1]));
      
      		// 7 提交并退出
      		boolean result = job.waitForCompletion(true);
      
      		System.exit(result ? 0 : 1);
      	}
      }
      

   在Windows上运行时会出现一系列操蛋问题,见这个问题

3. 将程序打包成jar包,右键工程,点击run as->Maven install进行打包,打包好之后会在target目录下找到生成的jar包,重命名为wc.jar,将其传送到hadoop目录下.
   使用下边命令执行MapReduce程序.

   hadoop jar wc.jar cn.maoritian.mapreduce.WordcountDriver /wcinput/input.txt /wcoutput/	
   

Hadoop序列化

1. Hadoop序列化的意义,为什么不用Java的序列化框架?
   MapReduce的数据序列化类型包含了8种基本数据类型,我们通过将这8种数据序列化类型组合,可以用来序列化Java Bean数据类型.
   Java的序列化是一个重量级序列化框架(Serializable).序列化后的对象会附带很多额外的信息(各种校验信息,Header,继承体系等),不便于在网络中高效传输.因此Hadoop自己开发了一套序列化机制(Writable).

2. Hadoop序列化接口的实现
   具体实现bean对象序列化步骤如下7步:

   1. 实现Writable接口
   2. 反序列化时，需要反射调用空参构造函数，所以必须有空参构造

      public Bean类名() {
      	super();
      }
      

   3. 重写序列化方法write()

      @Override
      public void write(DataOutput out) throws IOException {
      	out.writeLong(参数1);
      	out.writeLong(参数2);
      	out.writeLong(参数3);
      }
      

   4. 重写反序列化方法readFields()

      @Override
      public void readFields(DataInput in) throws IOException {
      	参数1 = in.readLong();
      	参数2 = in.readLong();
      	参数3 = in.readLong();
      }
      

   5. 注意反序列化的顺序和序列化的顺序完全一致
   6. 要想把结果显示在文件中,可以重写toString()方法,字段用\t分开,方便后续用
   7. 如果需要将自定义的bean放在key中传输,则还需要实现Comparable接口,因为MapReduce框中的Shuffle过程要求对key必须能排序.

      @Override
      public int compareTo(FlowBean o) {
      	// 倒序排列，从大到小
      	return this.hashCode() - o.hashCode();
      }
      

MapReduce案例实操2: FlowCount

1. 需求: 统计每个手机号耗费的总上行流量,下行流量,总流量:
   输入数据格式：

   7	13560436666	120.196.100.99	1116		954			200
   id	手机号码		网络ip			上行流量		下行流量     网络状态码
   

   输出数据格式:

   13560436666		1116		954			2070
   手机号码			上行流量		下行流量		总流量
   

2. 需求分析:
   
3. 编写程序:
   1. 编写流量统计的FlowBean对象

      import java.io.DataInput;
      import java.io.DataOutput;
      import java.io.IOException;
      
      import org.apache.hadoop.io.Writable;
      
      //1. 实现Writable接口
      public class FlowBean implements Writable {
      
      	private long upFlow;		// 上行流量
      	private long downFlow;		// 下行流量
      	private long sumFlow;		// 总流量
      
      	//2. 反序列化时需要反射调用空参构造函数
      	public FlowBean() {
      		super();
      	}
      
      	//3. 实现序列化方法
      	@Override
      	public void write(DataOutput out) throws IOException {
      		out.writeLong(upFlow);
      		out.writeLong(downFlow);
      		out.writeLong(sumFlow);
      	}
      
      	//4. 反序列化方法
      	//5. 反序列化方法读顺序必须和写序列化方法的写顺序必须一致
      	@Override
      	public void readFields(DataInput in) throws IOException {
      		this.upFlow = in.readLong();
      		this.downFlow = in.readLong();
      		this.sumFlow = in.readLong();
      	}
      
      	//6. 后边要将序列化对象输出,因此实现其toString()方法
      	@Override
      	public String toString() {
      		return upFlow + "\t" + downFlow + "\t" + sumFlow;
      	}
      	
      	public void set(long upFlow, long downFlow, long sumFlow) {
      		this.upFlow = upFlow;
      		this.downFlow = downFlow;
      		this.sumFlow = sumFlow;
      	}
      }
      

   2. 编写Mapper类

      import java.io.IOException;
      
      import org.apache.hadoop.io.LongWritable;
      import org.apache.hadoop.io.Text;
      import org.apache.hadoop.mapreduce.Mapper;
      
      import cn.maoritian.bean.FlowBean;
      
      public class FlowCountMapper extends Mapper<LongWritable, Text, Text, FlowBean> {
      	//Mapper对象的四个类型参数KEYIN, VALUEIN, KEYOUT, VALUEOUT
      	//	KEYIN: 		输入key的类型: LongWritable(偏移量)
      	//	VALUEIN: 	输入value的类型: Text(行内容)
      	//	KEYOUT: 	输出key的类型: Text(手机号)
      	//	VALUEOUT: 	输出value的类型: FlowBean(流量统计)
      
      	Text k = new Text();
      	FlowBean v = new FlowBean();
      
      	@Override
      	protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
      		// 1. 获取一行
      		String line = value.toString();
      
      		// 2. 切割获取字段
      		String[] fields = line.split("\t");
      		String phoneNum = fields[1]; 		// 手机号码
      		long upFlow = Long.parseLong(fields[fields.length - 3]);	// 上行流量
      		long downFlow = Long.parseLong(fields[fields.length - 2]);	// 下行流量
      
      		// 3. 输出
      		k.set(phoneNum);
      		v.set(downFlow, upFlow, upFlow + downFlow);
      		context.write(k, v);
      	}
      }
      

   3. 编写Reducer类

      import java.io.IOException;
      
      import org.apache.hadoop.io.Text;
      import org.apache.hadoop.mapreduce.Reducer;
      
      import cn.maoritian.bean.FlowBean;
      
      public class FlowCountReducer extends Reducer<Text, FlowBean, Text, FlowBean> {
      	// Reducer对象的四个类型参数KEYIN, VALUEIN, KEYOUT, VALUEOUT
      	// KEYIN: 输入key的类型: Text(手机号)
      	// VALUEIN: 输入value的类型: FlowBean(流量统计)
      	// KEYOUT: 输出key的类型: Text(手机号)
      	// VALUEOUT: 输出value的类型: FlowBean(流量统计)
      
      	long sum_upFlow = 0;
      	long sum_downFlow = 0;
      	FlowBean resultBean = new FlowBean();
      
      	@Override
      	protected void reduce(Text key, Iterable<FlowBean> values, Context context)throws IOException, InterruptedException {
      
      		//1. 遍历累加求和
      		for (FlowBean flowBean : values) {
      			sum_upFlow += flowBean.getUpFlow();
      			sum_downFlow += flowBean.getDownFlow();
      		}
      
      		//2. 输出
      		resultBean.set(sum_upFlow, sum_downFlow, sum_upFlow+sum_downFlow);
      		context.write(key, resultBean);
      	}
      }
      

   4. 编写Driver驱动类

      import java.io.IOException;
      
      import org.apache.hadoop.conf.Configuration;
      import org.apache.hadoop.fs.Path;
      import org.apache.hadoop.io.Text;
      import org.apache.hadoop.mapreduce.Job;
      import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
      import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
      
      import cn.maoritian.bean.FlowBean;
      
      public class FlowCountDriver {
      
      	public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
      
      		// 1. 获取配置信息以及封装任务
      		Configuration configuration = new Configuration();
      		Job job = Job.getInstance(configuration);
      
      		// 2.设置jar加载路径为驱动类的路径
      		job.setJarByClass(FlowCountDriver.class);
      
      		// 3.设置map和reduce类
      		job.setMapperClass(FlowCountMapper.class);
      		job.setReducerClass(FlowCountReducer.class);
      
      		// 4.设置map输出
      		job.setMapOutputKeyClass(Text.class);
      		job.setMapOutputValueClass(FlowBean.class);
      
      		// 5.设置最终输出kv类型
      		job.setOutputKeyClass(Text.class);
      		job.setOutputValueClass(FlowBean.class);
      
      		// 6.设置输入和输出路径
      		FileInputFormat.setInputPaths(job, new Path(args[0]));
      		FileOutputFormat.setOutputPath(job, new Path(args[1]));
      
      		// 7.提交并退出
      		boolean result = job.waitForCompletion(true);
      		System.exit(result ? 0 : 1);
      	}
      }