Hadoop-MapReduce-自定义OutputFormat

    OutputFormat是MapReduce输出的基类，所以实现MapReduce输出都继承了OutputFormat。
    TextOutputFormat：默认的输出格式，把每条记录写成文本行。它的键值可以是任意类型，因为TextOutputFormat调用toString()方法将它们转换为字符串。
    SequenceFileOutputFormat：将多个文件合并成一个文件输出。

自定义OutputFormat

    在一个MapReduce程序中根据数据的不同输出到不同的目录。比如下面将文件中的.com网址输出到com.txt，.net网址输出到net.txt。

www.alibaba.com
www.google.com
www.csdn.net
www.php.net
www.taobao.com
www.zhihu.com
www.youtube.com
www.github.com
www.cnki.net
www.minecraft.net
www.savefrom.net

    map中将value作为key。

public class OutputFormatMapper extends Mapper<LongWritable, Text, Text, NullWritable>{
	@Override
	protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, NullWritable>.Context context)
			throws IOException, InterruptedException {
		context.write(value, NullWritable.get());
	}
}

public class OutputFormatReducer extends Reducer<Text, NullWritable, Text, NullWritable> {
	@Override
	protected void reduce(Text key, Iterable<NullWritable> value,
			Reducer<Text, NullWritable, Text, NullWritable>.Context context) throws IOException, InterruptedException {
		context.write(key, NullWritable.get());
	}
}

    自定义OutputFormat类：

1. 创建一个类继承 FileOutputFormat，重写getRecordWriter() 方法，创建一个继承 RecordWriter 的类。

public class OutputFormatClass extends FileOutputFormat<Text, NullWritable> {
	@Override
	public RecordWriter<Text, NullWritable> getRecordWriter(TaskAttemptContext job)
			throws IOException, InterruptedException {
		return new OutPutFormatRecordWriter(job);
	}
}

public class OutPutFormatRecordWriter extends RecordWriter<Text, NullWritable> {

	FSDataOutputStream cfos;
	FSDataOutputStream nfos;
	
	public OutPutFormatRecordWriter(TaskAttemptContext job) {
		try {
			//1.获取文件系统
			FileSystem fs = FileSystem.get(job.getConfiguration());
			//2.获取输出流
			cfos = fs.create(new Path("e:/site/com.txt"));
			nfos = fs.create(new Path("e:/site/net.txt"));
		} catch (IOException e) {
			e.printStackTrace();
		}
	}

	@Override
	public void write(Text key, NullWritable value) throws IOException, InterruptedException {
		String site = key.toString();
		if (site.matches(".*com")) {
			cfos.write(site.getBytes());
			cfos.writeBytes("\r\n");
		} else {
			nfos.write(site.getBytes());
			nfos.writeBytes("\r\n");
		}
	}

	@Override
	public void close(TaskAttemptContext context) throws IOException, InterruptedException {
		IOUtils.closeStream(cfos);
		IOUtils.closeStream(nfos);
	}
}

    驱动类设置输出的OutputFormat，job.setOutputFormatClass(OutputFormatClass.class);。同时，虽然我们自定义了OutputFormat，但因为OutputFormat继承自FileOutputFormat，而FileOutputFormat需要输出一个 _SUCCESS 文件，所以需要指定输出目录 FileOutputFormat.setOutputPath(job, new Path(args[1]));

public class OutputFormatDriver {
	public static void main(String[] args) throws IllegalArgumentException, IOException, ClassNotFoundException, InterruptedException {
		args = new String[] { "e:/site.txt", "e:/output"};
		Configuration conf = new Configuration();
		Job job = Job.getInstance(conf);
		job.setJarByClass(OutputFormatDriver.class);
		job.setMapperClass(OutputFormatMapper.class);
		job.setReducerClass(OutputFormatReducer.class);
		job.setMapOutputKeyClass(Text.class);
		job.setMapOutputValueClass(NullWritable.class);
		job.setOutputKeyClass(Text.class);
		job.setOutputValueClass(NullWritable.class);
		FileInputFormat.setInputPaths(job, new Path(args[0]));
		//设置输出的OutputFormat
		job.setOutputFormatClass(OutputFormatClass.class);
		FileOutputFormat.setOutputPath(job, new Path(args[1]));
		boolean result = job.waitForCompletion(true);
		System.exit(result ? 0 : 1);
	}
}

com.txt：
www.alibaba.com
www.github.com
www.google.com
www.taobao.com
www.youtube.com
www.zhihu.com

---

net.txt：
www.cnki.net
www.csdn.net
www.minecraft.net
www.php.net
www.savefrom.net