hadoop -- topk

最新推荐文章于 2024-06-24 18:59:20 发布

游九河

最新推荐文章于 2024-06-24 18:59:20 发布

阅读量156

点赞数

分类专栏： hadoop 文章标签： hadoop mapreduce topk 最小堆最大堆

本文链接：https://blog.csdn.net/qq_40337206/article/details/89946217

版权

hadoop 专栏收录该内容

15 篇文章 0 订阅

订阅专栏

需求

对100万条在0~1000万之间的数据取 TopK
输入数据：
8995149,5191755,2093544,9816608,4360204,5507730,1289204,6166586,8955325,9567003,
8264570,5202810,5353570,279129,9281133,6205171,5684360,1666876,4727056,2383250,
9176282,2815703,5145176,7514591,8648912,4633139,4176111,8954834,1702148,7978927,
…

最小堆

最小堆，是一种经过排序的完全二叉树，其中任一非终端节点的数据值均不大于其左子节点和右子节点的值。

最大堆

最大堆要求根节点的数据值既大于或等于左子树的数据值，又大于或等于右子树的数据值。
图片来自百度

求topk：最简单版

在每个map中建一个包含K个元素的数组，添加元素时，都和数组中最小/最大的元素比较，大于/小于就替换。缺点：速度慢。时间复杂度：O(n*K)

求topk：改进版

在每个map中建一个包含K个元素的最小堆/最大堆，添加元素时，都和数组中堆顶的元素比较，大于/小于就添加。时间复杂度：O(n*logK)，最大topk使用最小堆，最小topk使用最大堆。

最大1000个元素的java代码实现

// An highlighted block

public class Topk extends Configured implements Tool {

	private static final NullWritable NULL = NullWritable.get();
	private static final int N = 1000;

	public static class Map extends Mapper<LongWritable, Text, IntWritable, NullWritable> {

		private static final String SPLIT = ",";
		PriorityQueue<Integer> priorityQueue = new PriorityQueue(N);

		@Override
		protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
			String line = value.toString();
			String[] ss = line.split(SPLIT);
			for (int i = 0; i < ss.length; i++) {
				Integer number = Integer.parseInt(ss[i]);

				Integer p = priorityQueue.peek();
				if (priorityQueue.size() < N || number >= p) {
					priorityQueue.offer(number);
				}
				if (priorityQueue.size() > N) {
					priorityQueue.poll();
				}
			}
		}

		@Override
		protected void cleanup(Mapper<LongWritable, Text, IntWritable, NullWritable>.Context context)
				throws IOException, InterruptedException {
			for (Integer integer : priorityQueue) {
				context.write(new IntWritable(integer), NULL);
			}
			priorityQueue.clear();
			priorityQueue = null;
		}

	}

	public static class Reduce extends Reducer<IntWritable, NullWritable, IntWritable, NullWritable> {

		PriorityQueue<Integer> priorityQueue = new PriorityQueue(N);

		@Override
		protected void reduce(IntWritable key, Iterable<NullWritable> values, Context context)
				throws IOException, InterruptedException {
			for (NullWritable nullWritable : values) {
				Integer number = key.get();

				Integer p = priorityQueue.peek();
				if (priorityQueue.size() < N || number >= p) {
					priorityQueue.offer(number);
				}
				if (priorityQueue.size() > N) {
					priorityQueue.poll();
				}
			}
		}

		@Override
		protected void cleanup(Reducer<IntWritable, NullWritable, IntWritable, NullWritable>.Context context)
				throws IOException, InterruptedException {
			for (Integer integer : priorityQueue) {
				context.write(new IntWritable(integer), NULL);
			}
			priorityQueue.clear();
			priorityQueue = null;
		}
	}

	public int run(String[] args) throws Exception {

		// create job
		Job job = Job.getInstance();
		// set job's name
		job.setJobName("mr topk");
		// set job's class
		job.setJarByClass(Topk.class);

		// job set map's class
		job.setMapperClass(Map.class);
		// job set reduce's class
		job.setReducerClass(Reduce.class);

		// job set class of map's output
		job.setMapOutputKeyClass(IntWritable.class);
		job.setMapOutputValueClass(NullWritable.class);

		// job set class of output's key
		job.setOutputKeyClass(IntWritable.class);
		// job set class of output's value
		job.setOutputValueClass(NullWritable.class);

		// set class of inputFormat
		job.setInputFormatClass(TextInputFormat.class);
		// set class of outputFormat
		// job.setOutputFormatClass(TextOutputFormat.class);

		FileInputFormat.setInputPaths(job, args[0]);
		FileOutputFormat.setOutputPath(job, new Path(args[1]));

		job.setNumReduceTasks(1);

		boolean success = job.waitForCompletion(true);
		return success ? 0 : 1;
	}

	public static void main(String[] args) throws Exception {

		int run = ToolRunner.run(new Topk(), args);
		System.exit(run);
	}
}

结果

游九河

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
hadoop -- topk

需求对100万条在0~1000万之间的数据取 TopK输入数据：8995149,5191755,2093544,9816608,4360204,5507730,1289204,6166586,8955325,9567003,8264570,5202810,5353570,279129,9281133,6205171,5684360,1666876,4727056,2383250,9176...
复制链接

扫一扫