MapReduce 归并排序利用之高效TopN案例

最新推荐文章于 2024-05-10 23:09:54 发布
最新推荐文章于 2024-05-10 23:09:54 发布
阅读量912 收藏 3
点赞数 1
分类专栏： hadoop mapreduce topn 文章标签： hadoop mapreduce
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/xiaohu21/article/details/108173994
版权
本文探讨了如何利用MapReduce的内部排序机制来实现高效TopN排序。通过介绍MapReduce排序原理、常规MapReduce实现TopN的问题，以及提出利用每个小阶段的排序进行归并排序的方法，详细讲解了如何自定义分区器、分组器以解决大数据量排序的内存问题，从而提升处理效率和速度。
摘要由CSDN通过智能技术生成

MapReduce 归并排序利用之高效TopN案例

1.MapReduce排序

  1. 根据mapreduce的内部机制可知，分区和排序贯穿整个过程。
  2. maptask的key，value输出到环形缓冲区中，
  3. 每个maptask的数据从环形缓冲区溢写到临时文件，
  4. 相同maptask的临时文件合并为一个文件，
  5. reduce从maptask产生的文件中针对key进行分组提取。

这些排序，默认都是根据key来进行字符串升序排序，也就是ASCII码表的字母顺序升序排序。

2.问题

  1. 当需要对一个很大的分布式文件中数据做排序，如何处理？
  2. 常规mapreduce方法有什么问题？
  3. 有哪些排序方法？
  4. 怎样利用mapreduce内部的排序机制来解决这个排序问题，进而提升处理效率和速度？

3.环境准备

  1. 准备一个电影评分原始数据. 实际可以使用for循环，创建很多Javabean对象，并赋予各个属性不同的值，然后利用fastjson把java bean转为字符串，再逐行写入到文本文件中。实际有十几兆，这里摘抄几十行数据。
{"movie":"1193","rate":"5","timeStamp":"978300760","uid":"1"}
{"movie":"661","rate":"3","timeStamp":"978302109","uid":"1"}
{"movie":"914","rate":"3","timeStamp":"978301968","uid":"1"}
{"movie":"3408","rate":"4","timeStamp":"978300275","uid":"1"}
{"movie":"2355","rate":"5","timeStamp":"978824291","uid":"1"}
{"movie":"1197","rate":"3","timeStamp":"978302268","uid":"1"}
{"movie":"1287","rate":"5","timeStamp":"978302039","uid":"1"}
{"movie":"2804","rate":"5","timeStamp":"978300719","uid":"1"}
{"movie":"594","rate":"4","timeStamp":"978302268","uid":"1"}
{"movie":"919","rate":"4","timeStamp":"978301368","uid":"1"}
{"movie":"595","rate":"5","timeStamp":"978824268","uid":"1"}
{"movie":"938","rate":"4","timeStamp":"978301752","uid":"1"}
{"movie":"2398","rate":"4","timeStamp":"978302281","uid":"1"}
{"movie":"2918","rate":"4","timeStamp":"978302124","uid":"1"}
{"movie":"1035","rate":"5","timeStamp":"978301753","uid":"1"}
{"movie":"2791","rate":"4","timeStamp":"978302188","uid":"1"}
{"movie":"2687","rate":"3","timeStamp":"978824268","uid":"1"}
{"movie":"2018","rate":"4","timeStamp":"978301777","uid":"1"}
{"movie":"3105","rate":"5","timeStamp":"978301713","uid":"1"}
{"movie":"2797","rate":"4","timeStamp":"978302039","uid":"1"}
{"movie":"2321","rate":"","timeStamp":"978302205","uid":"1"}
{"movie":"720","rate":"3","timeStamp":"978300760","uid":"1"}
{"movie":"1270","rate":"5","timeStamp":"978300055","uid":"1"}
{"movie":"527","rate":"5","timeStamp":"978824195","uid":"1"}
{"movie":"2340","rate":"3","timeStamp":"978300103","uid":"1"}
{"movie":"48","rate":"5","timeStamp":"978824351","uid":"1"}
{"movie":"1097","rate":"4","timeStamp":"978301953","uid":"1"}
{"movie":"1721","rate":"4","timeStamp":"978300055","uid":"1"}
{"movie":"2321","rate":"3","timeStamp":"978302205","uid":"1"}
{"movie":"1545","raete":"4","timeStamp":"978824139","uid":"1"}
{"movie":"745","rate":"3","timeStamp":"978824268","uid":"1"}
{"movie":"2294","rate":"4","timeStamp":"978824291","uid":"1"}
{"movie":"3186","rate":"4","timeStamp":"978300019","uid":"1"}
{"movie":"1566","rate":"a","timeStamp":"978824330","uid":"1"}
{"movie":"588","rate":"4","timeStamp":"978824268","uid":"1"}
{"movie":"1907","rate":"4","timeStamp":"978824330","uid":"1"}
{"movie":"783","rate":"4","timeStamp":"978824291","uid":"1"}
{"movie":"1836","rate":"5","timeStamp":"978300172","uid":"1"}
{"movie":"1022","rate":"5","timeStamp":"978300055","uid":"1"}
{"movie":"2762","rate":"4","timeStamp":"978302091","uid":"1"}
{"movie":"150","rate":"5","timeStamp":"978301777","uid":"1"}
{"movie":"1","rate":"5","timeStamp":"978824268","uid":"1"}
{"movie":"1961","rate":"5","timeStamp":"978301590","uid":"1"}
{"movie":"1962","rate":"4","timeStamp":"978301753","uid":"1"}
{"movie":"2692","rate":"4","timeStamp":"978301570","uid":"1"}
{"movie":"260","rate":"4","timeStamp":"978300760","uid":"1"}
{"movie":"1028","rate":"5","timeStamp":"978301777","uid":"1"}
{"movie":"1029","rate":"5","timeStamp":"978302205","uid":"1"}
{"movie":"1207","rate":"4","timeStamp":"978300719","uid":"1"}
{"movie":"2028","rate":"5","timeStamp":"978301619","uid":"1"}
{"movie":"531","rate":"4","timeStamp":"978302149","uid":"1"}
{"movie":"3114","rate":"4","timeStamp":"978302174","uid":"1"}
{"movie":"608","rate":"4","timeStamp":"978301398","uid":"1"}
{"movie":"1246","rate":"4","timeStamp":"978302091","uid":"1"}
{"movie":"1357","rate":"5","timeStamp":"978298709","uid":"2"}
{"movie":"3068","rate":"4","timeStamp":"978299000","uid":"2"}
{"movie":"1537","rate":"4","timeStamp":"978299620","uid":"2"}
{"movie":"647","rate":"3","timeStamp":"978299351","uid":"2"}
{"movie":"2194","rate":"4","timeStamp":"978299297","uid":"2"}
{"movie":"648","rate":"4","timeStamp":"978299913","uid":"2"}
{"movie":"2268","rate":"5","timeStamp":"978299297","uid":"2"}
{"movie":"2628","rate":"3","timeStamp":"978300051","uid":"2"}
{"movie":"1103","rate":"3","timeStamp":"978298905","uid":"2"}
{"movie":"2916","rate":"3","timeStamp":"978299809","uid":"2"}
{"movie":"3468","rate":"5","timeStamp":"978298542","uid":"2"}
{"movie":"1210","rate":"4","timeStamp":"978298151","uid":"2"}
{"movie":"1792","rate":"3","timeStamp":"978299941","uid":"2"}
{"movie":"1687","rate":"3","timeStamp":"978300174","uid":"2"}
{"movie":"1213","rate":"2","timeStamp":"978298458","uid":"2"}
{"movie":"3578","rate":"5","timeStamp":"978298958","uid":"2"}
{"movie":"2881","rate":"3","timeStamp":"978300002","uid":"2"}
{"movie":"3030","rate":"4","timeStamp":"978298434","uid":"2"}
{"movie":"1217","rate":"3","timeStamp":"978298151","uid":"2"}
{"movie":"3105","rate":"4","timeStamp":"978298673","uid":"2"}
{"movie":"434","rate":"2","timeStamp":"978300174","uid":"2"}
{"movie":"2126","rate":"3","timeStamp":"978300123","uid":"2"}
{"movie":"3107","rate":"2","timeStamp":"978300002","uid":"2"}
{"movie":"3108","rate":"3","timeStamp":"978299712","uid":"2"}
{"movie":"3035","rate":"4","timeStamp":"978298625","uid":"2"}
{"movie":"1253","rate":"3","timeStamp":"978299120","uid":"2"}
{"movie":"1610","rate":"5","timeStamp":"978299809","uid":"2"}
{"movie":"292","rate":"3","timeStamp":"978300123","uid":"2"}
{"movie":"2236","rate":"5","timeStamp":"978299220","uid":"2"}
{"movie":"3071","rate":"4","timeStamp":"978299120","uid":"2"}
{"movie":"902","rate":"2","timeStamp":"978298905","uid":"2"}
{"movie":"368","rate":"4","timeStamp":"978300002","uid":"2"}
{"movie":"1259","rate":"5","timeStamp":"978298841","uid":"2"}
{"movie":"3147","rate":"5","timeStamp":"978298652","uid":"2"}
{"movie":"1544","rate":"4","timeStamp":"978300174","uid":"2"}
{"movie":"1293","rate":"5","timeStamp":"978298261","uid":"2"}
{"movie":"1188","rate":"4","timeStamp":"978299620","uid":"2"}
{"movie":"3255","rate":"4","timeStamp":"978299321","uid":"2"}
{"movie":"3256","rate":"2","timeStamp":"978299839","uid":"2"}
{"movie":"3257","rate":"3","timeStamp":"978300073","uid":"2"}
{"movie":"110","rate":"5","timeStamp":"978298625","uid":"2"}
{"movie":"2278","rate":"3","timeStamp":"978299889","uid":"2"}
{"movie":"2490","rate":"3","timeStamp":"978299966","uid":"2"}
{"movie":"1834","rate":"4","timeStamp":"978298813","uid":"2"}
{"movie":"3471","rate":"5","timeStamp":"978298814","uid":"2"}
  1. 安装Idea2020版，创建一个maven项目，引入fastjson的依赖

4.常规的mapreduce实现topn排序

  1. 每部电影的每条评论都有rate评分这个属性，需要选出每部电影评分最高的3条数据，并且输出到文件
  2. 代码,从下列代码可以看出，如果数据量很大，再reduce的list排序时，内存可能会耗尽。
    java bean
package com.doit.hadoop.movie;

import org.apache.hadoop.io.Writable;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;

public class Movie implements Writable {
   
    /*
     * {"movie":"1193","rate":"5","timeStamp":"978300760","uid":"1"}
     * */
    private String movie;
    private double rate;
    private String timeStamp;
    private String uid;

    public String getMovie()
最低0.47元/天 解锁文章
闻香识代码
关注
  • 1
    点赞
  • 踩
  • 3
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
MapReduce编程小案例.5th—分组topn的简单实现
RobertDowneyLm的博客
05-15 2907
MapReduce编程小案例.5th—分组topn的简单实现需求：有如下一组数据：order001,u001,小米6,1999.9,2 order001,u001,雀巢咖啡,99.0,2 order001,u001,安慕希,250.0,2 order001,u001,经典红双喜,200.0,4 order001,u001,防水电脑包,400.0,2 order002,u002,小米手环,199.0...
hadoop之MapReduce---TopN案例
@小刘同学nice
04-24 893
需求 输出流量使用量在前10的用户信息 输入数据 13470253144 180 180 360 13509468723 7335 110349 117684 13560439638 918 4938 5856 13568436656 3597 25635 29232 13590439668 1116 954 2070 13630577991 6960 690 7650 13682846555...
【MapReduce】扩展案例 ---- TopN（求取数据集前N名）_topn maprreduce
pasehelpme的博客
05-10 756
FileOutputFormat.setOutputPath(job, new Path(“G:\Projects\IdeaProject-C\MapReduce\src\main\java\第七章_MR扩展案例\TopN\output1”));FileInputFormat.setInputPaths(job, new Path(“G:\Projects\IdeaProject-C\MapReduce\src\main\java\第二章_Hadoop序列化\output”));// 获取job对象。
【MapReduce】扩展案例 ---- TopN（求取数据集前N名）
懂得一千零一种,赋予你失败的方法！
01-07 1179
文章目录TopN（求取数据集前N名）需求需求分析代码实现① Bean类② Mapper 阶段③ Reducer 阶段④ Driver 阶段总结 TopN（求取数据集前N名） 需求 对之前统计的上、下行流量总值输出结果进行加工，输出流量使用量在前10的用户信息 （1）输入数据 （2）输出数据 返回顶部 需求分析 返回顶部 代码实现 ① Bean类 创建Bean类，封装上下行、总流量对象 创建无参构造方法 — 供反序列化使用 重写序列化、反序列化方法 重写toString()方法 继承自Wri..
MapReduce中的排序之 -- 归并排序
不积跬步无以至千里
10-03 1820
排序是大数据的核心精华. 快排和归并排序更是MapReduce中的精华部分. 以下是归并排序部分. 一、百度百科: 归并排序（MERGE-SORT）是建立在归并操作上的一种有效的排序算法,该算法是采用分治法（Divide and Conquer）的一个非常典型的应用。将已有序的子序列合并，得到完全有序的序列；即先使每个子序列有序，再使子序列段间有序。若将两个有序表合并成...
[MapReduce编程案例3-电影评分topn--高级编程API]
gefeng1209的博客
09-21 1018
1.问题：求每部电影评分最高的前n条记录 数据json字符串，如下： 之前的订单交易额排序实现，是在Reduce阶段完成的对同一个key的value进行排序 ArrayList<OrderBean> beans = new ArrayList(); //每次迭代创建新对象 for (OrderBean orderBean : values) { OrderBean be...
MapReduce编程之求Top3（TopN）
11-26
本案例中，我们面临的问题是利用MapReduce来找出分布在多个文件中的正整数的最大三个值，即求解“Top 3”。这个问题在数据分析、推荐系统等领域有广泛应用，比如找出最受欢迎的产品、热门搜索关键词等。 MapReduce...
MapReduce中实现案例中的排序案例
最新发布
06-26
在进行Hadoop架构中的案例之一排序，对应的JAVA源代码，进行MapReduce编译之后即可进行使用，其中具体操作步骤可见厦门大学大数据技术实验博客
词频统计+倒排索引+数据去重+TopN
01-24
在IT领域，尤其是在大数据处理和搜索引擎技术中，"词频统计+倒排索引+数据去重+TopN"是...通过运用MapReduce进行词频统计、构建倒排索引、执行数据去重和获取TopN结果，可以有效地处理大规模数据并提供高效的查询服务。
基本排序算法及其在MapReduce的应用
02-01
该文档为学习基本排序算法过程中的学习笔记，大部分内容从网络上其他渠道也能得到，仅用于记录备忘之用。冒泡、选择、插入三种作为基本的排序算法是必须要掌握的，而在MapReduce的实际应用中。在Map阶段，k-v溢写时...
数据排序MapReduce实例
08-25
数据排序MapReduce实例 对输入文件中每行进行排序，输入文件中的每行内容为一个数字，即一个数据。
第九章 MapReduce典型编程案例进阶之TopN算法案例实现
weixin_51524477的博客
11-28 632
TopN
编写MapReduce实现主播的播放量等数据的统计及TopN排序
qq_57152362的博客
11-24 219
这篇文章讲的很清楚 https://blog.csdn.net/weixin_47971206/article/details/121489733
MapReduce经典案例———TopN
qq_52623832的博客
04-18 1148
本案例的演示采用本地运行模式，对指定的本地 D:\\test\\input目录下的源文件（开头的文本截图，需要提前准备），实现TopN分析，得到文件中最大的五个数，并将结果输出到本地D:\\test\\output目录下(output目录不需要提前创建，否则会报错)，设置完毕后，运行程序。根据Map阶段的输出结果形式，同样在cn.itcast.mr.topN包下，自定义Reducer类，TopNReducer，如下图。使用MapReduce技术提取上述文本中最大五个数据，并将最终结果汇总到指定文件中。
MapReduce 三个经典案例（倒排索引、TopN、找共同好友）
西门催学不吹雪
05-22 2575
文章目录1. 倒排索引案例1.1 需求1.2 需求分析1.3 代码实现1.3.1 第一次处理1.3.2 第二次处理2. TopN 案例2.1 需求2.2 代码实现3. 找共同好友案例3.1 需求3.2 需求分析3.3 代码实现3.3.1 第一次处理3.3.2 第二次处理 1. 倒排索引案例 1.1 需求   有大量的文本（文档、网页），需要建立搜索索引。 输入数据 ① a.txt atguigu pingping atguigu ss atguigu ss     ② b.txt atguigu
MapReduce之页面访问次数的top—n，以及访问次数全局排序实现
SOARING
03-15 1548
数据样本： 2017/07/28 sina.com/lady/ 2017/07/28 sina.com/play 2017/07/28 sina.com/movie 2017/07/28 sina.com/music 2017/07/28 sina.com/sport 2017/07/28 sina.com/sport 2017/07/28 163.com/sport ......等 #中间...
idea编写mapreduceTopN程序,运用hdfs上的数据测试并将结果存到hdfs
Late_whale的博客 作者：代码搬运工老李的二师兄
12-10 1365
问题：选出蜀国中五虎将 原始数据如下： 序号 姓名 武力值 国家 1 刘备 68 蜀国 2 马超 90 蜀国 3 黄忠 91 蜀国 4 魏延 76 蜀国 5 姜维 92 蜀国 6 关羽 96 蜀国 7 严颜 78 蜀国 8 孟达 64 蜀国 9 张飞 88 蜀国 10马谡 76 蜀国 11 赵云 95 蜀国 12 法正 88 蜀国 预期结果如下： 6 关羽 96 蜀国 11 赵云 95 蜀国 5 ...
MapReduce案例-TopN(倒序排序)
m0_72168501的博客
12-14 1731
MapReduce案例-TopN(倒序排序)
MapReduce过程详解(基于hadoop2.x架构)
weixin_30699443的博客
05-29 328
本文基于hadoop2.x架构详细描述了mapreduce的执行过程，包括partition，combiner，shuffle等组件以及yarn平台与mapreduce编程模型的关系。 mapreduce的简介和优点 mapreduce是一个分布式运算程序的编程框架,是hadoop数据分析的核心. mapreduce的核心思想是将用户编写的逻辑代码和架构中的各个组件整合成一个分布式运算程序,实现...
MapReduce基本用法示例-自定义序列化、排序、分区、分组和topN
本文介绍了MapReduce的基本用法示例，主要包括自定义序列化、排序、分区、分组和topN。在使用这些功能之前，需要确保hadoop环境正常运行。 首先，在介绍具体用法之前，作者先介绍了项目的pom.xml文件和测试数据说明...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值