Hadoop学习

最新推荐文章于 2021-07-10 15:22:23 发布
最新推荐文章于 2021-07-10 15:22:23 发布
阅读量404 收藏 1
点赞数 1
分类专栏： Hadoop

一、    概念知识介绍

        Hadoop MapReduce是一个用于处理海量数据的分布式计算框架。这个框架解决了诸如数据分布式存储、作业调度、容错、机器间通信等复杂问题，可以使没有并行 处理或者分布式计算经验的工程师，也能很轻松地写出结构简单的、应用于成百上千台机器处理大规模数据的并行分布式程序。

       Hadoop MapReduce基于“分而治之”的思想，将计算任务抽象成map和reduce两个计算过程，可以简单理解为“分散运算—归并结果”的过程。一个 MapReduce程序首先会把输入数据分割成不相关的若干键/值对（key1/value1）集合，这些键/值对会由多个map任务来并行地处理。 MapReduce会对map的输出（一些中间键/值对key2/value2集合）按照key2进行排序，排序是用memcmp的方式对key在内存中 字节数组比较后进行升序排序，并将属于同一个key2的所有value2组合在一起作为reduce任务的输入，由reduce任务计算出最终结果并输出 key3/value3。作为一个优化，同一个计算节点上的key2/value2会通过combine在本地归并。基本流程如下：

       Hadoop和单机程序计算流程对比：

       常计算任务的输入和输出都是存放在文件里的，并且这些文件被存放在Hadoop分布式文件系统HDFS（Hadoop Distributed File System）中，系统会尽量调度计算任务到数据所在的节点上运行，而不是尽量将数据移动到计算节点上，减少大量数据在网络中传输，尽量节省带宽消耗。

       应用程序开发人员一般情况下需要关心的是图中灰色的部分，单机程序需要处理数据读取和写入、数据处理；Hadoop程序需要实现map和 reduce，而数据读取和写入、map和reduce之间的数据传输、容错处理等由Hadoop MapReduce和HDFS自动完成。

二、    开发环境搭建

       Map/Reduce程序依赖Hadoop集群，另外Eclipse需要安装依赖的hadoop包。

       Hadoop集群搭建：参考Hadoop 2.2.0集群搭建

1.   安装、配置Eclipse

       在官网下载合适的Eclipse，将hadoop开发所依赖的插件jar包拷贝到eclipse的安装文件夹plugins下。下载地址参考：hadoop2.2.0开发依赖的jar包，当然也可以自己编译。

       启动eclipse，选择Window—>Prefrances，若出现如下Hadoop Map/Reduce说明插件安装成功

2.   配置DFS，主要是数据文件的输入输出管理。

       Window—>Open Perspective—>other—>Map/Reduce，显示Map/Reduce视图。点击Map/Reduce Locations 的小象图标，新建Hadoop Location，输入如下：

       项目视图会出现DFS Location，用来管理输入、输出数据文件。

       需要配置hadoop安装文件夹：新建Map/Reduce工程单击Configure Hadoop install direction，输入hadoop的安装路径。

       右键单击DFS Location下的空文件夹上传一个文本文件，然后刷新，若文件出现了则说明环境配置成功。

三、    编程模型

       MapReduce编程模型的原理是：利用一个输入key/value pair集合来产生一个输出的key/value pair集合。MapReduce库的用户用两个函数表达这个计算：Map和Reduce。

       用户自定义的Map函数接受一个输入的key/value pair值，然后产生一个中间key/value pair值的集合。MapReduce库把所有具有相同中间key值I的中间value值集合在一起后传递给reduce函数。

       用户自定义的Reduce函数接受一个中间key的值I和相关的一个value值的集合。Reduce函数合并这些value值，形成一个较小的 value值的集合。一般的，每次Reduce函数调用只产生0或1个输出value值。通常我们通过一个迭代器把中间value值提供给Reduce函 数，这样我们就可以处理无法全部放入内存中的大量的value值的集合。

四、    小例子

1.      数据准备

       以Tomcat日志为例，日志格式如下：

  1. 127.0.0.1,-,-,[08/May/2014:13:42:40 +0800],GET / HTTP/1.1,200,11444  
  2. 127.0.0.1,-,-,[08/May/2014:13:42:42 +0800],GET /jygl/jaxrs/teachingManage/ClassBatchPlanService/getCurrentClassPlanVO HTTP/1.1,204,-  
  3. 127.0.0.1,-,-,[08/May/2014:13:42:42 +0800],GET /jygl/jaxrs/teachingManage/ClassBatchPlanService/getCurClassPlanVO HTTP/1.1,204,-  
  4. 127.0.0.1,-,-,[08/May/2014:13:42:47 +0800],GET /jygl/jaxrs/right/isValidUserByType/1-admin-superadmin HTTP/1.1,200,20  
  5. 127.0.0.1,-,-,[08/May/2014:13:42:47 +0800],GET /jygl/jaxrs/right/getUserByLoginName/admin HTTP/1.1,200,198  
  6. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:42:47 +0800],GET /jyglFront/right_login2home?loginName=admin&password=superadmin&type=1 HTTP/1.1,200,2525  
  7. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:42:47 +0800],GET /jyglFront/mainView/navigate/style/style.css HTTP/1.1,304,-  
  8. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:42:47 +0800],GET /jyglFront/mainView/navigate/js/tree.js HTTP/1.1,304,-  
  9. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:42:47 +0800],GET /jyglFront/mainView/navigate/js/jquery.js HTTP/1.1,304,-  
  10. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:42:47 +0800],GET /jyglFront/mainView/navigate/js/frame.js HTTP/1.1,304,-  
  11. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:42:47 +0800],GET /jyglFront/mainView/navigate/images/logo.png HTTP/1.1,304,-  
  12. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:42:47 +0800],GET /jyglFront/mainView/navigate/images/leftmenu_bg.gif HTTP/1.1,404,1105  
  13. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:42:47 +0800],GET /jyglFront/mainView/navigate/menuList.jsp HTTP/1.1,200,47603  
  14. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:42:47 +0800],GET /jyglFront/mainView/navigate/style/images/header_bg.jpg HTTP/1.1,304,-  
  15. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:42:47 +0800],GET /jyglFront/mainView/navigate/images/allmenu.gif HTTP/1.1,404,1093  
  16. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:42:47 +0800],GET /jyglFront/mainView/navigate/images/toggle_menu.gif HTTP/1.1,404,1105  
  17. 127.0.0.1,-,-,[08/May/2014:13:42:48 +0800],GET /jygl/jaxrs/article/getArticleList/10-1 HTTP/1.1,200,20913  
  18. 127.0.0.1,-,-,[08/May/2014:13:42:48 +0800],GET /jygl/jaxrs/article/getTotalArticleRecords HTTP/1.1,200,22  
  19. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:42:48 +0800],GET /jyglFront/baseInfo_articleList?flag=1 HTTP/1.1,200,8989  
  20. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:42:48 +0800],GET /jyglFront/mainView/studentView/style/images/nav_10.png HTTP/1.1,404,1117  
  21. 127.0.0.1,-,-,[08/May/2014:13:43:21 +0800],GET /jygl/jaxrs/right/isValidUserByType/1-admin-superadmin HTTP/1.1,200,20  
  22. 127.0.0.1,-,-,[08/May/2014:13:43:21 +0800],GET /jygl/jaxrs/right/getUserByLoginName/admin HTTP/1.1,200,198  
  23. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:21 +0800],GET /jyglFront/right_login2home?loginName=admin&password=superadmin&type=1 HTTP/1.1,200,2525  
  24. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:21 +0800],GET /jyglFront/mainView/navigate/js/tree.js HTTP/1.1,304,-  
  25. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:21 +0800],GET /jyglFront/mainView/navigate/js/jquery.js HTTP/1.1,304,-  
  26. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:21 +0800],GET /jyglFront/mainView/navigate/js/frame.js HTTP/1.1,304,-  
  27. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:21 +0800],GET /jyglFront/mainView/navigate/style/style.css HTTP/1.1,304,-  
  28. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:21 +0800],GET /jyglFront/mainView/navigate/menuList.jsp HTTP/1.1,200,47603  
  29. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:21 +0800],GET /jyglFront/mainView/navigate/images/logo.png HTTP/1.1,304,-  
  30. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:21 +0800],GET /jyglFront/mainView/navigate/images/leftmenu_bg.gif HTTP/1.1,404,1105  
  31. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:21 +0800],GET /jyglFront/mainView/navigate/images/toggle_menu.gif HTTP/1.1,404,1105  
  32. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:21 +0800],GET /jyglFront/mainView/navigate/style/images/header_bg.jpg HTTP/1.1,304,-  
  33. 127.0.0.1,-,-,[08/May/2014:13:43:21 +0800],GET /jygl/jaxrs/article/getArticleList/10-1 HTTP/1.1,200,20913  
  34. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:21 +0800],GET /jyglFront/mainView/navigate/images/allmenu.gif HTTP/1.1,404,1093  
  35. 127.0.0.1,-,-,[08/May/2014:13:43:21 +0800],GET /jygl/jaxrs/article/getTotalArticleRecords HTTP/1.1,200,22  
  36. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:21 +0800],GET /jyglFront/baseInfo_articleList?flag=1 HTTP/1.1,200,8989  
  37. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:21 +0800],GET /jyglFront/mainView/studentView/style/images/nav_10.png HTTP/1.1,404,1117  
  38. 127.0.0.1,-,-,[08/May/2014:13:43:25 +0800],GET /jygl/jaxrs/graduate/graduateBatchService/getGraduateBatchByConditions?graduateBatchName=&pageSize=10&pageNo=1 HTTP/1.1,200,597  
  39. 127.0.0.1,-,-,[08/May/2014:13:43:25 +0800],GET /jygl/jaxrs/graduate/graduateBatchService/getTotalGraduateBatchByCondition?graduateBatchName= HTTP/1.1,200,21  
  40. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:26 +0800],GET /jyglFront/graduate_initGraduateBatch HTTP/1.1,200,8766  
  41. 127.0.0.1,-,-,[08/May/2014:13:43:27 +0800],GET /jygl/jaxrs/exam/examParameterService/getAllStudyCenters HTTP/1.1,200,29089  
  42. 127.0.0.1,-,-,[08/May/2014:13:43:27 +0800],GET /jygl/jaxrs/exam/examParameterService/getAllGradeInfo HTTP/1.1,200,3785  
  43. 127.0.0.1,-,-,[08/May/2014:13:43:27 +0800],GET /jygl/jaxrs/enroll/educationLevelService/allEducationLevels HTTP/1.1,200,227  
  44. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:28 +0800],GET /jyglFront/graduate_initGraduateQulifyCheck HTTP/1.1,200,26397  
  45. 127.0.0.1,-,-,[08/May/2014:13:43:29 +0800],GET /jygl/jaxrs/exam/examParameterService/getAllStudyCenters HTTP/1.1,200,29089  
  46. 127.0.0.1,-,-,[08/May/2014:13:43:29 +0800],GET /jygl/jaxrs/exam/examParameterService/getAllGradeInfo HTTP/1.1,200,3785  
  47. 127.0.0.1,-,-,[08/May/2014:13:43:29 +0800],GET /jygl/jaxrs/enroll/educationLevelService/allEducationLevels HTTP/1.1,200,227  
  48. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:29 +0800],GET /jyglFront/graduate_initLeaveSchoolInfo HTTP/1.1,200,20125  
  49. 127.0.0.1,-,-,[08/May/2014:13:43:30 +0800],GET /jygl/jaxrs/exam/examParameterService/getAllStudyCenters HTTP/1.1,200,29089  
  50. 127.0.0.1,-,-,[08/May/2014:13:43:31 +0800],GET /jygl/jaxrs/exam/examParameterService/getAllGradeInfo HTTP/1.1,200,3785  
  51. 127.0.0.1,-,-,[08/May/2014:13:43:31 +0800],GET /jygl/jaxrs/enroll/educationLevelService/allEducationLevels HTTP/1.1,200,227  
  52. 127.0.0.1,-,-,[08/May/2014:13:43:31 +0800],GET /jygl/jaxrs/graduate/graduateBatchService/getAllGraduateBatch HTTP/1.1,200,597  
  53. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:13:43:31 +0800],GET /jyglFront/graduate_initGraduateInfo HTTP/1.1,200,28464  
  54. 127.0.0.1,-,-,[08/May/2014:14:27:10 +0800],GET / HTTP/1.1,200,11444  
  55. 127.0.0.1,-,-,[08/May/2014:14:27:12 +0800],GET /jygl/jaxrs/teachingManage/ClassBatchPlanService/getCurrentClassPlanVO HTTP/1.1,204,-  
  56. 127.0.0.1,-,-,[08/May/2014:14:27:12 +0800],GET /jygl/jaxrs/teachingManage/ClassBatchPlanService/getCurClassPlanVO HTTP/1.1,204,-  
  57. 127.0.0.1,-,-,[08/May/2014:14:27:34 +0800],GET /jygl/jaxrs/exam/examArrangeService/getExamBatchIdByLatest HTTP/1.1,200,43  
  58. 127.0.0.1,-,-,[08/May/2014:14:27:34 +0800],GET /jygl/jaxrs/exam/examArrangeService/getExamBatchNameByEBId/4af2a0424323412e014327739b1702bd HTTP/1.1,200,16  
  59. 127.0.0.1,-,-,[08/May/2014:14:27:35 +0800],GET /jygl/jaxrs/exam/examSubscribeService/getUtilObjectThirExamBatchsByEBNN/201403 HTTP/1.1,200,653  
  60. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:35 +0800],GET /jyglFront/exam_initgroupsubscribestatistic HTTP/1.1,200,13551  
  61. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:37 +0800],GET /jyglFront/exam_initsubstudentsubscribe HTTP/1.1,500,3900  
  62. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:41 +0800],GET /jyglFront/supervisor/intoInitAssignmentDetail HTTP/1.1,200,1808  
  63. 127.0.0.1,-,-,[08/May/2014:14:27:42 +0800],GET /jygl/jaxrs/right/isValidUserByType/1-admin-superadmin HTTP/1.1,200,20  
  64. 127.0.0.1,-,-,[08/May/2014:14:27:42 +0800],GET /jygl/jaxrs/right/getUserByLoginName/admin HTTP/1.1,200,198  
  65. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:42 +0800],GET /jyglFront/right_login2home?loginName=admin&password=superadmin&type=1 HTTP/1.1,200,2525  
  66. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:42 +0800],GET /jyglFront/mainView/navigate/js/tree.js HTTP/1.1,304,-  
  67. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:42 +0800],GET /jyglFront/mainView/navigate/style/style.css HTTP/1.1,304,-  
  68. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:42 +0800],GET /jyglFront/mainView/navigate/js/frame.js HTTP/1.1,304,-  
  69. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:42 +0800],GET /jyglFront/mainView/navigate/js/jquery.js HTTP/1.1,304,-  
  70. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:42 +0800],GET /jyglFront/mainView/navigate/menuList.jsp HTTP/1.1,200,47603  
  71. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:42 +0800],GET /jyglFront/mainView/navigate/images/leftmenu_bg.gif HTTP/1.1,404,1105  
  72. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:42 +0800],GET /jyglFront/mainView/navigate/images/allmenu.gif HTTP/1.1,404,1093  
  73. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:42 +0800],GET /jyglFront/mainView/navigate/images/logo.png HTTP/1.1,304,-  
  74. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:42 +0800],GET /jyglFront/mainView/navigate/style/images/header_bg.jpg HTTP/1.1,304,-  
  75. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:42 +0800],GET /jyglFront/mainView/navigate/images/toggle_menu.gif HTTP/1.1,404,1105  
  76. 127.0.0.1,-,-,[08/May/2014:14:27:42 +0800],GET /jygl/jaxrs/article/getArticleList/10-1 HTTP/1.1,200,20913  
  77. 127.0.0.1,-,-,[08/May/2014:14:27:42 +0800],GET /jygl/jaxrs/article/getTotalArticleRecords HTTP/1.1,200,22  
  78. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:42 +0800],GET /jyglFront/baseInfo_articleList?flag=1 HTTP/1.1,200,8989  
  79. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:43 +0800],GET /jyglFront/mainView/studentView/style/images/nav_10.png HTTP/1.1,404,1117  
  80. 127.0.0.1,-,-,[08/May/2014:14:27:44 +0800],GET /jygl/jaxrs/nationInfo/getAllNationInPage?pageSize=10&pageNo=1 HTTP/1.1,200,374  
  81. 127.0.0.1,-,-,[08/May/2014:14:27:44 +0800],GET /jygl/jaxrs/nationInfo/getTotalNations HTTP/1.1,200,22  
  82. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:44 +0800],GET /jyglFront/baseInfo_nationInfoList HTTP/1.1,200,7471  
  83. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:44 +0800],GET /jyglFront/common/css/menuStyle2.css HTTP/1.1,404,1060  
  84. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:44 +0800],GET /jyglFront/common/css/basic.css HTTP/1.1,200,1476  
  85. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:45 +0800],GET /jyglFront/common/css/_images/botton2.gif HTTP/1.1,404,1075  
  86. 127.0.0.1,-,-,[08/May/2014:14:27:47 +0800],GET /jygl/jaxrs/enroll/educationLevelService/allEducationLevels HTTP/1.1,200,227  
  87. 127.0.0.1,-,-,[08/May/2014:14:27:47 +0800],GET /jygl/jaxrs/enroll/gradeInfoService/allGradeInfos HTTP/1.1,200,3785  
  88. 127.0.0.1,-,-,[08/May/2014:14:27:47 +0800],GET /jygl/jaxrs/teaching/teachingPlanService/getSpeicalListByTwo?gradeID=&educationLevelID= HTTP/1.1,200,12061  
  89. 127.0.0.1,-,-,[08/May/2014:14:27:47 +0800],GET /jygl/jaxrs/enroll/studyCenterService/allStudyCentersByUtilObject HTTP/1.1,200,6006  
  90. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:48 +0800],GET /jyglFront/teaching/openReplaceChooseCourse HTTP/1.1,200,26455  
  91. 127.0.0.1,-,-,[08/May/2014:14:27:49 +0800],GET /jygl/jaxrs/teachingManage/ClassBatchPlanService/getCurClassBatchPlanVOList?newClassBatchName=&gradeName=&term=-1 HTTP/1.1,204,-  
  92. 127.0.0.1,-,-,[08/May/2014:14:27:49 +0800],GET /jygl/jaxrs/teachingManage/ClassBatchPlanService/getCurClassBatchPlanVOList?newClassBatchName=&gradeName=&term=-1 HTTP/1.1,204,-  
  93. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:49 +0800],GET /jyglFront/teaching/openChooseCourse HTTP/1.1,200,1611  
  94. 127.0.0.1,-,-,[08/May/2014:14:27:51 +0800],GET /jygl/jaxrs/enroll/gradeInfoService/currentGradeInfo HTTP/1.1,200,473  
  95. 127.0.0.1,-,-,[08/May/2014:14:27:51 +0800],GET /jygl/jaxrs/enroll/educationLevelService/allEducationLevels HTTP/1.1,200,227  
  96. 127.0.0.1,-,-,[08/May/2014:14:27:51 +0800],GET /jygl/jaxrs/enroll/gradeInfoService/allGradeInfos HTTP/1.1,200,3785  
  97. 127.0.0.1,-,-,[08/May/2014:14:27:51 +0800],GET /jygl/jaxrs/teaching/teachingPlanService/hasTeachingPlanInGrade?gradeId=4af2a042437c2c0801437ed1cdea0017 HTTP/1.1,200,20  
  98. 127.0.0.1,-,-,[08/May/2014:14:27:51 +0800],GET /jygl/jaxrs/teaching/teachingPlanService/hasTeachingPlanInGrade?gradeId=4af2a0423f41d66d013f5a1f766c00ce HTTP/1.1,200,20  
  99. 127.0.0.1,-,-,[08/May/2014:14:27:51 +0800],GET /jygl/jaxrs/teaching/teachingPlanService/teachingPlanListByEducationLevelAndGradeId?grade=4af2a042437c2c0801437ed1cdea0017&educationLevel= HTTP/1.1,200,4849  
  100. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:52 +0800],GET /jyglFront/teaching/teachingPlanList HTTP/1.1,200,22794  
  101. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:27:52 +0800],GET /jyglFront/js/jquery.form.js HTTP/1.1,200,30330  
  102. 127.0.0.1,-,-,[08/May/2014:14:28:02 +0800],GET /jygl/jaxrs/exam/examArrangeService/getExamBatchIdByLatest HTTP/1.1,200,43  
  103. 127.0.0.1,-,-,[08/May/2014:14:28:02 +0800],GET /jygl/jaxrs/exam/examArrangeService/getExamBatchNameByEBId/4af2a0424323412e014327739b1702bd HTTP/1.1,200,16  
  104. 127.0.0.1,-,-,[08/May/2014:14:28:02 +0800],GET /jygl/jaxrs/exam/examSubscribeService/getUtilObjectThirExamBatchsByEBNN/201403 HTTP/1.1,200,653  
  105. 0:0:0:0:0:0:0:1,-,-,[08/May/2014:14:28:02 +0800],GET /jyglFront/exam_initgroupsubscribestatistic HTTP/1.1,200,13551  
  106. 127.0.0.1,-,-,[08/May/2014:14:28:19 +0800],POST /jygl/jaxrs/right/addUserLog HTTP/1.1,200,-  
  107. 127.0.0.1,-,-,[08/May/2014:14:31:42 +0800],GET /jygl/jaxrs/exam/examSubscribeService/groupSubscribe/201403/0/0/201309/1 HTTP/1.1,200,-  

2.      要解决的问题：统计资源（URL）被访问的次数。

3.      编程实现

       想法：解析Tomcat日志，map的工作是将每一行日志中的URL截取作为key值，value为1表示1次，reduce的工作是将相同key值的行合并，value为总次数。

代码如下：

  1. package org.ly.ccnu;  
  2. import java.io.IOException;  
  3. import org.apache.hadoop.conf.Configuration;  
  4. import org.apache.hadoop.conf.Configured;  
  5. import org.apache.hadoop.fs.Path;  
  6. import org.apache.hadoop.io.IntWritable;  
  7. import org.apache.hadoop.io.LongWritable;  
  8. import org.apache.hadoop.io.NullWritable;  
  9. import org.apache.hadoop.io.Text;  
  10. import org.apache.hadoop.mapreduce.Job;  
  11. import org.apache.hadoop.mapreduce.Mapper;  
  12. import org.apache.hadoop.mapreduce.Reducer;  
  13. import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;  
  14. import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;  
  15. import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;  
  16. import org.apache.hadoop.util.Tool;  
  17. import org.apache.hadoop.util.ToolRunner;  
  18. public class SecondTest extends Configured implements Tool{  
  19.     enum Counter{  
  20.         LINESKIP,  
  21.     }     
  22.     public static class Map extends Mapper<LongWritable,Text,Text,IntWritable>{  
  23.         private static final IntWritable one = new IntWritable(1);   
  24.         public void map(LongWritable key,Text value,Context context) throws IOException, InterruptedException{  
  25.             String line = value.toString();  
  26.             try{  
  27.                 String[] lineSplit = line.split(",");  
  28.                 String requestUrl = lineSplit[4];  
  29.                 requestUrl = requestUrl.substring(requestUrl.indexOf(' ')+1, requestUrl.lastIndexOf(' '));  
  30.                 Text out = new Text(requestUrl);  
  31.                 context.write(out,one);  
  32.             }catch(java.lang.ArrayIndexOutOfBoundsException e){  
  33.                 context.getCounter(Counter.LINESKIP).increment(1);  
  34.             }             
  35.         }  
  36.     }     
  37.     public static class Reduce extends Reducer<Text,IntWritable,Text,IntWritable>{          
  38.         public void reduce(Text key, Iterable<IntWritable> values,Context context)throws IOException{  
  39.             int count =  0;    
  40.             for(IntWritable v : values){    
  41.                 count = count + 1;    
  42.             }    
  43.             try {  
  44.                 context.write(key, new IntWritable(count));  
  45.             } catch (InterruptedException e) {  
  46.                  e.printStackTrace();  
  47.             }             
  48.         }         
  49.     }     
  50.     @Override  
  51.     public int run(String[] args) throws Exception {  
  52.         Configuration conf = getConf();  
  53.         Job job = new Job(conf, "logAnalysis");  
  54.         job.setJarByClass(SecondTest.class);          
  55.         FileInputFormat.addInputPath(job, new Path(args[0]));  
  56.         FileOutputFormat.setOutputPath(job, new Path(args[1]));       
  57.         job.setMapperClass(Map.class);  
  58.         job.setReducerClass(Reduce.class);  
  59.         job.setOutputFormatClass(TextOutputFormat.class);         
  60.         //keep the same format with the output of Map and Reduce  
  61.         job.setOutputKeyClass(Text.class);  
  62.         job.setOutputValueClass(IntWritable.class);       
  63.         job.waitForCompletion(true);  
  64.         return job.isSuccessful()?0:1;  
  65.     }     
  66.     public static void main(String[] args)throws Exception{       
  67.         int res = ToolRunner.run(new Configuration(), new SecondTest(),args);         
  68.         System.exit(res);  
  69.     }  
  70. }  

4.      处理结果

[plain] view plain copy
  1. /   2  
  2. /jygl/jaxrs/article/getArticleList/10-1 3  
  3. /jygl/jaxrs/article/getTotalArticleRecords  3  
  4. /jygl/jaxrs/enroll/educationLevelService/allEducationLevels 5  
  5. /jygl/jaxrs/enroll/gradeInfoService/allGradeInfos   2  
  6. /jygl/jaxrs/enroll/gradeInfoService/currentGradeInfo    1  
  7. /jygl/jaxrs/enroll/studyCenterService/allStudyCentersByUtilObject   1  
  8. /jygl/jaxrs/exam/examArrangeService/getExamBatchIdByLatest  2  
  9. /jygl/jaxrs/exam/examArrangeService/getExamBatchNameByEBId/4af2a0424323412e014327739b1702bd 2  
  10. /jygl/jaxrs/exam/examParameterService/getAllGradeInfo   3  
  11. /jygl/jaxrs/exam/examParameterService/getAllStudyCenters    3  
  12. /jygl/jaxrs/exam/examSubscribeService/getUtilObjectThirExamBatchsByEBNN/201403  2  
  13. /jygl/jaxrs/exam/examSubscribeService/groupSubscribe/201403/0/0/201309/1    1  
  14. /jygl/jaxrs/graduate/graduateBatchService/getAllGraduateBatch   1  
  15. /jygl/jaxrs/graduate/graduateBatchService/getGraduateBatchByConditions?graduateBatchName=&pageSize=10&pageNo=1  1  
  16. /jygl/jaxrs/graduate/graduateBatchService/getTotalGraduateBatchByCondition?graduateBatchName=   1  
  17. /jygl/jaxrs/nationInfo/getAllNationInPage?pageSize=10&pageNo=1  1  
  18. /jygl/jaxrs/nationInfo/getTotalNations  1  
  19. /jygl/jaxrs/right/addUserLog    1  
  20. /jygl/jaxrs/right/getUserByLoginName/admin  3  
  21. /jygl/jaxrs/right/isValidUserByType/1-admin-superadmin  3  
  22. /jygl/jaxrs/teaching/teachingPlanService/getSpeicalListByTwo?gradeID=&educationLevelID= 1  
  23. /jygl/jaxrs/teaching/teachingPlanService/hasTeachingPlanInGrade?gradeId=4af2a0423f41d66d013f5a1f766c00ce    1  
  24. /jygl/jaxrs/teaching/teachingPlanService/hasTeachingPlanInGrade?gradeId=4af2a042437c2c0801437ed1cdea0017    1  
  25. /jygl/jaxrs/teaching/teachingPlanService/teachingPlanListByEducationLevelAndGradeId?grade=4af2a042437c2c0801437ed1cdea0017&educationLevel=  1  
  26. /jygl/jaxrs/teachingManage/ClassBatchPlanService/getCurClassBatchPlanVOList?newClassBatchName=&gradeName=&term=-1   2  
  27. /jygl/jaxrs/teachingManage/ClassBatchPlanService/getCurClassPlanVO  2  
  28. /jygl/jaxrs/teachingManage/ClassBatchPlanService/getCurrentClassPlanVO  2  
  29. /jyglFront/baseInfo_articleList?flag=1  3  
  30. /jyglFront/baseInfo_nationInfoList  1  
  31. /jyglFront/common/css/_images/botton2.gif   1  
  32. /jyglFront/common/css/basic.css 1  
  33. /jyglFront/common/css/menuStyle2.css    1  
  34. /jyglFront/exam_initgroupsubscribestatistic 2  
  35. /jyglFront/exam_initsubstudentsubscribe 1  
  36. /jyglFront/graduate_initGraduateBatch   1  
  37. /jyglFront/graduate_initGraduateInfo    1  
  38. /jyglFront/graduate_initGraduateQulifyCheck 1  
  39. /jyglFront/graduate_initLeaveSchoolInfo 1  
  40. /jyglFront/js/jquery.form.js    1  
  41. /jyglFront/mainView/navigate/images/allmenu.gif 3  
  42. /jyglFront/mainView/navigate/images/leftmenu_bg.gif 3  
  43. /jyglFront/mainView/navigate/images/logo.png    3  
  44. /jyglFront/mainView/navigate/images/toggle_menu.gif 3  
  45. /jyglFront/mainView/navigate/js/frame.js    3  
  46. /jyglFront/mainView/navigate/js/jquery.js   3  
  47. /jyglFront/mainView/navigate/js/tree.js 3  
  48. /jyglFront/mainView/navigate/menuList.jsp   3  
  49. /jyglFront/mainView/navigate/style/images/header_bg.jpg 3  
  50. /jyglFront/mainView/navigate/style/style.css    3  
  51. /jyglFront/mainView/studentView/style/images/nav_10.png 3  
  52. /jyglFront/right_login2home?loginName=admin&password=superadmin&type=1  3  
  53. /jyglFront/supervisor/intoInitAssignmentDetail  1  
  54. /jyglFront/teaching/openChooseCourse    1  
  55. /jyglFront/teaching/openReplaceChooseCourse 1  
  56. /jyglFront/teaching/teachingPlanList    1
efficiency_shang
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
MapReduce-Demo:一个演示MapReduce（Hadoop）程序，用于处理少量数据
05-09
MapReduce-Demo 一个演示MapReduce程序，用于处理少量数据 如何： * 1。 通过更新Maven依赖项来编译项目 * 2。 通过指向主要方法类（Students.java）从eclipse导出为.jar * 3。 将输入数据加载到HDFS COMMAND TO LOAD THE INPUT DATA TO HDFS: hadoop fs -put "/my_folder_location/students.txt" hdfs:/students.txt * 4。使用罐子执行作业并提供粗化（输入和输出文件路径） COMMAND TO EXECUTE THE MAPREDUCE JOB: hadoop jar students-high-mark.jar hdfs:/new-students.txt hdfs:/student-out9.txt * 5。 验证
Hadoop学习：Map/Reduce初探与小Demo实现
热门推荐
且行且码且思且娱
05-09 1万+
一、    概念知识介绍         Hadoop MapReduce是一个用于处理海量数据的分布式计算框架。这个框架解决了诸如数据分布式存储、作业调度、容错、机器间通信等复杂问题，可以使没有并行 处理或者分布式计算经验的工程师，也能很轻松地写出结构简单的、应用于成百上千台机器处理大规模数据的并行分布式程序。        Hadoop MapReduce基于“分而治之”的思想，将计算任务
MapReduce Demo
weixin_34194359的博客
11-08 69
功能：统计公司员工一个月内手机上网上行流量、下行流量及总流量。 测试数据如下：13612345678     6000    100013612345678     2000    300013812345678     2000    10013812345678     1500    30013512345678     9000    20013512345678     500     ...
Hadoop（一）MapReduce demo
anniewang2014的博客
09-18 155
Mapreduce基础编程模型：将一个大任务拆分成一个个小任务，再进行汇总。 MapReduce是分两个阶段：map阶段：拆；reduce阶段：聚合。 hadoop环境安装 安装： 1、解压 : tar -zxvf hadoop-2.4.1.tar.gz -C /root/training/ 2、设置环境变量： vi ~/.bash_profile HADOO...
Hadoop MapReduce
yfk的专栏
05-03 3389
Hadoop MapReduce是一个用于处理海量数据的分布式计算框架。这个框架解决了诸如数据分布式存储、作业调度、容错、机器间通信等复杂问题，可以使没有并行处理或者分布式计算经验的工程师，也能很轻松地写出结构简单的、应用于成百上千台机器处理大规模数据的并行分布式程序。 Hadoop MapReduce基于“分而治之”的思想，将计算任务抽象成map和reduce两个计算过程，可以简单理解为“分散运算—归并结果”的过程。一个MapReduce程序首先会把输入数据分割成不相关的若干键/值对（key1/valu
HADOOP学习
10-28
1、 hadoop官方网站，首页会有最新动态。 2、 Nutch ->谷歌GFS论文->doug 根据GFS设计了NDFS、06年启动hadoop项目。 3、 环境支持 操作系统 Linux 、WINDOWS-Cygwin、hadoop-for-windows JDK支持 下载jdk，解压jdk...
Hadoop学习笔记
02-26
Hadoop学习笔记，自己总结的一些Hadoop学习笔记，比较简单。
Hadoop学习资料
12-28
Hadoop学习资料总结，值得推荐阅读学习 很好 非常好 值得拥有
Hadoop-1.2-运行demo（MapReduce测试）
祁_z
07-10 425
专业术语叫：运行一个mapreduce（分布式计算） Hadoop提供的jar包demo： hadoop-mapreduce-examples-2.4.1.jar提供的demo有： pi计算圆周率、wordcount统计相同单词数量 ${hadoop安装目录}/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar 计算圆周率（自带的jar包） 启动一个job计算任务，将这个job分五成个map运行。 [root@wee...
hadoop入门1：MapReduce 入门demo - wordcont
每天进步一点
09-13 1044
今天初次接触MapReduce，虽然目前很少有人用MR，但是我相信这块还是要学一下的，因为这块思想会影响后面的学习。 第一个demo也是最经典的wordcount，wordcount基本上涵盖了80%的hadoop业务，适合入门；后面写的播客将不会进行多余的解释，模板基本上和wordcount一样，只是业务一样而已 mapper： import java.io.IOException; ...
Python开发mapreduce的demo
07-13
使用Python开发mapreduce的简单demo的代码，大家可以将代码放到环境中，然后修改具run.sh里面的路径执行就好。
demo of mapreduce
weixin_44736028的博客
06-05 125
package com; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; impo
Hadoop之MapReduce详解
Markble
08-18 4573
目录 一、 MapReduce概述 1.1 MapReduce定义 1.2 MapReduce优缺点 1.2.1 优点 1.2.2 缺点 1.3 MapReduce核心思想 1.4 MapReduce进程 1.5 官方WordCount源码 1.6 常用数据序列化类型 二、 Hadoop序列化 2.1 序列化概述 2.2 自定义bean对象实现序列化接口（Writable） 三、 MapReduce框架原理 3.1 InputFormat数据输入 3.1.1 切片与MapTas
mapreduce小demo
qq_42252606的博客
05-06 991
包含三个文件：主文件，mapper文件，reduce文件 主文件： public class MyWC { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(true); ...
Hadoop学习之路（一、入门demo）
Walker_zmc的博客
10-15 174
参考https://www.jianshu.com/p/35ef70dfb651?utm_campaign=maleskine&utm_content=note&utm_medium=seo_notes&utm_source=recommendation 一、用idea建一个普通maven项目 二、添加pom依赖 <dependencies> ...
mapreduce demo实现
maluhanx的博客
04-12 771
mapreduce主要的实现需要三个类:主方法类、map类、reduce类。1. 主方法类：public class BreadPointDriver { /** * * @param args * @throws IOException * @throws ClassNotFoundException * @throws Interrup...
mapreduce demo
linuf的专栏
06-07 428
1. jar files: D:\userdata\nian\Desktop\search code\Char01\HadoopExample\lib\hadoop-0.20.2-core.jar D:\userdata\nian\Desktop\search code\Char01\HadoopExample\lib\commons-cli-1.2.jar D:\userdata\
hadoop_api_demo
flyDeDog的博客
05-23 190
准备 配置客户端hosts 关闭防火墙 常用api import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.*; import org.apache.hadoop.fs.permission.FsPermission; import java.io.IOException; import ...
hadoop学习路线图
最新发布
10-26
Hadoop学习路线图是指在学习Hadoop分布式系统基础架构时，需要掌握的一些基础知识和技能的学习路线。根据引用和引用提供的信息，Hadoop学习路线图可以分为以下四个阶段： 1. Java零基础学习路线 2. Hadoop基础部分...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值