mapreduce简单实现

最新推荐文章于 2021-12-11 14:39:15 发布
最新推荐文章于 2021-12-11 14:39:15 发布
阅读量1k 收藏
点赞数
分类专栏： 数据挖掘
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/HUSTLX/article/details/50850033
版权

第一步：登陆服务器：

ssh 2014210***@thumedia.org -p 6349

2014210***@thumedia.org's password:

输入密码之后可见：

Welcome to Ubuntu 12.04.4 LTS (GNU/Linux 3.2.0-61-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

New release '14.04.1 LTS' available.

Run 'do-release-upgrade' to upgrade to it.

Last login: Sun Dec 14 21:01:46 2014 from cluster0

2014210***@cluster-3-1:~$

第二步，在/home/2014210***文件夹下编写mapper.py /reducer.py。将数据集hw2复制到/user/2014210***文件夹下。

其中mapper.py如下：

#!/usr/bin/env python
import sys
import re
for line in sys.stdin:
    line = line.strip()
    words = line.split()
    for word in words:
        mi=re.search('2014(.*)',word)
        if mi:
                print "%s\t%s" % (mi.group(), 1)

其中reducer.py如下：

#!/usr/bin/env python
import sys
word2count={}
for line in sys.stdin:
    word,count=line.split('\t',1)
    try:
        word=int(word)
        count=int(count)
    except ValueError:
        continue
 
    try:
        word2count[word] = word2count[word]+count
    except:
        word2count[word]=count
for word in word2count.keys():
    print '%s\t%s'%(word,word2count[word])

第三步，测试mapper.py：

2014210***@cluster-3-1:~$ echo "foo 2014210199 foo 2014210*** quux 2014210*** labs foo bar quux" | ./mapper.py

2014210199     1

2014210***     1

2014210***     1

2014210***@cluster-3-1:~$

测试reducer.py：

2014210***@cluster-3-1:~$ echo "foo 2014210199 foo 2014210*** quux 2014210*** labs foo bar quux" | ./mapper.py | sort -k1,1 | ./reducer.py

2014210***     2

2014210199     1

2014210***@cluster-3-1:~$

可见map和reduce均正确。

第四步：运行：

2014210***@cluster-3-1:~$ hadoop jar /opt/cloudera/parcels/CDH/jars/hadoop-streaming-2.5.0-mr1-cdh5.2.0.jar -input /user/2014210***/hw2 -output /user/2014210***/output -mapper ./mapper.py -file ./mapper.py -reducer ./reducer.py -file ./reducer.py

14/12/14 21:17:22 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.

packageJobJar: [./mapper.py, ./reducer.py] [/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/jars/hadoop-streaming-2.5.0-cdh5.2.0.jar] /tmp/streamjob6018778792983891193.jar tmpDir=null

14/12/14 21:17:24 INFO client.RMProxy: Connecting to ResourceManager at cluster-4-0/192.168.5.20:8032

14/12/14 21:17:25 INFO client.RMProxy: Connecting to ResourceManager at cluster-4-0/192.168.5.20:8032

14/12/14 21:17:26 INFO mapred.FileInputFormat: Total input paths to process : 19

14/12/14 21:17:26 INFO mapreduce.JobSubmitter: number of splits:19

14/12/14 21:17:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1417174627956_0387

14/12/14 21:17:27 INFO impl.YarnClientImpl: Submitted application application_1417174627956_0387

14/12/14 21:17:27 INFO mapreduce.Job: The url to track the job: http://cluster-4-0:8088/proxy/application_1417174627956_0387/

14/12/14 21:17:27 INFO mapreduce.Job: Running job: job_1417174627956_0387

14/12/14 21:17:37 INFO mapreduce.Job: Job job_1417174627956_0387 running in uber mode : false

14/12/14 21:17:37 INFO mapreduce.Job:  map 0% reduce 0%

14/12/14 21:17:50 INFO mapreduce.Job:  map 6% reduce 0%

14/12/14 21:17:51 INFO mapreduce.Job:  map 10% reduce 0%

14/12/14 21:17:52 INFO mapreduce.Job:  map 11% reduce 0%

14/12/14 21:17:53 INFO mapreduce.Job:  map 15% reduce 0%

14/12/14 21:17:54 INFO mapreduce.Job:  map 18% reduce 0%

14/12/14 21:17:55 INFO mapreduce.Job:  map 19% reduce 0%

14/12/14 21:17:56 INFO mapreduce.Job:  map 24% reduce 0%

14/12/14 21:17:57 INFO mapreduce.Job:  map 27% reduce 0%

14/12/14 21:17:58 INFO mapreduce.Job:  map 28% reduce 0%

14/12/14 21:17:59 INFO mapreduce.Job:  map 33% reduce 0%

14/12/14 21:18:00 INFO mapreduce.Job:  map 35% reduce 0%

14/12/14 21:18:01 INFO mapreduce.Job:  map 37% reduce 0%

14/12/14 21:18:02 INFO mapreduce.Job:  map 41% reduce 0%

14/12/14 21:18:03 INFO mapreduce.Job:  map 43% reduce 0%

14/12/14 21:18:04 INFO mapreduce.Job:  map 45% reduce 0%

14/12/14 21:18:05 INFO mapreduce.Job:  map 49% reduce 0%

14/12/14 21:18:06 INFO mapreduce.Job:  map 52% reduce 0%

14/12/14 21:18:07 INFO mapreduce.Job:  map 53% reduce 0%

14/12/14 21:18:08 INFO mapreduce.Job:  map 59% reduce 0%

14/12/14 21:18:09 INFO mapreduce.Job:  map 66% reduce 0%

14/12/14 21:18:10 INFO mapreduce.Job:  map 68% reduce 0%

14/12/14 21:18:11 INFO mapreduce.Job:  map 81% reduce 0%

14/12/14 21:18:12 INFO mapreduce.Job:  map 91% reduce 0%

14/12/14 21:18:13 INFO mapreduce.Job:  map 92% reduce 0%

14/12/14 21:18:14 INFO mapreduce.Job:  map 94% reduce 0%

14/12/14 21:18:15 INFO mapreduce.Job:  map 96% reduce 0%

14/12/14 21:18:16 INFO mapreduce.Job:  map 100% reduce 0%

14/12/14 21:18:21 INFO mapreduce.Job:  map 100% reduce 1%

14/12/14 21:18:22 INFO mapreduce.Job:  map 100% reduce 7%

14/12/14 21:18:23 INFO mapreduce.Job:  map 100% reduce 14%

14/12/14 21:18:24 INFO mapreduce.Job:  map 100% reduce 21%

14/12/14 21:18:25 INFO mapreduce.Job:  map 100% reduce 22%

14/12/14 21:18:26 INFO mapreduce.Job:  map 100% reduce 24%

14/12/14 21:18:27 INFO mapreduce.Job:  map 100% reduce 25%

14/12/14 21:18:28 INFO mapreduce.Job:  map 100% reduce 29%

14/12/14 21:18:29 INFO mapreduce.Job:  map 100% reduce 36%

14/12/14 21:18:30 INFO mapreduce.Job:  map 100% reduce 38%

14/12/14 21:18:31 INFO mapreduce.Job:  map 100% reduce 42%

14/12/14 21:18:33 INFO mapreduce.Job:  map 100% reduce 46%

14/12/14 21:18:35 INFO mapreduce.Job:  map 100% reduce 49%

14/12/14 21:18:36 INFO mapreduce.Job:  map 100% reduce 56%

14/12/14 21:18:37 INFO mapreduce.Job:  map 100% reduce 60%

14/12/14 21:18:38 INFO mapreduce.Job:  map 100% reduce 61%

14/12/14 21:18:39 INFO mapreduce.Job:  map 100% reduce 64%

14/12/14 21:18:40 INFO mapreduce.Job:  map 100% reduce 67%

14/12/14 21:18:41 INFO mapreduce.Job:  map 100% reduce 68%

14/12/14 21:18:42 INFO mapreduce.Job:  map 100% reduce 72%

14/12/14 21:18:43 INFO mapreduce.Job:  map 100% reduce 76%

14/12/14 21:18:44 INFO mapreduce.Job:  map 100% reduce 83%

14/12/14 21:18:46 INFO mapreduce.Job:  map 100% reduce 89%

14/12/14 21:18:47 INFO mapreduce.Job:  map 100% reduce 90%

14/12/14 21:18:48 INFO mapreduce.Job:  map 100% reduce 93%

14/12/14 21:18:49 INFO mapreduce.Job:  map 100% reduce 94%

14/12/14 21:18:51 INFO mapreduce.Job:  map 100% reduce 96%

14/12/14 21:18:52 INFO mapreduce.Job:  map 100% reduce 100%

14/12/14 21:18:56 INFO mapreduce.Job: Job job_1417174627956_0387 completed successfully

14/12/14 21:18:56 INFO mapreduce.Job: Counters: 49

         File System Counters

                   FILE: Number of bytes read=1275232

                   FILE: Number of bytes written=12371381

                   FILE: Number of read operations=0

                   FILE: Number of large read operations=0

                   FILE: Number of write operations=0

                   HDFS: Number of bytes read=2088993303

                   HDFS: Number of bytes written=544

                   HDFS: Number of read operations=273

                   HDFS: Number of large read operations=0

                   HDFS: Number of write operations=144

         Job Counters

                   Launched map tasks=19

                   Launched reduce tasks=72

                   Data-local map tasks=19

                   Total time spent by all maps in occupied slots (ms)=2391716

                   Total time spent by all reduces in occupied slots (ms)=4403300

                   Total time spent by all map tasks (ms)=597929

                   Total time spent by all reduce tasks (ms)=440330

                   Total vcore-seconds taken by all map tasks=597929

                   Total vcore-seconds taken by all reduce tasks=440330

                   Total megabyte-seconds taken by all map tasks=2449117184

                   Total megabyte-seconds taken by all reduce tasks=4508979200

         Map-Reduce Framework

                   Map input records=190000000

                   Map output records=1790077

                   Map output bytes=23271001

                   Map output materialized bytes=1300723

                   Input split bytes=2005

                   Combine input records=0

                   Combine output records=0

                   Reduce input groups=32

                   Reduce shuffle bytes=1300723

                   Reduce input records=1790077

                   Reduce output records=32

                   Spilled Records=3580154

                   Shuffled Maps =1368

                   Failed Shuffles=0

                   Merged Map outputs=1368

                   GC time elapsed (ms)=1157

                   CPU time spent (ms)=819760

                   Physical memory (bytes) snapshot=37610864640

                   Virtual memory (bytes) snapshot=822321184768

                   Total committed heap usage (bytes)=167987642368

         Shuffle Errors

                   BAD_ID=0

                   CONNECTION=0

                   IO_ERROR=0

                   WRONG_LENGTH=0

                   WRONG_MAP=0

                   WRONG_REDUCE=0

         File Input Format Counters

                   Bytes Read=2088991298

         File Output Format Counters

                   Bytes Written=544

14/12/14 21:18:56 INFO streaming.StreamJob: Output directory: /user/2014210***/output

2014210***@cluster-3-1:~$

第五步：将output复制到/home/2014210***文件夹中

2014210***@cluster-3-1:~$ hadoop fs -copyToLocal /user/2014210***/output /home/2014210***

2014210***@cluster-3-1:~$ ls

hw2  mapper.py  output  reducer.py

2014210***@cluster-3-1:~$ cd output/

2014210***@cluster-3-1:~/output$ ls

part-00000  part-00013  part-00026  part-00039  part-00052  part-00065

part-00001  part-00014  part-00027  part-00040  part-00053  part-00066

part-00002  part-00015  part-00028  part-00041  part-00054  part-00067

part-00003  part-00016  part-00029  part-00042  part-00055  part-00068

part-00004  part-00017  part-00030  part-00043  part-00056  part-00069

part-00005  part-00018  part-00031  part-00044  part-00057  part-00070

part-00006  part-00019  part-00032  part-00045  part-00058  part-00071

part-00007  part-00020  part-00033  part-00046  part-00059  _SUCCESS

part-00008  part-00021  part-00034  part-00047  part-00060

part-00009  part-00022  part-00035  part-00048  part-00061

part-00010  part-00023  part-00036  part-00049  part-00062

part-00011  part-00024  part-00037  part-00050  part-00063

part-00012  part-00025  part-00038  part-00051  part-00064

用more命令查看结果：

2014210***@cluster-3-1:~/output$ more part*

::::::::::::::

part-00000

::::::::::::::

::::::::::::::

part-00001

::::::::::::::

2014210987     56009

2014211092     56231

2014210822     55916

::::::::::::::

part-00002

::::::::::::::

::::::::::::::

part-00003

::::::::::::::

2014211094     56024

::::::::::::::

part-00004

::::::::::::::

::::::::::::::

part-00005

::::::::::::::

::::::::::::::

part-00006

::::::::::::::

2014210827     55762

::::::::::::::

part-00007

::::::::::::::

::::::::::::::

part-00008

::::::::::::::

2014211099     56058

::::::::::::::

part-00009

::::::::::::::

::::::::::::::

part-00010

::::::::::::::

::::::::::::::

part-00011

::::::::::::::

::::::::::::::

part-00012

::::::::::::::

::::::::::::::

part-00013

::::::::::::::

2014211005     56067

::::::::::::::

part-00014

::::::::::::::

::::::::::::::

part-00015

::::::::::::::

2014210806     56051

::::::::::::::

part-00016

::::::::::::::

::::::::::::::

part-00017

::::::::::::::

::::::::::::::

part-00018

::::::::::::::

::::::::::::::

part-00019

::::::::::::::

::::::::::::::

part-00020

::::::::::::::

::::::::::::::

part-00021

::::::::::::::

::::::::::::::

part-00022

::::::::::::::

::::::::::::::

part-00023

::::::::::::::

::::::::::::::

part-00024

::::::::::::::

::::::::::::::

part-00025

::::::::::::::

2014210990     56303

::::::::::::::

part-00026

::::::::::::::

::::::::::::::

part-00027

::::::::::::::

::::::::::::::

part-00028

::::::::::::::

::::::::::::::

part-00029

::::::::::::::

::::::::::::::

part-00030

::::::::::::::

2014210830     55630

::::::::::::::

part-00031

::::::::::::::

2014210996     55980

::::::::::::::

part-00032

::::::::::::::

::::::::::::::

part-00033

::::::::::::::

2014211100     55803

::::::::::::::

part-00034

::::::::::::::

2014210834     56005

::::::::::::::

part-00035

::::::::::::::

2014210970     56025

2014211102     55520

::::::::::::::

part-00036

::::::::::::::

::::::::::::::

part-00037

::::::::::::::

2014210972     56566

::::::::::::::

part-00038

::::::::::::::

2014210838     55563

::::::::::::::

part-00039

::::::::::::::

::::::::::::::

part-00040

::::::::::::::

2014210810     56113

::::::::::::::

part-00041

::::::::::::::

2014210811     55899

::::::::::::::

part-00042

::::::::::::::

2014210812     56001

::::::::::::::

part-00043

::::::::::::::

2014211083     55875

2014210813     56485

::::::::::::::

part-00044

::::::::::::::

2014211084     55704

::::::::::::::

part-00045

::::::::::::::

2014211085     55587

::::::::::::::

part-00046

::::::::::::::

::::::::::::::

part-00047

::::::::::::::

2014211087     55539

::::::::::::::

part-00048

::::::::::::::

2014211088     55773

2014210818     55954

::::::::::::::

part-00049

::::::::::::::

2014210819     55811

::::::::::::::

part-00050

::::::::::::::

::::::::::::::

part-00051

::::::::::::::

::::::::::::::

part-00052

::::::::::::::

::::::::::::::

part-00053

::::::::::::::

::::::::::::::

part-00054

::::::::::::::

::::::::::::::

part-00055

::::::::::::::

::::::::::::::

part-00056

::::::::::::::

::::::::::::::

part-00057

::::::::::::::

::::::::::::::

part-00058

::::::::::::::

::::::::::::::

part-00059

::::::::::::::

::::::::::::::

part-00060

::::::::::::::

::::::::::::::

part-00061

::::::::::::::

::::::::::::::

part-00062

::::::::::::::

::::::::::::::

part-00063

::::::::::::::

::::::::::::::

part-00064

::::::::::::::

::::::::::::::

part-00065

::::::::::::::

::::::::::::::

part-00066

::::::::::::::

::::::::::::::

part-00067

::::::::::::::

2014210981     56055

2014210750     56156

::::::::::::::

part-00068

::::::::::::::

2014210982     55879

::::::::::::::

part-00069

::::::::::::::

::::::::::::::

part-00070

::::::::::::::

::::::::::::::

part-00071

::::::::::::::

2014210***     55733

2014210***@cluster-3-1:~/output$



hustlx
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
基于RPC的MapReduce简单实现.zip
03-13
标题中的“基于RPC的MapReduce简单实现”表明我们将探讨一种使用远程过程调用（RPC）技术来实现MapReduce框架的方法。MapReduce是Google提出的一种分布式计算模型，它将大规模数据处理任务分解为两个主要阶段：Map...
MapReduce极简实现
NULL
06-29 706
0 概述 MapReduce是一种广泛运用的分布式-大数据计算编程模型，最初由Google发表，其开源实现为Hadoop。 MapReduce 的编程模型非常简单，正如名字一样，用户仅仅需要实现一个 Map 函数，一个 Reduce 函数。 Map 函数，即映射函数：它会接受一个 key-value 对，然后把这个 key-value 对转换成 0 到多个新的 key-value 对并输出出...
MapReduce学习笔记（5）—— Map 端 join 实现
一角残叶的博客
08-09 338
1 订单、产品信息表
运行 C++ 版本的 mapreduce （Hadoop streaming）
生活不止眼前的枸杞， 还有蜂蜜和咖啡！
03-22 1018
mapper.cpp文件：#include<string> #include<iostream> using namespace std; int main() { string line; while(cin>>line) { cout<<line<<"\t"<<1<&amp
使用Python实现Hadoop MapReduce程序
风雪夜归子
09-11 1115
转自：使用Python实现Hadoop MapReduce程序 英文原文：Writing an Hadoop MapReduce Program in Python 根据上面两篇文章，下面是我在自己的ubuntu上的运行过程。文字基本采用博文使用Python实现Hadoop MapReduce程序， 打字很浪费时间滴。 在这个实例中，我将会向大家介绍如何使用Python 为 Hadoop编
Hadoop Streaming with Python(入门)
a18850765326的博客
12-11 899
首发于大数据分析挖掘写文章登录Hadoop Streaming with Python(新手向)孙云峰CodeLover/BugHunter/134 人赞同了该文章概述Hadoop Streaming是Hadoop提供的一种编程工具，允许用户用任何可执行程序和脚本作为mapper和reducer来完成Map/Reduce任务，这意味着你如果只是hadoop的一个轻度使用者，你完全可以用Hadoop Streaming+Python/Ruby/Golang/C艹 等任何你熟悉的语言来完成你的大数据探索需求，又
大数据MapReduce实现IP定位
最新发布
05-17
本主题聚焦于利用MapReduce实现IP定位，这是一个非常实用的功能，通常用于网络监控、广告定向、用户行为分析等多种场景。通过MapReduce，我们可以高效地处理大量IP地址，将其转换为对应地理位置的信息。 MapReduce...
大数据MapReduce实现Word Count
05-16
本教程将详细讲解如何利用MapReduce实现Word Count，这是一个经典的例子，用于展示分布式计算的基本原理。 MapReduce由两个主要阶段组成：Map阶段和Reduce阶段。在Word Count案例中，我们的目标是统计文本中每个...
mapreduce实现文档数据去重
02-11
MapReduce实现文档数据去重 MapReduce是一种编程模型和相关实现，用于大规模数据处理，广泛应用于大数据处理领域。本文将介绍如何使用MapReduce实现文档数据去重，通过环境配置、编程、测试等步骤，实现文档数据...
用MapReduce实现KMeans算法
04-22
使用Java编程语言实现MapReduce任务，需要继承`org.apache.hadoop.mapreduce.Mapper`和`org.apache.hadoop.mapreduce.Reducer`类，重写`map()`和`reduce()`方法。同时，需要利用Hadoop的API进行数据读写。 通过...
一个MapReduce简单程序示例
05-28
一个MapReduce简单程序示例 MapReduce hadoop
运行Mapreduce，运行一半卡着不动了
weixin_43872683的博客
01-18 1517
@[root@master mr_inverted_test]# bash run.sh rmr: DEPRECATED: Please use ‘rm -r’ instead. 19/01/17 23:22:23 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emp...
python hadoop streaming_Python Hadoop Streaming Error“错误streaming.StreamJob：作业不成功！”和堆栈跟踪：ExitCodeExc...
weixin_39527879的博客
12-18 790
I am trying to run python script on Hadoop cluster using Hadoop Streaming for sentiment analysis.The Same script I am running on Local machine which is running Properly and giving output.to run on loc...
MapReduce的简单代码实现
yuechu4492的博客
04-26 1032
需求：统计某个文档里面每个城市访问该网站的访客数
mapreduce的原理和简单程序实现
qq_42982169的博客
10-18 900
五个关于mapreduce的简单程序实现 mapreduce的简介 什么是mapreduce？ 是一种分布式运算程序 一个mapreduce程序会分成两个阶段，map阶段和reduce阶段 map阶段会有一个实体程序，不用用户自己开发 reduce阶段也会有一个实体程序，不用用户自己开发 用户只需要开发map程序和reduce程序所要调用的数据处理逻辑方法 Map阶段的逻辑方法：xxxMapper...
Mapreduce的简单实现和步骤
weixin_42868529的博客
09-20 938
package com.qfedu.bigdata.MR; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apac...
Hadoop Streaming详解
端木胥的博客
09-22 1088
刚来公司的时候就发现大家都在使用Hadoop Streaming 来执行任务，当时还在想名字里有Streaming，Hadoop也拿来做实时计算工具了吗？后来发现原来只是我学得太少，对编程工具的理解太片面了。 那么接下来就来看看什么是Hadoop Streaming，为什么要使用Hadoop Streaming、Hadoop Streaming的原理以及怎么使用Hadoop Streaming。 ...
MapReduce编程实现wordcount
05-16
Word count是MapReduce中最经典的一个例子，下面是一个简单的MapReduce编程实现word count的示例。 首先，我们需要定义Map函数和Reduce函数。 Map函数将输入数据中的每个单词转换成键值对(key,value)的形式，其中...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值