python版mapreduce题目实现寻找共同好友

看到一篇不知道是好好玩还是好玩玩童鞋的博客，发现一道好玩的mapreduce题目，地址http://www.cnblogs.com/songhaowan/p/7239578.html

如图

 

由于自己太笨，看到一大堆java代码就头晕、心慌，所以用python把这个题目研究了一下。

 

---

题目：寻找共同好友。比如A的好友中有C，B的好友中有C，那么C就是AB的共同好友。

A:B,C,D,F,E,O

B:A,C,E,K

C:F,A,D,I

D:A,E,F,L

E:B,C,D,M,L

F:A,B,C,D,E,O,M

G:A,C,D,E,F

H:A,C,D,E,O

I:A,O

J:B,O

K:A,C,D

L:D,E,F

M:E,F,G

O:A,H,I,J

 

---

 

m.py

#-*-encoding:utf-8-*-
#!/home/hadoop/anaconda2/bin/python
import sys
result = {}
for line in sys.stdin:
    line = line.strip()
    if len(line)==0:
        continue
    key,vals = line.split(':')
    val = vals.split(',')
    result[key] = val
    if len(result)==1:
        continue
    else:
        for i in result[key]:
            for j in result:
                if i in result[j]:
                    if j<key:
                        print j+key,i
                    elif j>key:
                        print key+j,i

r.py

#-*-encoding:utf-8-*-
import sys
result = {}
for line in sys.stdin:
    line = line.strip()
    k,v = line.split(' ')
    if k in result:
        result[k].append(v)
    else:
        result[k] = [v]
for key,val in result.items():
    print key,val

执行的命令

hadoop jar /home/hadoop/hadoop-2.7.2/hadoop-streaming-2.7.2.jar \
-files /home/hadoop/test/m.py,/home/hadoop/test/r.py \
-input GTHY -output GTHYout \
-mapper 'python m.py' -reducer 'python r.py'

---

执行情况

packageJobJar: [/tmp/hadoop-unjar2310332345933071298/] [] /tmp/streamjob8006362102585628853.jar tmpDir=null
17/08/31 14:47:59 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.228.200:18040
17/08/31 14:48:00 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.228.200:18040
17/08/31 14:48:00 INFO mapred.FileInputFormat: Total input paths to process : 1
17/08/31 14:48:00 INFO mapreduce.JobSubmitter: number of splits:2
17/08/31 14:48:01 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504148710826_0003
17/08/31 14:48:01 INFO impl.YarnClientImpl: Submitted application application_1504148710826_0003
17/08/31 14:48:01 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1504148710826_0003/
17/08/31 14:48:01 INFO mapreduce.Job: Running job: job_1504148710826_0003
17/08/31 14:48:08 INFO mapreduce.Job: Job job_1504148710826_0003 running in uber mode : false
17/08/31 14:48:08 INFO mapreduce.Job:  map 0% reduce 0%
17/08/31 14:48:16 INFO mapreduce.Job:  map 100% reduce 0%
17/08/31 14:48:21 INFO mapreduce.Job:  map 100% reduce 100%
17/08/31 14:48:21 INFO mapreduce.Job: Job job_1504148710826_0003 completed successfully
17/08/31 14:48:21 INFO mapreduce.Job: Counters: 49
    File System Counters
        FILE: Number of bytes read=558
        FILE: Number of bytes written=362357
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=462
        HDFS: Number of bytes written=510
        HDFS: Number of read operations=9
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=2
        Launched reduce tasks=1
        Data-local map tasks=2
        Total time spent by all maps in occupied slots (ms)=11376
        Total time spent by all reduces in occupied slots (ms)=2888
        Total time spent by all map tasks (ms)=11376
        Total time spent by all reduce tasks (ms)=2888
        Total vcore-milliseconds taken by all map tasks=11376
        Total vcore-milliseconds taken by all reduce tasks=2888
        Total megabyte-milliseconds taken by all map tasks=11649024
        Total megabyte-milliseconds taken by all reduce tasks=2957312
    Map-Reduce Framework
        Map input records=27
        Map output records=69
        Map output bytes=414
        Map output materialized bytes=564
        Input split bytes=192
        Combine input records=0
        Combine output records=0
        Reduce input groups=69
        Reduce shuffle bytes=564
        Reduce input records=69
        Reduce output records=33
        Spilled Records=138
        Shuffled Maps =2
        Failed Shuffles=0
        Merged Map outputs=2
        GC time elapsed (ms)=421
        CPU time spent (ms)=2890
        Physical memory (bytes) snapshot=709611520
        Virtual memory (bytes) snapshot=5725220864
        Total committed heap usage (bytes)=487063552
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=270
    File Output Format Counters 
        Bytes Written=510
17/08/31 14:48:21 INFO streaming.StreamJob: Output directory: GTHYout

最终结果

hadoop@master:~/test$ hadoop fs -text GTHYout/part-00000
BD ['A', 'E']    
BE ['C']    
BF ['A', 'C', 'E']    
BG ['A', 'C', 'E']    
BC ['A']    
DF ['A', 'E']    
DG ['A', 'E', 'F']    
DE ['L']    
HJ ['O']    
HK ['A', 'C', 'D']    
HI ['A', 'O']    
HO ['A']    
HL ['D', 'E']    
FG ['A', 'C', 'D', 'E']    
LM ['E', 'F']    
KO ['A']    
AC ['D', 'F']    
AB ['C', 'E']    
AE ['B', 'C', 'D']    
AD ['E', 'F']    
AG ['C', 'D', 'E', 'F']    
AF ['B', 'C', 'D', 'E', 'O']    
EG ['C', 'D']    
EF ['B', 'C', 'D', 'M']    
CG ['A', 'D', 'F']    
CF ['A', 'D']    
CE ['D']    
CD ['A', 'F']    
IK ['A']    
IJ ['O']    
IO ['A']    
HM ['E']    
KL ['D']    

突然发现代码中居然一句注释都没有。果然自己还是太辣鸡，还没养成好习惯。

由于刚接触大数据不久，对java不熟悉，摸索地很慢。希望python的轻便能助我在大数据的世界探索更多。

有错的地方还请大佬多多指出~

转载于:https://www.cnblogs.com/zlsjjn/p/7458511.html