MapRedcue的演示(协同过滤)
做一个关于电影推荐。你于你好友之间的浏览电影以及电影评分的推荐的协同过滤。
百度百科：
协同过滤简单来说是利用某兴趣相投、拥有共同经验之群体的喜好来推荐用户感兴趣的信息，个人通过合作的机制给予信息相当程度的回应(如评分)并记录下来以达到过滤的目的进而帮助别人筛选信息，回应不一定局限于特别感兴趣的，特别不感兴趣信息的纪录也相当重要。
协同过滤又可分为评比(rating)或者群体过滤(social filtering)协同过滤以其出色的速度和健壮性，在全球互联网领域炙手可热。
元数据：1盗梦空间4.0
1碟中谍1.5
1禁闭岛4.0
2王的盛宴2.5
2盗梦空间5.0
2肖申克的救赎3.0
2禁闭岛2.5
3教父3.5
3盗梦空间5.0
30074.5
3星球大战5.0
3华尔街之狼2.5
3禁闭岛1.5
3碟中谍2.0
4碟中谍2.5
4禁闭岛1.5
40072.0
5沉默的羔羊4.5
5肖申克的救赎5.0
5碟中谍3.0
6沉默的羔羊5.0
6碟中谍4.0
6王的盛宴4.0
7王的盛宴5.0
7碟中谍1.5
7肖申克的救赎3.5
7华尔街之狼2.5
8禁闭岛3.0
8盗梦空间4.0
8教父2.5
8星球大战3.5
8肖申克的救赎3.0
8王的盛宴3.0
8华尔街之狼2.0
9沉默的羔羊5.0
9禁闭岛4.0
9教父5.0
9盗梦空间4.5
9星球大战4.0
9王的盛宴1.5
9华尔街之狼3.0
10盗梦空间3.5
10沉默的羔羊5.0
10禁闭岛2.5
10肖申克的救赎2.0
10王的盛宴4.0
10教父2.0
10碟中谍1.5
11禁闭岛3.0
11碟中谍2.5
11盗梦空间3.0
11肖申克的救赎3.0
12华尔街之狼3.0
12星球大战3.5
12碟中谍1.5
13教父3.5
13肖申克的救赎2.5
13华尔街之狼5.0
13星球大战5.0
13沉默的羔羊1.5
13盗梦空间3.0
13禁闭岛3.0
140074.0
14星球大战2.5
14华尔街之狼1.5
14教父4.5
14盗梦空间3.0
14沉默的羔羊3.5
15沉默的羔羊4.0
15华尔街之狼2.5
15肖申克的救赎5.0
16肖申克的救赎2.0
160075.0
16盗梦空间3.5
17星球大战4.5
17禁闭岛1.5
170074.5
180074.5
18华尔街之狼5.0
18沉默的羔羊1.5
18盗梦空间2.0
19星球大战4.5
19华尔街之狼3.0
19肖申克的救赎5.0
190072.0
19王的盛宴4.0
19碟中谍2.5
19沉默的羔羊3.0
200072.0
20教父4.0
20星球大战2.5
20盗梦空间4.5
20华尔街之狼3.0
20碟中谍4.5
20肖申克的救赎3.0
20禁闭岛2.0
21王的盛宴2.0
21碟中谍2.5
21禁闭岛2.5
21盗梦空间1.5
21肖申克的救赎4.5
22沉默的羔羊2.0
22教父4.0
22肖申克的救赎3.5
22王的盛宴1.5
22禁闭岛1.5
23盗梦空间3.5
23华尔街之狼4.0
230072.0
23肖申克的救赎4.5
240074.0
24华尔街之狼5.0
24教父1.5
24禁闭岛1.5
25王的盛宴3.0
25星球大战2.0
25沉默的羔羊5.0
25禁闭岛2.0
260072.0
26肖申克的救赎3.5
26星球大战4.5
26教父4.5
27沉默的羔羊5.0
27禁闭岛1.5
27肖申克的救赎5.0
280075.0
28星球大战5.0
28盗梦空间3.0
28王的盛宴4.0
28沉默的羔羊2.0
28教父2.5
28华尔街之狼5.0
28肖申克的救赎4.0
29肖申克的救赎3.0
29盗梦空间3.0
29星球大战3.5
29王的盛宴5.0
29碟中谍3.5
29禁闭岛1.5
30沉默的羔羊4.5
30星球大战1.5
30教父1.5
31盗梦空间3.0
31肖申克的救赎4.0
31王的盛宴3.0
32碟中谍2.0
32禁闭岛2.5
32盗梦空间3.0
33禁闭岛5.0
33教父3.0
33肖申克的救赎4.5
33华尔街之狼4.5
33盗梦空间4.0
34星球大战2.0
34沉默的羔羊3.0
340075.0
34禁闭岛2.0
35华尔街之狼4.5
350071.5
35盗梦空间3.5
35星球大战1.5
35教父2.5
36碟中谍2.0
36肖申克的救赎4.0
36教父1.5
36王的盛宴5.0
37肖申克的救赎2.0
37沉默的羔羊4.0
37王的盛宴2.5
37盗梦空间5.0
37教父2.5
38华尔街之狼1.5
38星球大战4.0
38王的盛宴3.0
390073.5
39教父2.0
39盗梦空间3.5
39王的盛宴3.5
40华尔街之狼3.0
40沉默的羔羊4.5
40盗梦空间5.0
400072.5
40碟中谍3.5
40星球大战1.5
40教父3.0
40王的盛宴2.0
41教父2.5
41禁闭岛4.5
410071.5
41沉默的羔羊1.5
41肖申克的救赎2.0
41盗梦空间3.0
41星球大战4.0
42华尔街之狼1.5
42王的盛宴1.5
42教父4.0
43华尔街之狼3.5
43教父5.0
43碟中谍4.5
44沉默的羔羊5.0
44教父4.5
44肖申克的救赎4.0
44盗梦空间2.5
44碟中谍4.5
44星球大战1.5
44王的盛宴5.0
45华尔街之狼3.0
45王的盛宴4.5
45禁闭岛2.0
46王的盛宴2.5
46盗梦空间4.0
46星球大战4.5
460072.0
46教父1.5
47教父2.5
47华尔街之狼3.0
470075.0
47碟中谍1.5
47禁闭岛4.0
48星球大战5.0
48教父4.5
48盗梦空间2.5
49沉默的羔羊4.0
49肖申克的救赎5.0
49王的盛宴2.5
49星球大战1.5
49碟中谍2.0
49华尔街之狼4.5
49盗梦空间4.5
50盗梦空间2.0
50禁闭岛1.5
50沉默的羔羊2.0
思路：
step1：过滤得到每个用户看过的所有电影
输出：key:用户1 value:{ 1 盗梦空间:4.0,碟中谍:1.5,禁闭岛:4.0}1盗梦空间:4.0,碟中谍:1.5,禁闭岛:4.0
2王的盛宴:2.5,盗梦空间:5.0,肖申克的救赎:3.0,禁闭岛:2.5
3碟中谍:2.0,教父:3.5,盗梦空间:5.0,007:4.5,星球大战:5.0,华尔街之狼:2.5,禁闭岛:1.5
4碟中谍:2.5,禁闭岛:1.5,007:2.0
5沉默的羔羊:4.5,肖申克的救赎:5.0,碟中谍:3.0
6沉默的羔羊:5.0,碟中谍:4.0,王的盛宴:4.0
7王的盛宴:5.0,碟中谍:1.5,肖申克的救赎:3.5,华尔街之狼:2.5
8禁闭岛:3.0,盗梦空间:4.0,教父:2.5,星球大战:3.5,肖申克的救赎:3.0,王的盛宴:3.0,华尔街之狼:2.0
9禁闭岛:4.0,教父:5.0,盗梦空间:4.5,沉默的羔羊:5.0,星球大战:4.0,王的盛宴:1.5,华尔街之狼:3.0
10肖申克的救赎:2.0,盗梦空间:3.5,沉默的羔羊:5.0,禁闭岛:2.5,王的盛宴:4.0,教父:2.0,碟中谍:1.5
11禁闭岛:3.0,碟中谍:2.5,盗梦空间:3.0,肖申克的救赎:3.0
12华尔街之狼:3.0,星球大战:3.5,碟中谍:1.5
13华尔街之狼:5.0,教父:3.5,肖申克的救赎:2.5,星球大战:5.0,沉默的羔羊:1.5,盗梦空间:3.0,禁闭岛:3.0
14星球大战:2.5,007:4.0,华尔街之狼:1.5,教父:4.5,盗梦空间:3.0,沉默的羔羊:3.5
15沉默的羔羊:4.0,华尔街之狼:2.5,肖申克的救赎:5.0
16肖申克的救赎:2.0,007:5.0,盗梦空间:3.5
17禁闭岛:1.5,星球大战:4.5,007:4.5
18007:4.5,华尔街之狼:5.0,沉默的羔羊:1.5,盗梦空间:2.0
19星球大战:4.5,华尔街之狼:3.0,肖申克的救赎:5.0,007:2.0,王的盛宴:4.0,碟中谍:2.5,沉默的羔羊:3.0
20007:2.0,教父:4.0,星球大战:2.5,盗梦空间:4.5,华尔街之狼:3.0,碟中谍:4.5,肖申克的救赎:3.0,禁闭岛:2.0
21碟中谍:2.5,禁闭岛:2.5,盗梦空间:1.5,肖申克的救赎:4.5,王的盛宴:2.0
22禁闭岛:1.5,沉默的羔羊:2.0,教父:4.0,肖申克的救赎:3.5,王的盛宴:1.5
23盗梦空间:3.5,华尔街之狼:4.0,007:2.0,肖申克的救赎:4.5
24007:4.0,华尔街之狼:5.0,教父:1.5,禁闭岛:1.5
25星球大战:2.0,禁闭岛:2.0,沉默的羔羊:5.0,王的盛宴:3.0
26肖申克的救赎:3.5,星球大战:4.5,教父:4.5,007:2.0
27沉默的羔羊:5.0,禁闭岛:1.5,肖申克的救赎:5.0
28华尔街之狼:5.0,007:5.0,星球大战:5.0,盗梦空间:3.0,王的盛宴:4.0,沉默的羔羊:2.0,教父:2.5,肖申克的救赎:4.0
29肖申克的救赎:3.0,盗梦空间:3.0,星球大战:3.5,王的盛宴:5.0,碟中谍:3.5,禁闭岛:1.5
30沉默的羔羊:4.5,星球大战:1.5,教父:1.5
31盗梦空间:3.0,肖申克的救赎:4.0,王的盛宴:3.0
32禁闭岛:2.5,碟中谍:2.0,盗梦空间:3.0
33教父:3.0,肖申克的救赎:4.5,华尔街之狼:4.5,盗梦空间:4.0,禁闭岛:5.0
34星球大战:2.0,沉默的羔羊:3.0,007:5.0,禁闭岛:2.0
35教父:2.5,华尔街之狼:4.5,007:1.5,盗梦空间:3.5,星球大战:1.5
36碟中谍:2.0,肖申克的救赎:4.0,教父:1.5,王的盛宴:5.0
37肖申克的救赎:2.0,沉默的羔羊:4.0,王的盛宴:2.5,盗梦空间:5.0,教父:2.5
38华尔街之狼:1.5,星球大战:4.0,王的盛宴:3.0
39教父:2.0,007:3.5,盗梦空间:3.5,王的盛宴:3.5
40盗梦空间:5.0,007:2.5,沉默的羔羊:4.5,碟中谍:3.5,星球大战:1.5,教父:3.0,华尔街之狼:3.0,王的盛宴:2.0
41007:1.5,教父:2.5,禁闭岛:4.5,沉默的羔羊:1.5,肖申克的救赎:2.0,盗梦空间:3.0,星球大战:4.0
42华尔街之狼:1.5,王的盛宴:1.5,教父:4.0
43华尔街之狼:3.5,教父:5.0,碟中谍:4.5
44王的盛宴:5.0,沉默的羔羊:5.0,教父:4.5,肖申克的救赎:4.0,盗梦空间:2.5,碟中谍:4.5,星球大战:1.5
45华尔街之狼:3.0,禁闭岛:2.0,王的盛宴:4.5
46007:2.0,王的盛宴:2.5,盗梦空间:4.0,星球大战:4.5,教父:1.5
47教父:2.5,华尔街之狼:3.0,007:5.0,碟中谍:1.5,禁闭岛:4.0
48星球大战:5.0,教父:4.5,盗梦空间:2.5
49沉默的羔羊:4.0,肖申克的救赎:5.0,王的盛宴:2.5,星球大战:1.5,碟中谍:2.0,华尔街之狼:4.5,盗梦空间:4.5
50盗梦空间:2.0,禁闭岛:1.5,沉默的羔羊:2.0
step2: 根据 step-out 通过所有用户看过的电影矩阵
输出：007:007 19(电影出现的次数)007:00719
007:华尔街之狼11
007:教父12
007:星球大战12
007:沉默的羔羊7
007:王的盛宴5
007:盗梦空间12
007:碟中谍6
007:禁闭岛8
007:肖申克的救赎7
华尔街之狼:00711
华尔街之狼:华尔街之狼23
华尔街之狼:教父14
华尔街之狼:星球大战13
华尔街之狼:沉默的羔羊9
华尔街之狼:王的盛宴10
华尔街之狼:盗梦空间13
华尔街之狼:碟中谍9
华尔街之狼:禁闭岛9
华尔街之狼:肖申克的救赎10
教父:00712
教父:华尔街之狼14
教父:教父25
教父:星球大战15
教父:沉默的羔羊11
教父:王的盛宴12
教父:盗梦空间17
教父:碟中谍8
教父:禁闭岛11
教父:肖申克的救赎12
星球大战:00712
星球大战:华尔街之狼13
星球大战:教父15
星球大战:星球大战23
星球大战:沉默的羔羊12
星球大战:王的盛宴11
星球大战:盗梦空间15
星球大战:碟中谍8
星球大战:禁闭岛10
星球大战:肖申克的救赎10
沉默的羔羊:0077
沉默的羔羊:华尔街之狼9
沉默的羔羊:教父11
沉默的羔羊:星球大战12
沉默的羔羊:沉默的羔羊21
沉默的羔羊:王的盛宴11
沉默的羔羊:盗梦空间12
沉默的羔羊:碟中谍7
沉默的羔羊:禁闭岛9
沉默的羔羊:肖申克的救赎12
王的盛宴:0075
王的盛宴:华尔街之狼10
王的盛宴:教父12
王的盛宴:星球大战11
王的盛宴:沉默的羔羊11
王的盛宴:王的盛宴23
王的盛宴:盗梦空间14
王的盛宴:碟中谍10
王的盛宴:禁闭岛9
王的盛宴:肖申克的救赎14
盗梦空间:00712
盗梦空间:华尔街之狼13
盗梦空间:教父17
盗梦空间:星球大战15
盗梦空间:沉默的羔羊12
盗梦空间:王的盛宴14
盗梦空间:盗梦空间29
盗梦空间:碟中谍11
盗梦空间:禁闭岛15
盗梦空间:肖申克的救赎17
碟中谍:0076
碟中谍:华尔街之狼9
碟中谍:教父8
碟中谍:星球大战8
碟中谍:沉默的羔羊7
碟中谍:王的盛宴10
碟中谍:盗梦空间11
碟中谍:碟中谍20
碟中谍:禁闭岛10
碟中谍:肖申克的救赎11
禁闭岛:0078
禁闭岛:华尔街之狼9
禁闭岛:教父11
禁闭岛:星球大战10
禁闭岛:沉默的羔羊9
禁闭岛:王的盛宴9
禁闭岛:盗梦空间15
禁闭岛:碟中谍10
禁闭岛:禁闭岛24
禁闭岛:肖申克的救赎12
肖申克的救赎:0077
肖申克的救赎:华尔街之狼10
肖申克的救赎:教父12
肖申克的救赎:星球大战10
肖申克的救赎:沉默的羔羊12
肖申克的救赎:王的盛宴14
肖申克的救赎:盗梦空间17
肖申克的救赎:碟中谍11
肖申克的救赎:禁闭岛12
肖申克的救赎:肖申克的救赎25
step3: 根据 step-out 用户评分矩阵
输出： 007 用户40：2.500740:2.5
00741:1.5
00735:1.5
00746:2.0
00717:4.5
0074:2.0
00723:2.0
00728:5.0
00747:5.0
00716:5.0
00719:2.0
00714:4.0
00718:4.5
00739:3.5
00724:4.0
0073:4.5
00726:2.0
00720:2.0
00734:5.0
华尔街之狼38:1.5
华尔街之狼7:2.5
华尔街之狼47:3.0
华尔街之狼40:3.0
华尔街之狼15:2.5
华尔街之狼23:4.0
华尔街之狼19:3.0
华尔街之狼24:5.0
华尔街之狼18:5.0
华尔街之狼49:4.5
华尔街之狼13:5.0
华尔街之狼28:5.0
华尔街之狼45:3.0
华尔街之狼12:3.0
华尔街之狼3:2.5
华尔街之狼9:3.0
华尔街之狼33:4.5 .。。。。。
step4: 根据step3-out和step4-out 计算推荐结果列表(计算电影的评分，根据电影出现的数据*用户对应得评分)
输入：007:007 19/007 40:2.5
输出：19 华尔街之狼,22.034教父,60.0
23教父,24.0
24教父,48.0
35教父,18.0
46教父,24.0
47教父,60.0
14教父,48.0
26教父,24.0
16教父,60.0
39教父,42.0
17教父,54.0
28教父,60.0
18教父,54.0
19教父,24.0
3教父,54.0
4教父,24.0
40教父,30.0
41教父,18.0
20教父,24.0
34碟中谍,30.0
23碟中谍,12.0
24碟中谍,24.0
35碟中谍,9.0
46碟中谍,12.0
47碟中谍,30.0
14碟中谍,24.0
26碟中谍,12.0
16碟中谍,30.0
39碟中谍,21.0
17碟中谍,27.0
28碟中谍,30.0
18碟中谍,27.0
19碟中谍,12.0
3碟中谍,27.0
4碟中谍,12.0
40碟中谍,15.0
41碟中谍,9.0
20碟中谍,12.0
34王的盛宴,25.0
23王的盛宴,10.0
24王的盛宴,20.0
.....
step5: 根据step-out4 合并所有数据
输出：19 华尔街之狼,289.01碟中谍,114.0
1教父,124.0
1肖申克的救赎,132.5
1王的盛宴,107.0
1盗梦空间,192.5
1禁闭岛,171.0
1007,89.0
1华尔街之狼,101.5
1沉默的羔羊,94.5
1星球大战,112.0
2碟中谍,138.0
2教父,178.5
2王的盛宴,192.0
2肖申克的救赎,225.0
2盗梦空间,268.5
2禁闭岛,193.5
2007,113.5
2华尔街之狼,142.5
2沉默的羔羊,146.0
2星球大战,157.5
3教父,369.0
3碟中谍,227.5
3王的盛宴,248.0
3肖申克的救赎,273.5
3盗梦空间,410.5
3禁闭岛,278.0
3007,299.0
3华尔街之狼,317.5
3沉默的羔羊,240.0
3星球大战,360.0
4教父,60.5
4碟中谍,77.0
4王的盛宴,48.5
4肖申克的救赎,59.5
4盗梦空间,74.0
.....
step6: 根据step5-out 排除用户看过得电影，然后把类似得，评分高得优先推荐(排序)。
输出： 1 肖申克的救赎 132.5 1 教父 124.0 1 星球大战 112.0 1 王的盛宴 107.0 1 华尔街之狼 101.5Movie [userid=1, movieName=肖申克的救赎, score=132.5]
Movie [userid=1, movieName=教父, score=124.0]
Movie [userid=1, movieName=星球大战, score=112.0]
Movie [userid=1, movieName=王的盛宴, score=107.0]
Movie [userid=1, movieName=华尔街之狼, score=101.5]
Movie [userid=1, movieName=沉默的羔羊, score=94.5]
Movie [userid=1, movieName=007, score=89.0]
Movie [userid=2, movieName=教父, score=178.5]
Movie [userid=2, movieName=星球大战, score=157.5]
Movie [userid=2, movieName=沉默的羔羊, score=146.0]
Movie [userid=2, movieName=华尔街之狼, score=142.5]
Movie [userid=2, movieName=碟中谍, score=138.0]
Movie [userid=2, movieName=007, score=113.5]
Movie [userid=3, movieName=肖申克的救赎, score=273.5]
Movie [userid=3, movieName=王的盛宴, score=248.0]
Movie [userid=3, movieName=沉默的羔羊, score=240.0]
Movie [userid=4, movieName=盗梦空间, score=74.0]
Movie [userid=4, movieName=教父, score=60.5]
Movie [userid=4, movieName=星球大战, score=59.0]
Movie [userid=4, movieName=华尔街之狼, score=58.0]
Movie [userid=4, movieName=王的盛宴, score=48.5]
Movie [userid=4, movieName=沉默的羔羊, score=45.0]
Movie [userid=5, movieName=盗梦空间, score=172.0]
Movie [userid=5, movieName=王的盛宴, score=149.5]
Movie [userid=5, movieName=教父, score=133.5]
Movie [userid=5, movieName=禁闭岛, score=130.5]
Movie [userid=5, movieName=星球大战, score=128.0]
Movie [userid=5, movieName=华尔街之狼, score=117.5]
Movie [userid=5, movieName=007, score=84.5]
Movie [userid=6, movieName=盗梦空间, score=160.0]
Movie [userid=6, movieName=肖申克的救赎, score=160.0]
Movie [userid=6, movieName=星球大战, score=136.0]
Movie [userid=6, movieName=教父, score=135.0]
Movie [userid=6, movieName=华尔街之狼, score=121.0]
Movie [userid=6, movieName=禁闭岛, score=121.0]
Movie [userid=6, movieName=007, score=79.0]
Movie [userid=7, movieName=盗梦空间, score=178.5]
Movie [userid=7, movieName=教父, score=149.0]
Movie [userid=7, movieName=星球大战, score=134.5]
Movie [userid=7, movieName=沉默的羔羊, score=130.0]
Movie [userid=7, movieName=禁闭岛, score=124.5]
Movie [userid=7, movieName=007, score=86.0]
Movie [userid=8, movieName=沉默的羔羊, score=231.5]
Movie [userid=8, movieName=碟中谍, score=203.0]
Movie [userid=8, movieName=007, score=202.0]
Movie [userid=9, movieName=肖申克的救赎, score=335.5]
Movie [userid=9, movieName=007, score=269.5]
Movie [userid=9, movieName=碟中谍, score=238.5]
Movie [userid=10, movieName=星球大战, score=243.5]
.....
代码：package com.huhu.day06;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.WritableComparable;
public class Movie implements WritableComparable {
private int userid;
private String movieName;
private float score;
public Movie() {
super();
}
public Movie(int userid, String movieName, float score) {
super();
this.userid = userid;
this.movieName = movieName;
this.score = score;
}
public int getUserid() {
return userid;
}
public void setUserid(int userid) {
this.userid = userid;
}
public String getMovieName() {
return movieName;
}
public void setMovieName(String movieName) {
this.movieName = movieName;
}
public float getScore() {
return score;
}
public void setScore(float score) {
this.score = score;
}
@Override
public int hashCode() {
final int prime = 31;
int result = 1;
result = prime * result + ((movieName == null) ? 0 : movieName.hashCode());
result = prime * result + Float.floatToIntBits(score);
result = prime * result + userid;
return result;
}
@Override
public boolean equals(Object obj) {
if (this == obj)
return true;
if (obj == null)
return false;
if (getClass() != obj.getClass())
return false;
Movie other = (Movie) obj;
if (movieName == null) {
if (other.movieName != null)
return false;
} else if (!movieName.equals(other.movieName))
return false;
if (Float.floatToIntBits(score) != Float.floatToIntBits(other.score))
return false;
if (userid != other.userid)
return false;
return true;
}
@Override
public String toString() {
return "Movie [userid=" + userid + ", movieName=" + movieName + ", score=" + score + "]";
}
@Override
public void readFields(DataInput in) throws IOException {
this.userid = in.readInt();
this.movieName = in.readUTF();
this.score = in.readFloat();
}
@Override
public void write(DataOutput out) throws IOException {
out.writeInt(userid);
out.writeUTF(movieName);
out.writeFloat(score);
}
@Override
public int compareTo(Movie o) {
if (this.userid == o.getUserid()) {
if (this.score == o.score) {
return this.movieName.compareTo(o.movieName);
} else {
return (int) (o.getScore() - this.score);
}
} else {
return this.userid - o.getUserid();
}
}
}
package com.huhu.day06;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
* 输入： 1 盗梦空间 4.0
*
* 输出 ： 1 盗梦空间:4.0,碟中谍:1.5,禁闭岛:4.0
*
* @author huhu_k
*
*/
public class Step1 {
static class MyMapper extends Mapper {
@Override
protected void map(LongWritable key, Text value, Mapper.Context context)
throws IOException, InterruptedException {
String[] line = value.toString().split("\t");
// k:用户 v: item电影名称：分数
context.write(new IntWritable(Integer.valueOf(line[0])), new Text(line[1] + ":" + line[2]));
}
}
// 相同key的数据相遇 k:1 v:{}
static class MyReduce extends Reducer {
Text va = new Text();
@Override
protected void reduce(IntWritable key, Iterable values, Context context)
throws IOException, InterruptedException {
StringBuffer br = new StringBuffer();
for (Text v : values) {
br.append("," + v.toString());
}
va.set(br.toString().replaceFirst(",", ""));
// k:用户id v：电影1：评分,电影2：评分.....
// 1 盗梦空间:4.0,碟中谍:1.5,禁闭岛:4.0
context.write(key, va);
}
}
public Job getJob(Configuration conf) throws Exception {
Path inpath = new Path("hdfs://ry-hadoop1:8020/in/items.txt");
Path outpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step1");
Job job = Job.getInstance(conf, this.getClass().getSimpleName());
job.setJarByClass(this.getClass());
job.setMapperClass(MyMapper.class);
job.setMapOutputKeyClass(IntWritable.class);
job.setMapOutputValueClass(Text.class);
job.setReducerClass(MyReduce.class);
job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(Text.class);
FileInputFormat.addInputPath(job, inpath);
FileOutputFormat.setOutputPath(job, outpath);
return job;
}
}
package com.huhu.day06;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
* 输入：1 盗梦空间:4.0,碟中谍:1.5,禁闭岛:4.0
*
* 输出 ：007 40:2.5
*
* @author huhu_k
*
*/
public class Step3 {
static class MyMapper extends Mapper {
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String[] line = value.toString().split("\t");
String userid = line[0];
String[] item = line[1].split(",");
for (String it : item) {
context.write(new Text(it.split(":")[0]), new Text(userid + ":"+new Text(it.split(":")[1])));
}
}
}
public Job getJob(Configuration conf) throws Exception {
Path inpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step1");
Path outpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step3");
Job job = Job.getInstance(conf, this.getClass().getSimpleName());
job.setJarByClass(Step3.class);
job.setMapperClass(MyMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
FileInputFormat.addInputPath(job, inpath);
FileOutputFormat.setOutputPath(job, outpath);
return job;
}
}
package com.huhu.day06;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
* 商品同现矩阵 输出 ：1 盗梦空间:4.0,碟中谍:1.5,禁闭岛:4.0
*
* 输出：007:007 19
*
* @author huhu_k
*
*/
public class Step2 {
static class MyMapper extends Mapper {
private Text k = new Text();
private final IntWritable one = new IntWritable(1);
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
//盗梦空间:4.0 碟中谍:1.5 禁闭岛:4.0
String[] line = value.toString().split("\t")[1].split(",");
System.err.println(line+"----------------------");
for (int i = 0; i < line.length; i++) {
for (int j = 0; j < line.length; j++) {
k.set(line[i].split(":")[0] + ":" + line[j].split(":")[0]);
// k: 盗梦空间:碟中谍 1 每个人看过所有电影的乘积
context.write(k, one);
}
}
}
}
static class MyReducer extends Reducer {
// k: 盗梦空间:碟中谍 1
@Override
protected void reduce(Text key, Iterable values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable v : values) {
sum += v.get();
}
// k:007:007 v:19
context.write(key, new IntWritable(sum));
}
}
public Job getJob(Configuration conf) throws Exception {
Path inpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step1");
Path outpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step2");
Job job = Job.getInstance(conf, this.getClass().getSimpleName());
job.setJarByClass(this.getClass());
job.setMapperClass(MyMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setReducerClass(MyReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, inpath);
FileOutputFormat.setOutputPath(job, outpath);
return job;
}
}package com.huhu.day06;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
* 输入：1 盗梦空间:4.0,碟中谍:1.5,禁闭岛:4.0
*
* 输出 ：007 40:2.5
*
* @author huhu_k
*
*/
public class Step3 {
static class MyMapper extends Mapper {
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String[] line = value.toString().split("\t");
String userid = line[0];
String[] item = line[1].split(",");
for (String it : item) {
context.write(new Text(it.split(":")[0]), new Text(userid + ":"+new Text(it.split(":")[1])));
}
}
}
public Job getJob(Configuration conf) throws Exception {
Path inpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step1");
Path outpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step3");
Job job = Job.getInstance(conf, this.getClass().getSimpleName());
job.setJarByClass(Step3.class);
job.setMapperClass(MyMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
FileInputFormat.addInputPath(job, inpath);
FileOutputFormat.setOutputPath(job, outpath);
return job;
}
}package com.huhu.day06;
import java.io.IOException;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.FileSplit;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
* step4: 计算推荐结果列表 输入：007:007 19 输入：007 40:2.5
*
* 输出：19 华尔街之狼,22.0
*
* @author huhu_k
*
*/
public class Step4 {
static class MyMapper extends Mapper {
String filename = "";
@Override
protected void setup(Context context) throws IOException, InterruptedException {
FileSplit inputSplit = (FileSplit) context.getInputSplit();
// 获取父文件名
filename = inputSplit.getPath().getParent().getName();
}
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String[] line = value.toString().split("\t");
if ("step2".equals(filename)) {
String[] v1 = line[0].split(":");
String item1 = v1[0];
String item2 = v1[1];
String num = line[1];
context.write(new Text(item1), new Text("A:" + item2 + "," + num));
} else if ("step3".equals(filename)) {
String item = line[0];
String userid = line[1].split(":")[0];
String score = line[1].split(":")[1];
context.write(new Text(item), new Text("B:" + userid + "," + score));
}
}
}
// 相同key的数据相遇 k:1 v:{}
static class MyReduce extends Reducer {
// k: 电影名称 v：A:007,20 来自于商品同现矩阵
// k: 电影名称 v：B:1,4 来自于用户评分矩阵
String[] info = null;
@Override
protected void reduce(Text key, Iterable values, Context context)
throws IOException, InterruptedException {
Map mapA = new HashMap<>();
Map mapB = new HashMap<>();
for (Text va : values) {
String src = va.toString();
if (src.startsWith("A:")) {
// A:电影名称，次数
info = src.substring(2).split(",");
// mapA k: 电影名称 v:次数
mapA.put(info[0], info[1]);
} else if (src.startsWith("B:")) {
// B:用户，评分
info = src.substring(2).split(",");
// mapB k: 用户 v:评分
mapB.put(info[0], info[1]);
}
}
float result = 0;
Iterator iterator = mapA.keySet().iterator();
while (iterator.hasNext()) {
String item = iterator.next();
int num = Integer.parseInt(mapA.get(item));
Iterator iterator2 = mapB.keySet().iterator();
while (iterator2.hasNext()) {
String userid = iterator2.next();
float score = Float.valueOf(mapB.get(userid));
result = score * num;
context.write(new Text(userid), new Text(item + "," + result));
}
}
}
}
public Job getJob(Configuration conf) throws Exception {
Path inpath1 = new Path("hdfs://ry-hadoop1:8020/out/day06/step2");
Path inpath2 = new Path("hdfs://ry-hadoop1:8020/out/day06/step3");
Path outpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step4");
Job job = Job.getInstance(conf, this.getClass().getSimpleName());
job.setJarByClass(Step4.class);
job.setMapperClass(MyMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
job.setReducerClass(MyReduce.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
FileInputFormat.addInputPath(job, inpath1);
FileInputFormat.addInputPath(job, inpath2);
FileOutputFormat.setOutputPath(job, outpath);
return job;
}
}package com.huhu.day06;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
* step5: 合并结果列表 输入：19 华尔街之狼,22.0
*
* 输出：19 华尔街之狼,229.0
*
* @author huhu_k
*
*/
public class Step5 {
static class MyMapper extends Mapper {
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String[] line = value.toString().split("\t");
context.write(new IntWritable(Integer.parseInt(line[0])), new Text(line[1]));
}
}
// 相同key的数据相遇 k:1 v:{}
static class MyReduce extends Reducer {
// k: 电影名称 v：A:007,20 来自于商品同现矩阵
// k: 电影名称 v：B:1,4 来自于用户评分矩阵
@Override
protected void reduce(IntWritable key, Iterable values, Context context)
throws IOException, InterruptedException {
Map map = new HashMap<>();
for (Text v : values) {
String[] str = v.toString().split(",");
String item = str[0];
float score = Float.valueOf(str[1]);
if (map.containsKey(item)) {
map.put(item, map.get(item) + score);
} else {
map.put(item, score);
}
}
for (Map.Entry m : map.entrySet()) {
context.write(key, new Text(m.getKey() + "," + m.getValue()));
}
}
}
public Job getJob(Configuration conf) throws Exception {
Path inpath1 = new Path("hdfs://ry-hadoop1:8020/out/day06/step4");
Path outpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step5");
Job job = Job.getInstance(conf, this.getClass().getSimpleName());
job.setJarByClass(Step5.class);
job.setMapperClass(MyMapper.class);
job.setMapOutputKeyClass(IntWritable.class);
job.setMapOutputValueClass(Text.class);
job.setReducerClass(MyReduce.class);
job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(Text.class);
FileInputFormat.addInputPath(job, inpath1);
FileOutputFormat.setOutputPath(job, outpath);
return job;
}
}package com.huhu.day06;
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.net.URI;
import java.util.HashMap;
import java.util.Map;
import java.util.TreeSet;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.filecache.DistributedCache;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
* 输入：排除 用户看过的电影，并且 排序输出 降序 输入：19 华尔街之狼,289.0
*
* 输出 ：1 肖申克的救赎 132.5 1 教父 124.0 1 星球大战 112.0 1 王的盛宴 107.0 1 华尔街之狼 101.5
*
* @author huhu_k
*
*/
public class Step6 {
static class MyMapper extends Mapper {
private Map map;
private Path[] localFiles;
@Override
protected void setup(Context context) throws IOException, InterruptedException {
map = new HashMap<>();
Configuration conf = context.getConfiguration();
localFiles = DistributedCache.getLocalCacheFiles(conf);
for (Path p : localFiles) {
BufferedReader br = new BufferedReader(new FileReader(p.toString()));
String word = "";
while ((word = br.readLine()) != null) {
String[] s = word.split("\t");
int userid = Integer.parseInt(s[0]);
String item = s[1];
if (map.containsKey(userid)) {
// 1 007;008
map.put(userid, map.get(userid) + ";" + item);
} else {
// 1 007
map.put(userid, item);
}
}
br.close();
}
}
@Override
protected void map(LongWritable key, Text value, Mapper.Context context)
throws IOException, InterruptedException {
String[] line = value.toString().split("\t");
// 19 华尔街之狼,289.0
int userid = Integer.valueOf(line[0]);
String movie = line[1].split(",")[0];
String movieList = map.get(userid);
// map得到null 第一次来 没有任何数据
if (movieList == null || movieList.length() == 0 || movieList == "" || !movieList.contains(movie)) {
context.write(new IntWritable(userid), new Text(line[1]));
}
}
}
// 相同key的数据相遇 k:1 v:{}
static class MyReduce extends Reducer {
private TreeSet set = new TreeSet<>();
@Override
protected void reduce(IntWritable key, Iterable values, Context context)
throws IOException, InterruptedException {
for (Text v : values) {
Movie m = new Movie(Integer.parseInt(key.toString()), v.toString().split(",")[0],
Float.parseFloat(v.toString().split(",")[1]));
set.add(m);
}
}
@Override
protected void cleanup(Context context) throws IOException, InterruptedException {
for (Movie m : set) {
context.write(m, NullWritable.get());
}
}
}
public Job getJob(Configuration conf) throws Exception {
Path inpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step5");
Path outpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step6");
DistributedCache.addCacheFile(new URI("hdfs://ry-hadoop1:8020/in/items.txt"), conf);
Job job = Job.getInstance(conf, this.getClass().getSimpleName());
job.setJarByClass(Step6.class);
job.setMapperClass(MyMapper.class);
job.setMapOutputKeyClass(IntWritable.class);
job.setMapOutputValueClass(Text.class);
job.setReducerClass(MyReduce.class);
job.setOutputKeyClass(Movie.class);
job.setOutputValueClass(NullWritable.class);
FileInputFormat.addInputPath(job, inpath);
FileOutputFormat.setOutputPath(job, outpath);
return job;
}
}
package com.huhu.day06;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class Movie_App extends ToolRunner implements Tool {
private Configuration con;
public static void main(String[] args) throws Exception {
Movie_App m = new Movie_App();
int num = ToolRunner.run(m.getConf(), m, args);
System.exit(num);
}
@Override
public Configuration getConf() {
if (con == null) {
return new Configuration();
}
return con;
}
@Override
public void setConf(Configuration arg0) {
}
@Override
public int run(String[] arg0) throws Exception {
Step1 step1 = new Step1();
Step2 step2 = new Step2();
Step3 step3 = new Step3();
Step4 step4 = new Step4();
Step5 step5 = new Step5();
Step6 step6 = new Step6();
ControlledJob controlledJob1 = new ControlledJob(step1.getJob(getConf()).getConfiguration());
ControlledJob controlledJob2 = new ControlledJob(step2.getJob(getConf()).getConfiguration());
ControlledJob controlledJob3 = new ControlledJob(step3.getJob(getConf()).getConfiguration());
ControlledJob controlledJob4 = new ControlledJob(step4.getJob(getConf()).getConfiguration());
ControlledJob controlledJob5 = new ControlledJob(step5.getJob(getConf()).getConfiguration());
ControlledJob controlledJob6 = new ControlledJob(step6.getJob(getConf()).getConfiguration());
controlledJob2.addDependingJob(controlledJob1);
controlledJob3.addDependingJob(controlledJob1);
controlledJob4.addDependingJob(controlledJob2);
controlledJob4.addDependingJob(controlledJob3);
controlledJob5.addDependingJob(controlledJob4);
controlledJob6.addDependingJob(controlledJob5);
JobControl jobControl = new JobControl("Movive");
jobControl.addJob(controlledJob1);
jobControl.addJob(controlledJob2);
jobControl.addJob(controlledJob3);
jobControl.addJob(controlledJob4);
jobControl.addJob(controlledJob5);
jobControl.addJob(controlledJob6);
Thread t = new Thread(jobControl);
t.start();
while (!jobControl.allFinished()) {
t.sleep(1000);
}
jobControl.stop();
return 0;
}
}
关于什么是矩阵：我看了两篇比较好得，推荐给大家。
87
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
