MapRedcue的演示(协同过滤)
做一个关于电影推荐。你于你好友之间的浏览电影以及电影评分的推荐的协同过滤。
百度百科:
协同过滤简单来说是利用某兴趣相投、拥有共同经验之群体的喜好来推荐用户感兴趣的信息,个人通过合作的机制给予信息相当程度的回应(如评分)并记录下来以达到过滤的目的进而帮助别人筛选信息,回应不一定局限于特别感兴趣的,特别不感兴趣信息的纪录也相当重要。
协同过滤又可分为评比(rating)或者群体过滤(social filtering)协同过滤以其出色的速度和健壮性,在全球互联网领域炙手可热。
元数据:1盗梦空间4.0
1碟中谍1.5
1禁闭岛4.0
2王的盛宴2.5
2盗梦空间5.0
2肖申克的救赎3.0
2禁闭岛2.5
3教父3.5
3盗梦空间5.0
30074.5
3星球大战5.0
3华尔街之狼2.5
3禁闭岛1.5
3碟中谍2.0
4碟中谍2.5
4禁闭岛1.5
40072.0
5沉默的羔羊4.5
5肖申克的救赎5.0
5碟中谍3.0
6沉默的羔羊5.0
6碟中谍4.0
6王的盛宴4.0
7王的盛宴5.0
7碟中谍1.5
7肖申克的救赎3.5
7华尔街之狼2.5
8禁闭岛3.0
8盗梦空间4.0
8教父2.5
8星球大战3.5
8肖申克的救赎3.0
8王的盛宴3.0
8华尔街之狼2.0
9沉默的羔羊5.0
9禁闭岛4.0
9教父5.0
9盗梦空间4.5
9星球大战4.0
9王的盛宴1.5
9华尔街之狼3.0
10盗梦空间3.5
10沉默的羔羊5.0
10禁闭岛2.5
10肖申克的救赎2.0
10王的盛宴4.0
10教父2.0
10碟中谍1.5
11禁闭岛3.0
11碟中谍2.5
11盗梦空间3.0
11肖申克的救赎3.0
12华尔街之狼3.0
12星球大战3.5
12碟中谍1.5
13教父3.5
13肖申克的救赎2.5
13华尔街之狼5.0
13星球大战5.0
13沉默的羔羊1.5
13盗梦空间3.0
13禁闭岛3.0
140074.0
14星球大战2.5
14华尔街之狼1.5
14教父4.5
14盗梦空间3.0
14沉默的羔羊3.5
15沉默的羔羊4.0
15华尔街之狼2.5
15肖申克的救赎5.0
16肖申克的救赎2.0
160075.0
16盗梦空间3.5
17星球大战4.5
17禁闭岛1.5
170074.5
180074.5
18华尔街之狼5.0
18沉默的羔羊1.5
18盗梦空间2.0
19星球大战4.5
19华尔街之狼3.0
19肖申克的救赎5.0
190072.0
19王的盛宴4.0
19碟中谍2.5
19沉默的羔羊3.0
200072.0
20教父4.0
20星球大战2.5
20盗梦空间4.5
20华尔街之狼3.0
20碟中谍4.5
20肖申克的救赎3.0
20禁闭岛2.0
21王的盛宴2.0
21碟中谍2.5
21禁闭岛2.5
21盗梦空间1.5
21肖申克的救赎4.5
22沉默的羔羊2.0
22教父4.0
22肖申克的救赎3.5
22王的盛宴1.5
22禁闭岛1.5
23盗梦空间3.5
23华尔街之狼4.0
230072.0
23肖申克的救赎4.5
240074.0
24华尔街之狼5.0
24教父1.5
24禁闭岛1.5
25王的盛宴3.0
25星球大战2.0
25沉默的羔羊5.0
25禁闭岛2.0
260072.0
26肖申克的救赎3.5
26星球大战4.5
26教父4.5
27沉默的羔羊5.0
27禁闭岛1.5
27肖申克的救赎5.0
280075.0
28星球大战5.0
28盗梦空间3.0
28王的盛宴4.0
28沉默的羔羊2.0
28教父2.5
28华尔街之狼5.0
28肖申克的救赎4.0
29肖申克的救赎3.0
29盗梦空间3.0
29星球大战3.5
29王的盛宴5.0
29碟中谍3.5
29禁闭岛1.5
30沉默的羔羊4.5
30星球大战1.5
30教父1.5
31盗梦空间3.0
31肖申克的救赎4.0
31王的盛宴3.0
32碟中谍2.0
32禁闭岛2.5
32盗梦空间3.0
33禁闭岛5.0
33教父3.0
33肖申克的救赎4.5
33华尔街之狼4.5
33盗梦空间4.0
34星球大战2.0
34沉默的羔羊3.0
340075.0
34禁闭岛2.0
35华尔街之狼4.5
350071.5
35盗梦空间3.5
35星球大战1.5
35教父2.5
36碟中谍2.0
36肖申克的救赎4.0
36教父1.5
36王的盛宴5.0
37肖申克的救赎2.0
37沉默的羔羊4.0
37王的盛宴2.5
37盗梦空间5.0
37教父2.5
38华尔街之狼1.5
38星球大战4.0
38王的盛宴3.0
390073.5
39教父2.0
39盗梦空间3.5
39王的盛宴3.5
40华尔街之狼3.0
40沉默的羔羊4.5
40盗梦空间5.0
400072.5
40碟中谍3.5
40星球大战1.5
40教父3.0
40王的盛宴2.0
41教父2.5
41禁闭岛4.5
410071.5
41沉默的羔羊1.5
41肖申克的救赎2.0
41盗梦空间3.0
41星球大战4.0
42华尔街之狼1.5
42王的盛宴1.5
42教父4.0
43华尔街之狼3.5
43教父5.0
43碟中谍4.5
44沉默的羔羊5.0
44教父4.5
44肖申克的救赎4.0
44盗梦空间2.5
44碟中谍4.5
44星球大战1.5
44王的盛宴5.0
45华尔街之狼3.0
45王的盛宴4.5
45禁闭岛2.0
46王的盛宴2.5
46盗梦空间4.0
46星球大战4.5
460072.0
46教父1.5
47教父2.5
47华尔街之狼3.0
470075.0
47碟中谍1.5
47禁闭岛4.0
48星球大战5.0
48教父4.5
48盗梦空间2.5
49沉默的羔羊4.0
49肖申克的救赎5.0
49王的盛宴2.5
49星球大战1.5
49碟中谍2.0
49华尔街之狼4.5
49盗梦空间4.5
50盗梦空间2.0
50禁闭岛1.5
50沉默的羔羊2.0
思路:
step1:过滤得到每个用户看过的所有电影
输出:key:用户1 value:{ 1 盗梦空间:4.0,碟中谍:1.5,禁闭岛:4.0}1盗梦空间:4.0,碟中谍:1.5,禁闭岛:4.0
2王的盛宴:2.5,盗梦空间:5.0,肖申克的救赎:3.0,禁闭岛:2.5
3碟中谍:2.0,教父:3.5,盗梦空间:5.0,007:4.5,星球大战:5.0,华尔街之狼:2.5,禁闭岛:1.5
4碟中谍:2.5,禁闭岛:1.5,007:2.0
5沉默的羔羊:4.5,肖申克的救赎:5.0,碟中谍:3.0
6沉默的羔羊:5.0,碟中谍:4.0,王的盛宴:4.0
7王的盛宴:5.0,碟中谍:1.5,肖申克的救赎:3.5,华尔街之狼:2.5
8禁闭岛:3.0,盗梦空间:4.0,教父:2.5,星球大战:3.5,肖申克的救赎:3.0,王的盛宴:3.0,华尔街之狼:2.0
9禁闭岛:4.0,教父:5.0,盗梦空间:4.5,沉默的羔羊:5.0,星球大战:4.0,王的盛宴:1.5,华尔街之狼:3.0
10肖申克的救赎:2.0,盗梦空间:3.5,沉默的羔羊:5.0,禁闭岛:2.5,王的盛宴:4.0,教父:2.0,碟中谍:1.5
11禁闭岛:3.0,碟中谍:2.5,盗梦空间:3.0,肖申克的救赎:3.0
12华尔街之狼:3.0,星球大战:3.5,碟中谍:1.5
13华尔街之狼:5.0,教父:3.5,肖申克的救赎:2.5,星球大战:5.0,沉默的羔羊:1.5,盗梦空间:3.0,禁闭岛:3.0
14星球大战:2.5,007:4.0,华尔街之狼:1.5,教父:4.5,盗梦空间:3.0,沉默的羔羊:3.5
15沉默的羔羊:4.0,华尔街之狼:2.5,肖申克的救赎:5.0
16肖申克的救赎:2.0,007:5.0,盗梦空间:3.5
17禁闭岛:1.5,星球大战:4.5,007:4.5
18007:4.5,华尔街之狼:5.0,沉默的羔羊:1.5,盗梦空间:2.0
19星球大战:4.5,华尔街之狼:3.0,肖申克的救赎:5.0,007:2.0,王的盛宴:4.0,碟中谍:2.5,沉默的羔羊:3.0
20007:2.0,教父:4.0,星球大战:2.5,盗梦空间:4.5,华尔街之狼:3.0,碟中谍:4.5,肖申克的救赎:3.0,禁闭岛:2.0
21碟中谍:2.5,禁闭岛:2.5,盗梦空间:1.5,肖申克的救赎:4.5,王的盛宴:2.0
22禁闭岛:1.5,沉默的羔羊:2.0,教父:4.0,肖申克的救赎:3.5,王的盛宴:1.5
23盗梦空间:3.5,华尔街之狼:4.0,007:2.0,肖申克的救赎:4.5
24007:4.0,华尔街之狼:5.0,教父:1.5,禁闭岛:1.5
25星球大战:2.0,禁闭岛:2.0,沉默的羔羊:5.0,王的盛宴:3.0
26肖申克的救赎:3.5,星球大战:4.5,教父:4.5,007:2.0
27沉默的羔羊:5.0,禁闭岛:1.5,肖申克的救赎:5.0
28华尔街之狼:5.0,007:5.0,星球大战:5.0,盗梦空间:3.0,王的盛宴:4.0,沉默的羔羊:2.0,教父:2.5,肖申克的救赎:4.0
29肖申克的救赎:3.0,盗梦空间:3.0,星球大战:3.5,王的盛宴:5.0,碟中谍:3.5,禁闭岛:1.5
30沉默的羔羊:4.5,星球大战:1.5,教父:1.5
31盗梦空间:3.0,肖申克的救赎:4.0,王的盛宴:3.0
32禁闭岛:2.5,碟中谍:2.0,盗梦空间:3.0
33教父:3.0,肖申克的救赎:4.5,华尔街之狼:4.5,盗梦空间:4.0,禁闭岛:5.0
34星球大战:2.0,沉默的羔羊:3.0,007:5.0,禁闭岛:2.0
35教父:2.5,华尔街之狼:4.5,007:1.5,盗梦空间:3.5,星球大战:1.5
36碟中谍:2.0,肖申克的救赎:4.0,教父:1.5,王的盛宴:5.0
37肖申克的救赎:2.0,沉默的羔羊:4.0,王的盛宴:2.5,盗梦空间:5.0,教父:2.5
38华尔街之狼:1.5,星球大战:4.0,王的盛宴:3.0
39教父:2.0,007:3.5,盗梦空间:3.5,王的盛宴:3.5
40盗梦空间:5.0,007:2.5,沉默的羔羊:4.5,碟中谍:3.5,星球大战:1.5,教父:3.0,华尔街之狼:3.0,王的盛宴:2.0
41007:1.5,教父:2.5,禁闭岛:4.5,沉默的羔羊:1.5,肖申克的救赎:2.0,盗梦空间:3.0,星球大战:4.0
42华尔街之狼:1.5,王的盛宴:1.5,教父:4.0
43华尔街之狼:3.5,教父:5.0,碟中谍:4.5
44王的盛宴:5.0,沉默的羔羊:5.0,教父:4.5,肖申克的救赎:4.0,盗梦空间:2.5,碟中谍:4.5,星球大战:1.5
45华尔街之狼:3.0,禁闭岛:2.0,王的盛宴:4.5
46007:2.0,王的盛宴:2.5,盗梦空间:4.0,星球大战:4.5,教父:1.5
47教父:2.5,华尔街之狼:3.0,007:5.0,碟中谍:1.5,禁闭岛:4.0
48星球大战:5.0,教父:4.5,盗梦空间:2.5
49沉默的羔羊:4.0,肖申克的救赎:5.0,王的盛宴:2.5,星球大战:1.5,碟中谍:2.0,华尔街之狼:4.5,盗梦空间:4.5
50盗梦空间:2.0,禁闭岛:1.5,沉默的羔羊:2.0
step2: 根据 step-out 通过所有用户看过的电影矩阵
输出:007:007 19(电影出现的次数)007:00719
007:华尔街之狼11
007:教父12
007:星球大战12
007:沉默的羔羊7
007:王的盛宴5
007:盗梦空间12
007:碟中谍6
007:禁闭岛8
007:肖申克的救赎7
华尔街之狼:00711
华尔街之狼:华尔街之狼23
华尔街之狼:教父14
华尔街之狼:星球大战13
华尔街之狼:沉默的羔羊9
华尔街之狼:王的盛宴10
华尔街之狼:盗梦空间13
华尔街之狼:碟中谍9
华尔街之狼:禁闭岛9
华尔街之狼:肖申克的救赎10
教父:00712
教父:华尔街之狼14
教父:教父25
教父:星球大战15
教父:沉默的羔羊11
教父:王的盛宴12
教父:盗梦空间17
教父:碟中谍8
教父:禁闭岛11
教父:肖申克的救赎12
星球大战:00712
星球大战:华尔街之狼13
星球大战:教父15
星球大战:星球大战23
星球大战:沉默的羔羊12
星球大战:王的盛宴11
星球大战:盗梦空间15
星球大战:碟中谍8
星球大战:禁闭岛10
星球大战:肖申克的救赎10
沉默的羔羊:0077
沉默的羔羊:华尔街之狼9
沉默的羔羊:教父11
沉默的羔羊:星球大战12
沉默的羔羊:沉默的羔羊21
沉默的羔羊:王的盛宴11
沉默的羔羊:盗梦空间12
沉默的羔羊:碟中谍7
沉默的羔羊:禁闭岛9
沉默的羔羊:肖申克的救赎12
王的盛宴:0075
王的盛宴:华尔街之狼10
王的盛宴:教父12
王的盛宴:星球大战11
王的盛宴:沉默的羔羊11
王的盛宴:王的盛宴23
王的盛宴:盗梦空间14
王的盛宴:碟中谍10
王的盛宴:禁闭岛9
王的盛宴:肖申克的救赎14
盗梦空间:00712
盗梦空间:华尔街之狼13
盗梦空间:教父17
盗梦空间:星球大战15
盗梦空间:沉默的羔羊12
盗梦空间:王的盛宴14
盗梦空间:盗梦空间29
盗梦空间:碟中谍11
盗梦空间:禁闭岛15
盗梦空间:肖申克的救赎17
碟中谍:0076
碟中谍:华尔街之狼9
碟中谍:教父8
碟中谍:星球大战8
碟中谍:沉默的羔羊7
碟中谍:王的盛宴10
碟中谍:盗梦空间11
碟中谍:碟中谍20
碟中谍:禁闭岛10
碟中谍:肖申克的救赎11
禁闭岛:0078
禁闭岛:华尔街之狼9
禁闭岛:教父11
禁闭岛:星球大战10
禁闭岛:沉默的羔羊9
禁闭岛:王的盛宴9
禁闭岛:盗梦空间15
禁闭岛:碟中谍10
禁闭岛:禁闭岛24
禁闭岛:肖申克的救赎12
肖申克的救赎:0077
肖申克的救赎:华尔街之狼10
肖申克的救赎:教父12
肖申克的救赎:星球大战10
肖申克的救赎:沉默的羔羊12
肖申克的救赎:王的盛宴14
肖申克的救赎:盗梦空间17
肖申克的救赎:碟中谍11
肖申克的救赎:禁闭岛12
肖申克的救赎:肖申克的救赎25
step3: 根据 step-out 用户评分矩阵
输出: 007 用户40:2.500740:2.5
00741:1.5
00735:1.5
00746:2.0
00717:4.5
0074:2.0
00723:2.0
00728:5.0
00747:5.0
00716:5.0
00719:2.0
00714:4.0
00718:4.5
00739:3.5
00724:4.0
0073:4.5
00726:2.0
00720:2.0
00734:5.0
华尔街之狼38:1.5
华尔街之狼7:2.5
华尔街之狼47:3.0
华尔街之狼40:3.0
华尔街之狼15:2.5
华尔街之狼23:4.0
华尔街之狼19:3.0
华尔街之狼24:5.0
华尔街之狼18:5.0
华尔街之狼49:4.5
华尔街之狼13:5.0
华尔街之狼28:5.0
华尔街之狼45:3.0
华尔街之狼12:3.0
华尔街之狼3:2.5
华尔街之狼9:3.0
华尔街之狼33:4.5 .。。。。。
step4: 根据step3-out和step4-out 计算推荐结果列表(计算电影的评分,根据电影出现的数据*用户对应得评分)
输入:007:007 19/007 40:2.5
输出:19 华尔街之狼,22.034教父,60.0
23教父,24.0
24教父,48.0
35教父,18.0
46教父,24.0
47教父,60.0
14教父,48.0
26教父,24.0
16教父,60.0
39教父,42.0
17教父,54.0
28教父,60.0
18教父,54.0
19教父,24.0
3教父,54.0
4教父,24.0
40教父,30.0
41教父,18.0
20教父,24.0
34碟中谍,30.0
23碟中谍,12.0
24碟中谍,24.0
35碟中谍,9.0
46碟中谍,12.0
47碟中谍,30.0
14碟中谍,24.0
26碟中谍,12.0
16碟中谍,30.0
39碟中谍,21.0
17碟中谍,27.0
28碟中谍,30.0
18碟中谍,27.0
19碟中谍,12.0
3碟中谍,27.0
4碟中谍,12.0
40碟中谍,15.0
41碟中谍,9.0
20碟中谍,12.0
34王的盛宴,25.0
23王的盛宴,10.0
24王的盛宴,20.0
.....
step5: 根据step-out4 合并所有数据
输出:19 华尔街之狼,289.01碟中谍,114.0
1教父,124.0
1肖申克的救赎,132.5
1王的盛宴,107.0
1盗梦空间,192.5
1禁闭岛,171.0
1007,89.0
1华尔街之狼,101.5
1沉默的羔羊,94.5
1星球大战,112.0
2碟中谍,138.0
2教父,178.5
2王的盛宴,192.0
2肖申克的救赎,225.0
2盗梦空间,268.5
2禁闭岛,193.5
2007,113.5
2华尔街之狼,142.5
2沉默的羔羊,146.0
2星球大战,157.5
3教父,369.0
3碟中谍,227.5
3王的盛宴,248.0
3肖申克的救赎,273.5
3盗梦空间,410.5
3禁闭岛,278.0
3007,299.0
3华尔街之狼,317.5
3沉默的羔羊,240.0
3星球大战,360.0
4教父,60.5
4碟中谍,77.0
4王的盛宴,48.5
4肖申克的救赎,59.5
4盗梦空间,74.0
.....
step6: 根据step5-out 排除用户看过得电影,然后把类似得,评分高得优先推荐(排序)。
输出: 1 肖申克的救赎 132.5 1 教父 124.0 1 星球大战 112.0 1 王的盛宴 107.0 1 华尔街之狼 101.5Movie [userid=1, movieName=肖申克的救赎, score=132.5]
Movie [userid=1, movieName=教父, score=124.0]
Movie [userid=1, movieName=星球大战, score=112.0]
Movie [userid=1, movieName=王的盛宴, score=107.0]
Movie [userid=1, movieName=华尔街之狼, score=101.5]
Movie [userid=1, movieName=沉默的羔羊, score=94.5]
Movie [userid=1, movieName=007, score=89.0]
Movie [userid=2, movieName=教父, score=178.5]
Movie [userid=2, movieName=星球大战, score=157.5]
Movie [userid=2, movieName=沉默的羔羊, score=146.0]
Movie [userid=2, movieName=华尔街之狼, score=142.5]
Movie [userid=2, movieName=碟中谍, score=138.0]
Movie [userid=2, movieName=007, score=113.5]
Movie [userid=3, movieName=肖申克的救赎, score=273.5]
Movie [userid=3, movieName=王的盛宴, score=248.0]
Movie [userid=3, movieName=沉默的羔羊, score=240.0]
Movie [userid=4, movieName=盗梦空间, score=74.0]
Movie [userid=4, movieName=教父, score=60.5]
Movie [userid=4, movieName=星球大战, score=59.0]
Movie [userid=4, movieName=华尔街之狼, score=58.0]
Movie [userid=4, movieName=王的盛宴, score=48.5]
Movie [userid=4, movieName=沉默的羔羊, score=45.0]
Movie [userid=5, movieName=盗梦空间, score=172.0]
Movie [userid=5, movieName=王的盛宴, score=149.5]
Movie [userid=5, movieName=教父, score=133.5]
Movie [userid=5, movieName=禁闭岛, score=130.5]
Movie [userid=5, movieName=星球大战, score=128.0]
Movie [userid=5, movieName=华尔街之狼, score=117.5]
Movie [userid=5, movieName=007, score=84.5]
Movie [userid=6, movieName=盗梦空间, score=160.0]
Movie [userid=6, movieName=肖申克的救赎, score=160.0]
Movie [userid=6, movieName=星球大战, score=136.0]
Movie [userid=6, movieName=教父, score=135.0]
Movie [userid=6, movieName=华尔街之狼, score=121.0]
Movie [userid=6, movieName=禁闭岛, score=121.0]
Movie [userid=6, movieName=007, score=79.0]
Movie [userid=7, movieName=盗梦空间, score=178.5]
Movie [userid=7, movieName=教父, score=149.0]
Movie [userid=7, movieName=星球大战, score=134.5]
Movie [userid=7, movieName=沉默的羔羊, score=130.0]
Movie [userid=7, movieName=禁闭岛, score=124.5]
Movie [userid=7, movieName=007, score=86.0]
Movie [userid=8, movieName=沉默的羔羊, score=231.5]
Movie [userid=8, movieName=碟中谍, score=203.0]
Movie [userid=8, movieName=007, score=202.0]
Movie [userid=9, movieName=肖申克的救赎, score=335.5]
Movie [userid=9, movieName=007, score=269.5]
Movie [userid=9, movieName=碟中谍, score=238.5]
Movie [userid=10, movieName=星球大战, score=243.5]
.....
代码:package com.huhu.day06;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.WritableComparable;
public class Movie implements WritableComparable {
private int userid;
private String movieName;
private float score;
public Movie() {
super();
}
public Movie(int userid, String movieName, float score) {
super();
this.userid = userid;
this.movieName = movieName;
this.score = score;
}
public int getUserid() {
return userid;
}
public void setUserid(int userid) {
this.userid = userid;
}
public String getMovieName() {
return movieName;
}
public void setMovieName(String movieName) {
this.movieName = movieName;
}
public float getScore() {
return score;
}
public void setScore(float score) {
this.score = score;
}
@Override
public int hashCode() {
final int prime = 31;
int result = 1;
result = prime * result + ((movieName == null) ? 0 : movieName.hashCode());
result = prime * result + Float.floatToIntBits(score);
result = prime * result + userid;
return result;
}
@Override
public boolean equals(Object obj) {
if (this == obj)
return true;
if (obj == null)
return false;
if (getClass() != obj.getClass())
return false;
Movie other = (Movie) obj;
if (movieName == null) {
if (other.movieName != null)
return false;
} else if (!movieName.equals(other.movieName))
return false;
if (Float.floatToIntBits(score) != Float.floatToIntBits(other.score))
return false;
if (userid != other.userid)
return false;
return true;
}
@Override
public String toString() {
return "Movie [userid=" + userid + ", movieName=" + movieName + ", score=" + score + "]";
}
@Override
public void readFields(DataInput in) throws IOException {
this.userid = in.readInt();
this.movieName = in.readUTF();
this.score = in.readFloat();
}
@Override
public void write(DataOutput out) throws IOException {
out.writeInt(userid);
out.writeUTF(movieName);
out.writeFloat(score);
}
@Override
public int compareTo(Movie o) {
if (this.userid == o.getUserid()) {
if (this.score == o.score) {
return this.movieName.compareTo(o.movieName);
} else {
return (int) (o.getScore() - this.score);
}
} else {
return this.userid - o.getUserid();
}
}
}
package com.huhu.day06;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
* 输入: 1 盗梦空间 4.0
*
* 输出 : 1 盗梦空间:4.0,碟中谍:1.5,禁闭岛:4.0
*
* @author huhu_k
*
*/
public class Step1 {
static class MyMapper extends Mapper {
@Override
protected void map(LongWritable key, Text value, Mapper.Context context)
throws IOException, InterruptedException {
String[] line = value.toString().split("\t");
// k:用户 v: item电影名称:分数
context.write(new IntWritable(Integer.valueOf(line[0])), new Text(line[1] + ":" + line[2]));
}
}
// 相同key的数据相遇 k:1 v:{}
static class MyReduce extends Reducer {
Text va = new Text();
@Override
protected void reduce(IntWritable key, Iterable values, Context context)
throws IOException, InterruptedException {
StringBuffer br = new StringBuffer();
for (Text v : values) {
br.append("," + v.toString());
}
va.set(br.toString().replaceFirst(",", ""));
// k:用户id v:电影1:评分,电影2:评分.....
// 1 盗梦空间:4.0,碟中谍:1.5,禁闭岛:4.0
context.write(key, va);
}
}
public Job getJob(Configuration conf) throws Exception {
Path inpath = new Path("hdfs://ry-hadoop1:8020/in/items.txt");
Path outpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step1");
Job job = Job.getInstance(conf, this.getClass().getSimpleName());
job.setJarByClass(this.getClass());
job.setMapperClass(MyMapper.class);
job.setMapOutputKeyClass(IntWritable.class);
job.setMapOutputValueClass(Text.class);
job.setReducerClass(MyReduce.class);
job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(Text.class);
FileInputFormat.addInputPath(job, inpath);
FileOutputFormat.setOutputPath(job, outpath);
return job;
}
}
package com.huhu.day06;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
* 输入:1 盗梦空间:4.0,碟中谍:1.5,禁闭岛:4.0
*
* 输出 :007 40:2.5
*
* @author huhu_k
*
*/
public class Step3 {
static class MyMapper extends Mapper {
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String[] line = value.toString().split("\t");
String userid = line[0];
String[] item = line[1].split(",");
for (String it : item) {
context.write(new Text(it.split(":")[0]), new Text(userid + ":"+new Text(it.split(":")[1])));
}
}
}
public Job getJob(Configuration conf) throws Exception {
Path inpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step1");
Path outpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step3");
Job job = Job.getInstance(conf, this.getClass().getSimpleName());
job.setJarByClass(Step3.class);
job.setMapperClass(MyMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
FileInputFormat.addInputPath(job, inpath);
FileOutputFormat.setOutputPath(job, outpath);
return job;
}
}
package com.huhu.day06;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
* 商品同现矩阵 输出 :1 盗梦空间:4.0,碟中谍:1.5,禁闭岛:4.0
*
* 输出:007:007 19
*
* @author huhu_k
*
*/
public class Step2 {
static class MyMapper extends Mapper {
private Text k = new Text();
private final IntWritable one = new IntWritable(1);
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
//盗梦空间:4.0 碟中谍:1.5 禁闭岛:4.0
String[] line = value.toString().split("\t")[1].split(",");
System.err.println(line+"----------------------");
for (int i = 0; i < line.length; i++) {
for (int j = 0; j < line.length; j++) {
k.set(line[i].split(":")[0] + ":" + line[j].split(":")[0]);
// k: 盗梦空间:碟中谍 1 每个人看过所有电影的乘积
context.write(k, one);
}
}
}
}
static class MyReducer extends Reducer {
// k: 盗梦空间:碟中谍 1
@Override
protected void reduce(Text key, Iterable values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable v : values) {
sum += v.get();
}
// k:007:007 v:19
context.write(key, new IntWritable(sum));
}
}
public Job getJob(Configuration conf) throws Exception {
Path inpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step1");
Path outpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step2");
Job job = Job.getInstance(conf, this.getClass().getSimpleName());
job.setJarByClass(this.getClass());
job.setMapperClass(MyMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setReducerClass(MyReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, inpath);
FileOutputFormat.setOutputPath(job, outpath);
return job;
}
}package com.huhu.day06;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
* 输入:1 盗梦空间:4.0,碟中谍:1.5,禁闭岛:4.0
*
* 输出 :007 40:2.5
*
* @author huhu_k
*
*/
public class Step3 {
static class MyMapper extends Mapper {
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String[] line = value.toString().split("\t");
String userid = line[0];
String[] item = line[1].split(",");
for (String it : item) {
context.write(new Text(it.split(":")[0]), new Text(userid + ":"+new Text(it.split(":")[1])));
}
}
}
public Job getJob(Configuration conf) throws Exception {
Path inpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step1");
Path outpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step3");
Job job = Job.getInstance(conf, this.getClass().getSimpleName());
job.setJarByClass(Step3.class);
job.setMapperClass(MyMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
FileInputFormat.addInputPath(job, inpath);
FileOutputFormat.setOutputPath(job, outpath);
return job;
}
}package com.huhu.day06;
import java.io.IOException;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.FileSplit;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
* step4: 计算推荐结果列表 输入:007:007 19 输入:007 40:2.5
*
* 输出:19 华尔街之狼,22.0
*
* @author huhu_k
*
*/
public class Step4 {
static class MyMapper extends Mapper {
String filename = "";
@Override
protected void setup(Context context) throws IOException, InterruptedException {
FileSplit inputSplit = (FileSplit) context.getInputSplit();
// 获取父文件名
filename = inputSplit.getPath().getParent().getName();
}
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String[] line = value.toString().split("\t");
if ("step2".equals(filename)) {
String[] v1 = line[0].split(":");
String item1 = v1[0];
String item2 = v1[1];
String num = line[1];
context.write(new Text(item1), new Text("A:" + item2 + "," + num));
} else if ("step3".equals(filename)) {
String item = line[0];
String userid = line[1].split(":")[0];
String score = line[1].split(":")[1];
context.write(new Text(item), new Text("B:" + userid + "," + score));
}
}
}
// 相同key的数据相遇 k:1 v:{}
static class MyReduce extends Reducer {
// k: 电影名称 v:A:007,20 来自于商品同现矩阵
// k: 电影名称 v:B:1,4 来自于用户评分矩阵
String[] info = null;
@Override
protected void reduce(Text key, Iterable values, Context context)
throws IOException, InterruptedException {
Map mapA = new HashMap<>();
Map mapB = new HashMap<>();
for (Text va : values) {
String src = va.toString();
if (src.startsWith("A:")) {
// A:电影名称,次数
info = src.substring(2).split(",");
// mapA k: 电影名称 v:次数
mapA.put(info[0], info[1]);
} else if (src.startsWith("B:")) {
// B:用户,评分
info = src.substring(2).split(",");
// mapB k: 用户 v:评分
mapB.put(info[0], info[1]);
}
}
float result = 0;
Iterator iterator = mapA.keySet().iterator();
while (iterator.hasNext()) {
String item = iterator.next();
int num = Integer.parseInt(mapA.get(item));
Iterator iterator2 = mapB.keySet().iterator();
while (iterator2.hasNext()) {
String userid = iterator2.next();
float score = Float.valueOf(mapB.get(userid));
result = score * num;
context.write(new Text(userid), new Text(item + "," + result));
}
}
}
}
public Job getJob(Configuration conf) throws Exception {
Path inpath1 = new Path("hdfs://ry-hadoop1:8020/out/day06/step2");
Path inpath2 = new Path("hdfs://ry-hadoop1:8020/out/day06/step3");
Path outpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step4");
Job job = Job.getInstance(conf, this.getClass().getSimpleName());
job.setJarByClass(Step4.class);
job.setMapperClass(MyMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
job.setReducerClass(MyReduce.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
FileInputFormat.addInputPath(job, inpath1);
FileInputFormat.addInputPath(job, inpath2);
FileOutputFormat.setOutputPath(job, outpath);
return job;
}
}package com.huhu.day06;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
* step5: 合并结果列表 输入:19 华尔街之狼,22.0
*
* 输出:19 华尔街之狼,229.0
*
* @author huhu_k
*
*/
public class Step5 {
static class MyMapper extends Mapper {
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String[] line = value.toString().split("\t");
context.write(new IntWritable(Integer.parseInt(line[0])), new Text(line[1]));
}
}
// 相同key的数据相遇 k:1 v:{}
static class MyReduce extends Reducer {
// k: 电影名称 v:A:007,20 来自于商品同现矩阵
// k: 电影名称 v:B:1,4 来自于用户评分矩阵
@Override
protected void reduce(IntWritable key, Iterable values, Context context)
throws IOException, InterruptedException {
Map map = new HashMap<>();
for (Text v : values) {
String[] str = v.toString().split(",");
String item = str[0];
float score = Float.valueOf(str[1]);
if (map.containsKey(item)) {
map.put(item, map.get(item) + score);
} else {
map.put(item, score);
}
}
for (Map.Entry m : map.entrySet()) {
context.write(key, new Text(m.getKey() + "," + m.getValue()));
}
}
}
public Job getJob(Configuration conf) throws Exception {
Path inpath1 = new Path("hdfs://ry-hadoop1:8020/out/day06/step4");
Path outpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step5");
Job job = Job.getInstance(conf, this.getClass().getSimpleName());
job.setJarByClass(Step5.class);
job.setMapperClass(MyMapper.class);
job.setMapOutputKeyClass(IntWritable.class);
job.setMapOutputValueClass(Text.class);
job.setReducerClass(MyReduce.class);
job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(Text.class);
FileInputFormat.addInputPath(job, inpath1);
FileOutputFormat.setOutputPath(job, outpath);
return job;
}
}package com.huhu.day06;
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.net.URI;
import java.util.HashMap;
import java.util.Map;
import java.util.TreeSet;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.filecache.DistributedCache;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
* 输入:排除 用户看过的电影,并且 排序输出 降序 输入:19 华尔街之狼,289.0
*
* 输出 :1 肖申克的救赎 132.5 1 教父 124.0 1 星球大战 112.0 1 王的盛宴 107.0 1 华尔街之狼 101.5
*
* @author huhu_k
*
*/
public class Step6 {
static class MyMapper extends Mapper {
private Map map;
private Path[] localFiles;
@Override
protected void setup(Context context) throws IOException, InterruptedException {
map = new HashMap<>();
Configuration conf = context.getConfiguration();
localFiles = DistributedCache.getLocalCacheFiles(conf);
for (Path p : localFiles) {
BufferedReader br = new BufferedReader(new FileReader(p.toString()));
String word = "";
while ((word = br.readLine()) != null) {
String[] s = word.split("\t");
int userid = Integer.parseInt(s[0]);
String item = s[1];
if (map.containsKey(userid)) {
// 1 007;008
map.put(userid, map.get(userid) + ";" + item);
} else {
// 1 007
map.put(userid, item);
}
}
br.close();
}
}
@Override
protected void map(LongWritable key, Text value, Mapper.Context context)
throws IOException, InterruptedException {
String[] line = value.toString().split("\t");
// 19 华尔街之狼,289.0
int userid = Integer.valueOf(line[0]);
String movie = line[1].split(",")[0];
String movieList = map.get(userid);
// map得到null 第一次来 没有任何数据
if (movieList == null || movieList.length() == 0 || movieList == "" || !movieList.contains(movie)) {
context.write(new IntWritable(userid), new Text(line[1]));
}
}
}
// 相同key的数据相遇 k:1 v:{}
static class MyReduce extends Reducer {
private TreeSet set = new TreeSet<>();
@Override
protected void reduce(IntWritable key, Iterable values, Context context)
throws IOException, InterruptedException {
for (Text v : values) {
Movie m = new Movie(Integer.parseInt(key.toString()), v.toString().split(",")[0],
Float.parseFloat(v.toString().split(",")[1]));
set.add(m);
}
}
@Override
protected void cleanup(Context context) throws IOException, InterruptedException {
for (Movie m : set) {
context.write(m, NullWritable.get());
}
}
}
public Job getJob(Configuration conf) throws Exception {
Path inpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step5");
Path outpath = new Path("hdfs://ry-hadoop1:8020/out/day06/step6");
DistributedCache.addCacheFile(new URI("hdfs://ry-hadoop1:8020/in/items.txt"), conf);
Job job = Job.getInstance(conf, this.getClass().getSimpleName());
job.setJarByClass(Step6.class);
job.setMapperClass(MyMapper.class);
job.setMapOutputKeyClass(IntWritable.class);
job.setMapOutputValueClass(Text.class);
job.setReducerClass(MyReduce.class);
job.setOutputKeyClass(Movie.class);
job.setOutputValueClass(NullWritable.class);
FileInputFormat.addInputPath(job, inpath);
FileOutputFormat.setOutputPath(job, outpath);
return job;
}
}
package com.huhu.day06;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class Movie_App extends ToolRunner implements Tool {
private Configuration con;
public static void main(String[] args) throws Exception {
Movie_App m = new Movie_App();
int num = ToolRunner.run(m.getConf(), m, args);
System.exit(num);
}
@Override
public Configuration getConf() {
if (con == null) {
return new Configuration();
}
return con;
}
@Override
public void setConf(Configuration arg0) {
}
@Override
public int run(String[] arg0) throws Exception {
Step1 step1 = new Step1();
Step2 step2 = new Step2();
Step3 step3 = new Step3();
Step4 step4 = new Step4();
Step5 step5 = new Step5();
Step6 step6 = new Step6();
ControlledJob controlledJob1 = new ControlledJob(step1.getJob(getConf()).getConfiguration());
ControlledJob controlledJob2 = new ControlledJob(step2.getJob(getConf()).getConfiguration());
ControlledJob controlledJob3 = new ControlledJob(step3.getJob(getConf()).getConfiguration());
ControlledJob controlledJob4 = new ControlledJob(step4.getJob(getConf()).getConfiguration());
ControlledJob controlledJob5 = new ControlledJob(step5.getJob(getConf()).getConfiguration());
ControlledJob controlledJob6 = new ControlledJob(step6.getJob(getConf()).getConfiguration());
controlledJob2.addDependingJob(controlledJob1);
controlledJob3.addDependingJob(controlledJob1);
controlledJob4.addDependingJob(controlledJob2);
controlledJob4.addDependingJob(controlledJob3);
controlledJob5.addDependingJob(controlledJob4);
controlledJob6.addDependingJob(controlledJob5);
JobControl jobControl = new JobControl("Movive");
jobControl.addJob(controlledJob1);
jobControl.addJob(controlledJob2);
jobControl.addJob(controlledJob3);
jobControl.addJob(controlledJob4);
jobControl.addJob(controlledJob5);
jobControl.addJob(controlledJob6);
Thread t = new Thread(jobControl);
t.start();
while (!jobControl.allFinished()) {
t.sleep(1000);
}
jobControl.stop();
return 0;
}
}
关于什么是矩阵:我看了两篇比较好得,推荐给大家。