bi-graph with Map-Reduce implementation

instance: ( u , i , r u i , t u i ) (u, i, r_{ui}, t_{ui}) (u,i,rui,tui)
describes user u u u applies item i i i on time t u i t_{ui} tui with score r u i r_{ui} rui.

formular:
S i j = 1 k j λ ∑ u r u i r u j k u ρ g ( t u i , t u j ) g ( t 1 , t 2 ) = exp ⁡ [ − ( t 1 − t 2 ) 2 2 τ 2 ] S_{ij} = \frac 1 {k_j^\lambda} \sum_u \frac {r_{ui} r_{uj}} {k_u^\rho} g(t_{ui}, t_{uj}) \\ g(t_1, t_2) = \exp[ - \frac {(t_1 - t_2)^2} {2\tau^2}] Sij=kjλ1ukuρruirujg(tui,tuj)g(t1,t2)=exp[2τ2(t1t2)2]

parameters: λ , ρ , τ \lambda, \rho, \tau λ,ρ,τ

source data:
( u , p , r , t ) (u, p, r, t) (u,p,r,t)

  1. MAP:
    p : ( u , r , t ) p: (u,r,t) p:(u,r,t)
    REDUCE:
    p : [ ( u , r , t ) , ( ) , . . . ] p: [(u,r,t), (), ...] p:[(u,r,t),(),...]
    calc: k p = ( ∑ r ) λ k_p = (\sum r)^\lambda kp=(r)λ,
    u : ( p , r , t , k p ) u:(p,r,t, k_p) u:(p,r,t,kp)
  2. REDUCE:
    u : [ ( p , r , t , k p ) , ( ) , . . . ] u: [(p,r,t, k_p), (), ...] u:[(p,r,t,kp),(),...]
  3. Map
    calc: k u = ( ∑ r ) ρ k_u = (\sum r)^\rho ku=(r)ρ,
    p 0 : { ( p i , s i ) , ⋯   , } p_0: \{(p_i, s_i), \cdots,\} p0:{(pi,si),,}
    with : s i → r 0 r i k u k p i g ( t 0 − t i ) s_i \to \frac {r_0r_i} {k_u k_{pi} }g(t_0 - t_i) sikukpir0rig(t0ti)

Reduce:
p 0 : [ ( p i , s i ) , ( ) , . . . ] p_0: [(p_i,s_i), (), ...] p0:[(pi,si),(),...]
with: s i → ∑ j = i s j s_i \to \sum_{j = i} s_j sij=isj

Note

map(3)可合并到Reduce(2)中,但会极大增加2结果文件的大小,且会略微增加总耗时
时间对比:

#合并前
job1 time: 115s    1.9m
job2 time: 110s    1.83m
job3 time: 4551s   75.8m
total time: 4776s  79.6m

# 合并后
job1 time: 110s    1.83m
job2 time: 1217s   20.3m
job3 time: 3656s   60.9m
total time: 4983s  83m 
  • 1
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值