Cloud Computing(3)_Basic MapReduce Algorithm Design_Pairs&Stripes

How do we aggregate partial counts efficiently?

Pairs

An algorithm.
This algorithm illustrates the use of complex keys in order to coordinate distributed computations.

• Each mapper takes a sentence
• Reducers sum up counts associated with these pairs

//"pairs" approach
class MAPPER
    method MAP(docid a, doc d)
        for all term w∈doc d do
            for all term u∈NEIGHBORS(w) do
                EMIT( pair(w, u) , count 1)   //EMIT count for each co-occurrence

class REDUCER
    method REDUCE(pair p, counts[c1, c2, ...])
        s = 0
        for all count c ∈counts[c1, c2, ...] do
            s = s + c
        EMIT(pair p, count s)

For each term emit pairs: ( (a,b), 1 ) 键值是一个pair（a,b）

“Pairs Analysis”(数组短，但数目多)

• Advantages

  • Easy to implement, easy to understand: map就是找pair，reduce就是统计

• Disadvantages

  • Lots of pairs to sort and shuffle around, upper bound = (n!)(n个单词，就有n的阶乘个pairs)
  • Not many opportunities for combiners to work

Stripes

Co-occurrence information is first stored in an associative array, denoted H.
The mapper emits key-value pairs with words as keys and corresponding associative arrays as values, where each associative array encodes the co-occurrence counts of the neighbors of a particular word.

• Each mapper takes a sentence
• Reducers perform element-wise sum of associative arrays

//"stripes" approach
class MAPPER
    method MAP(docid a, doc d)
        for all term w∈doc d do
            H = new ASSOCIATIVEARRAY
            for all term u∈NEIGHBORS(w) do
                H{u} = H{u} + 1   //Tally words co-occurring with w
            EMIT( term w , Stripe H)  

class REDUCER
    method REDUCE(term w , Stripes [H1, H2, H3,...])
        Hf = new ASSOCIATIVEARRAY
        for all stripe H ∈stripes[H1, H2, H3, ...] do
            sum(Hf,H)
        EMIT(term w , Stripe Hf)

For each term emit stripes: a->{b:1, c:2, d:2, ….} 键值是“a”

“Stripes Analysis”(数组长，但数目少)

• Advantages

  • Far less sorting and shuffling of key-value pairs
  • Can make better use of combiners

• Disadvantages

  • More difficult to implement
  • Underlying object more heavyweight
  • Fundamental limitation in terms of size of event space

Pairs vs. Stripes

• 处理量不大，处理资源数目少，用pairs；反之，stripes较优