HDPCD-Java-复习笔记（4）

Map Aggregation

Aggregation

The term refers to a Mapper combining its <key, value> pairs, with the goal of reducing the amount of network traffic between the Mapper and the Reducer.

There are two ways to perform Map Aggregation in Hadoop:

Combiners --- The MapReduce framework has the concept of a Combiner, where you write a class that defines the aggregation, and the framework decides when to perform the aggregation.

In-map Aggregation --- The Mapper contains logic that aggregates records, typically accomplished by buffering records in memory prior to writing them out.

Overview of Combiners

The < key ,value > records output by the Mapper are serialized, so the Combiner has to deserialize them.

A Combiner only aggregates data on one node. It does not combine the output of multiple Mappers.

Reduce-side Combining

The Combiner is also used in the reduce phase if the intermediate <key,value> pairs from Mappers are spilled to disk. 

The fact that the Reducer uses the Combiner behind-the-scenes to improve file I/O.

Counters

The pre-defined counters include usefulinformation,like the number of map input records, or the amount of byteswritten to HDFS.

The Hadoop counters are global -they are asummation of events that occurs across the entire cluster.

 

User-defined Counters

Two ways to define your own counter in Hadoop:

1.Use an enum to define a group,and the elements in the enum become the counter names.

2.Use strings for the group name and counter name.

Combiner Example

public class WordCountCombiner
   extends Reducer<Text, IntWritable, Text, IntWritable> {
    private IntWritable outputValue = new IntWritable();
    @Override
    protected void reduce(Text key,
             Iterable<IntWritable> values, Context context)
              throws IOException, InterruptedException {
        int sum = 0;
        for(IntWritable count : values) {
            sum += count.get();
        }
        outputValue.set(sum);
        context.write(key, outputValue);
    }
}




In-Map Aggregation








In-Map Aggregation Example

 
 
public class TopResultsMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
    private ArrayList<Word> words = new ArrayList<Word>();
    private PriorityQueue<Word> queue;
    private int maxResults;

    @Override
    protected void setup(Context context)
        throws IOException, InterruptedException {
        maxResults = Integer.parseInt(context.getConfiguration()
                                             .get("maxResults"));
    }

    @Override
    protected void map(LongWritable key, Text value, Context context)
        throws IOException, InterruptedException {
        String[] input = StringUtils.split(value.toString(), '\\', ' ');

        for (String word : input) {
            Word currentWord = new Word(word, 1);

            if (words.contains(currentWord)) {
                //increment the existing Word's frequency
                for (Word w : words) {
                    if (w.equals(currentWord)) {
                        w.frequency++;

                        break;
                    }
                }
            } else {
                words.add(currentWord);
            }
        }
    }

    @Override
    protected void cleanup(Context context)
        throws IOException, InterruptedException {
        Text outputKey = new Text();
        IntWritable outputValue = new IntWritable();
        queue = new PriorityQueue<Word>(words.size());
        queue.addAll(words);

        for (int i = 1; i <= maxResults; i++) {
            Word tail = queue.poll();

            if (tail != null) {
                outputKey.set(tail.value);
                outputValue.set(tail.frequency);
                context.write(outputKey, outputValue);
            }
        }
    }
}
 
 




public class Word implements Comparable<Word> {
    public String value;
    public int frequency;

    public Word(String value, int frequency) {
        this.value = value;
        this.frequency = frequency;
    }

    @Override
    public boolean equals(Object obj) {
        if (obj instanceof Word) {
            return value.equalsIgnoreCase(((Word) obj).value);
        } else {
            return false;
        }
    }

    @Override
    public int compareTo(Word w) {
        return w.frequency - this.frequency;
    }
}





public enum MyCounters {
     GOOD_RECORDS, BAD_RECORDS
}
context.getCounter(MyCounters.GOOD_RECORDS).increment(1);