I write this stuff since it confused me a lot.
First, the input Iterable values for reduce method is not sorted in any
order. It is mentioned on Page 277 in Hadoop: The Definitive Guide,
Third Edition.
Job has three methods related to second sort:
1) setPartitionerClass
2) setSortComparatorClass
3) setGroupingComparatorClass
For 0.18.3 API, JobConf's following three methods are used:
1) setPartitionerClass
2) setOutputKeyComparatorClass
3) setOutputValueGroupingComparator
The confusing part is setGroupingComparatorClass. For the following
input for reduce:
<k1, v1> <k2, v2> // sorted by key from map phase
If setGroupingComparatorClass decides that k1 is equivalent to k2 even
thought k1 and k2 are different, the output from reduce method will be:
<k1, <v1, v2>> // <v1, v2> are in the original key order, k2 is lost.
Now let's talk about MaxTemperatureUsingSecondarySort. Input is like:
1901 99
1901 98
1902 80
1902 10
The output from map is:
<<1901, 99>, null>
<<1901, 98>, null>
<<1902, 80>, null>
<<1902, 10>, null>
The reduce method will be invoked twice. The output is:
<1901, 99>
<1902, 80>
Now turns to SecondarySort in Hadoop examples. For the same input, the
output from map is:
<<1901, 98>, 98>
<<1901, 99>, 99>
<<1902, 10>, 10>
<<1902, 80>, 80>
The reduce method will also be invoked twice. The output is:
<1901, 98>
<1901, 99>
<1902, 10>
<1902, 80>
First, the input Iterable values for reduce method is not sorted in any
order. It is mentioned on Page 277 in Hadoop: The Definitive Guide,
Third Edition.
Job has three methods related to second sort:
1) setPartitionerClass
2) setSortComparatorClass
3) setGroupingComparatorClass
For 0.18.3 API, JobConf's following three methods are used:
1) setPartitionerClass
2) setOutputKeyComparatorClass
3) setOutputValueGroupingComparator
The confusing part is setGroupingComparatorClass. For the following
input for reduce:
<k1, v1> <k2, v2> // sorted by key from map phase
If setGroupingComparatorClass decides that k1 is equivalent to k2 even
thought k1 and k2 are different, the output from reduce method will be:
<k1, <v1, v2>> // <v1, v2> are in the original key order, k2 is lost.
Now let's talk about MaxTemperatureUsingSecondarySort. Input is like:
1901 99
1901 98
1902 80
1902 10
The output from map is:
<<1901, 99>, null>
<<1901, 98>, null>
<<1902, 80>, null>
<<1902, 10>, null>
The reduce method will be invoked twice. The output is:
<1901, 99>
<1902, 80>
Now turns to SecondarySort in Hadoop examples. For the same input, the
output from map is:
<<1901, 98>, 98>
<<1901, 99>, 99>
<<1902, 10>, 10>
<<1902, 80>, 80>
The reduce method will also be invoked twice. The output is:
<1901, 98>
<1901, 99>
<1902, 10>
<1902, 80>