Partitioners负责划分Maper输出的中间键值对的key,分配中间键值对到不同的Reducer。Maper输出的中间结果交给指定的Partitioner,确保中间结果分发到指定的Reduce任务。在每个Reducer中,键按排序顺序处理(Within each reducer, keys are processed in sorted order)。Combiners是MapReduce中的一个优化,允许在shuffle和排序阶段之前在本地进行聚合。Combiners的首要目标是通过最小化键值对的数量来节省尽可能多的带宽,键值对将通过网络在mappers和reducers之间进行shuffle操作(The primary goal of combiners is to save as much bandwidth as possible by minimizing the number of key/value pairs that will be shuffled across the network between mappers and reducers)。我们可以把Combiners理解为发生在shuffle和sort阶段之前,对Mapper输出进行操作的"mini-reducers"。每个Combiner单独操作于一个Mapper,因此不能访
[Hadoop]MapReduce中的Partitioner与Combiner
最新推荐文章于 2023-06-30 18:45:00 发布