spatialhadoop2.3源码阅读(八) RTree索引生成方法(二)

这一章主要介绍MapReduce的具体实现。

1. Map

 /**
   * The map class maps each object to the cell with maximum overlap.
   * @author Ahmed Eldawy
   *
   */
  public static class RepartitionMapNoReplication<T extends Shape> extends MapReduceBase
      implements Mapper<Rectangle, T, IntWritable, T> {
    /**List of cells used by the mapper*/
    private CellInfo[] cellInfos;
    
    /**Used to output intermediate records*/
    private IntWritable cellId = new IntWritable();
    
    @Override
    public void configure(JobConf job) {
      try {
        cellInfos = SpatialSite.getCells(job);
        super.configure(job);
      } catch (IOException e) {
        throw new RuntimeException("Error loading cells", e);
      }
    }
    
    /**
     * Map function
     * @param dummy
     * @param shape
     * @param output
     * @param reporter
     * @throws IOException
     */
    public void map(Rectangle cellMbr, T shape,
        OutputCollector<IntWritable, T> output, Reporter reporter)
        throws IOException {
      Rectangle shape_mbr = shape.getMBR();
      if (shape_mbr == null)
        return;
      double maxOverlap = -1.0;
      int bestCell = -1;
      // Only send shape to output if its lowest corner lies in the cellMBR
      // This ensures that a replicated shape in an already partitioned file
      // doesn't get send to output from all partitions
      if (!cellMbr.isValid() || cellMbr.contains(shape_mbr.x1, shape_mbr.y1)) {
        for (int cellIndex = 0; cellIndex < cellInfos.length; cellIndex++) {
          Rectangle overlap = cellInfos[cellIndex].getIntersection(shape_mbr);
          if (overlap != null) {
            double overlapArea = overlap.getWidth() * overlap.getHeight();
            if (bestCell == -1 || overlapArea > maxOverlap) {
              maxOverlap = overlapArea;
              bestCell = cellIndex;
            }
          }
        }
      }
      if (bestCell != -1) {
        cellId.set((int) cellInfos[bestCell].cellId);
        output.collect(cellId, shape);
      } else {
        LOG.warn("Shape: "+shape+" doesn't overlap any partitions");
      }
    }
  }

Map类大致可以分为两部分：configure方法和map方法。

configure方法的主要功能是获得上一章所讲的CellInfo数组。

接下来重点介绍map方法。

35行：获得输入数据的最小包围矩形。

43行：与spatialhadoop2.3源码阅读(六) grid 索引生成方法(二)中相比，当前cellMbr的值为Rectangle: (NaN,0.0)-(0.0,0.0)，所以if验证的前       半部分始终为true

44-51：遍历所有的网格，判断当前输入数据与哪一个网格的相交面积最大，则认为输入数据属于哪一个网格。

45：获得当前网格与输入数据的相交矩形，若不相交，则为null

47-50：若相交，与前一次的相交面积进行比较，若大于则更新。

55-57：将得出的当前输入数据所属的网格索引和输入数据一起输出。

2. Reduce

Reduce类与spatialhadoop2.3源码阅读(六) grid 索引生成方法(二)中的reduce完全相同，详情见该章。

3. outputCommiter

Commiter类与spatialhadoop2.3源码阅读(六) grid 索引生成方法(二)中的Commiter完全相同，详情见该章。

commiter的作用是将生成的所有包含_master的文件合并为同一个，生成_master.grid文件