MapReduce源码分析之MapTask分析(二)

SpillThread分析

为什么需要Spill

         内存大小总是有效，因此在Mapper在处理过程中，数据持续输出到内存中时，必然需要有机制能将内存中的数据换出，合理的刷出到磁盘上。SpillThread就是用来完成这部分工作。

         SpillThread的线程处理函数只是做一层封装，当索引表中的kvstart和kvend指向一样的索引位置时，会持续处于等待过程，等待外部通知需要触发spill动作，当有spill请求时，会触发StartSpill来唤醒SpillThread线程，进入到sortAndSpill。

    下面就是SpillThread线程体函数。

    protected class SpillThread extends Thread {
      @Override
      public void run() {
        spillLock.lock();
        spillThreadRunning = true;
        try {
          while (true) {
            spillDone.signal();
            while (kvstart == kvend) {
			// 等待被唤醒
              spillReady.await();
            }
            try {
              spillLock.unlock();
			// spill处理
              sortAndSpill();
            } catch (...) {
			...
            } finally {
              spillLock.lock();
			// 重置索引区，更新buf缓冲区的尾部位置信息
              if (bufend < bufindex && bufindex < bufstart) {
                bufvoid = kvbuffer.length;
              }
              kvstart = kvend;
              bufstart = bufend;
            }
          }
        } catch (InterruptedException e) {
          Thread.currentThread().interrupt();
        } finally {
          spillLock.unlock();
          spillThreadRunning = false;
        }
      }
    }

线程函数内的处理逻辑比较简单，主要分为三个步骤：

1.等待唤醒

2.对内存中的数据进行排序并将数据溢出写入到磁盘,这部分内部分析见下文。

3.重置索引区和缓存区的end标记

sortAndSpill

内存数据的溢出处理是有此函数进行封装，下面我们将该函数按块进行详细分析。

    private void sortAndSpill() throws IOException, ClassNotFoundException,
                                       InterruptedException {
      //approximate the length of the output file to be the length of the
      //buffer + header lengths for the partitions
      long size = (bufend >= bufstart
          ? bufend - bufstart
          : (bufvoid - bufend) + bufstart) +
                  partitions * APPROX_HEADER_LENGTH;
      FSDataOutputStream out = null;
      try {
		// part1
        // create spill file
        final SpillRecord spillRec = new SpillRecord(partitions);
        final Path filename =
            mapOutputFile.getSpillFileForWrite(numSpills, size);
        out = rfs.create(filename);
		
		// part2
        final int endPosition = (kvend > kvstart)
          ? kvend
          : kvoffsets.length + kvend;
        sorter.sort(MapOutputBuffer.this, kvstart, endPosition, reporter);
	
		
        int spindex = kvstart;
        IndexRecord rec = new IndexRecord();
        InMemValBytes value = new InMemValBytes();
        for (int i = 0; i < partitions; ++i) {
          IFile.Writer<K, V> writer = null;
          try {
			// part3
            long segmentStart = out.getPos();
            writer = new Writer<K, V>(job, out, keyClass, valClass, codec,
                                      spilledRecordsCounter);
			// part4
            if (combinerRunner == null) {
              // spill directly
              DataInputBuffer key = new DataInputBuffer();
              while (spindex < endPosition &&
                  kvindices[kvoffsets[spindex % kvoffsets.length]
                            + PARTITION] == i) {
                final int kvoff = kvoffsets[spindex % kvoffsets.length];
                getVBytesForOffset(kvoff, value);
                key.reset(kvbuffer, kvindices[kvoff + KEYSTART],
                          (kvindices[kvoff + VALSTART] - 
                           kvindices[kvoff + KEYSTART]));
                writer.append(key, value);
                ++spindex;
              }
            } else {
			// part5
              int spstart = spindex;
              while (spindex < endPosition &&
                  kvindices[kvoffsets[spindex % kvoffsets.length]
                            + PARTITION] == i) {
                ++spindex;
              }
              // Note: we would like to avoid the combiner if we've fewer
              // than some threshold of records for a partition
              if (spstart != spindex) {
                combineCollector.setWriter(writer);
                RawKeyValueIterator kvIter =
                  new MRResultIterator(spstart, spindex);
                combinerRunner.combine(kvIter, combineCollector);
              }
            }
		
			// part6
            // close the writer
            writer.close();
			
            // record offsets
            rec.startOffset = segmentStart;
            rec.rawLength = writer.getRawLength();
            rec.partLength = writer.getCompressedLength();
            spillRec.putIndex(rec, i);

            writer = null;
          } finally {
            if (null != writer) writer.close();
          }
        }
		
		// part7
        if (totalIndexCacheMemory >= INDEX_CACHE_MEMORY_LIMIT) {
          // create spill index file
          Path indexFilename =
              mapOutputFile.getSpillIndexFileForWrite(numSpills, partitions
                  * MAP_OUTPUT_INDEX_RECORD_LENGTH);
          spillRec.writeToFile(indexFilename, job);
        } else {
          indexCacheList.add(spillRec);
          totalIndexCacheMemory +=
            spillRec.size() * MAP_OUTPUT_INDEX_RECORD_LENGTH;
        }
        LOG.info("Finished spill " + numSpills);
        ++numSpills;
      } finally {
        if (out != null) out.close();
      }
    }

part1:创建SpillRecord,创建文件流

    SpillRecord是一个记录集，用于记录分区在数据文件中的文件起始位置，原始长度，压缩后的长度信息。

    SpillRecord的成员只有两个。一个是buf,长度为分区个数*每条分区索引信息占用的长度，另一个是为记录方便转换成的LogBuffer。

    每条分区索引信息占用的长度由MAP_OUTPUT_INDEX_RECORD_LENGTH来表示，占用24个字节，即3个Long。

  public SpillRecord(int numPartitions) {
    buf = ByteBuffer.allocate(
        numPartitions * MapTask.MAP_OUTPUT_INDEX_REC