Hadoop学习（9）----MapReduce阶段的Split阶段

一、介绍

在上一届的信息中我们已经知道MapReduce框架，在处理过程中主要分为四个阶段：Split（分片）阶段、Map阶段、Shuffle（混排、重拍）阶段、Reduce阶段。接下来笔者将会分别详细介绍着四个阶段，也会加上Hadoop2.6.0的源码进行分析。从而加深读者对Split阶段的理解。

我们知道从文件开始读取，经过一些列处理后，文件数据将以<key,value>键值对的方式进入到Map阶段中，这就是Split的主要任务，下面我们详细介绍这一些类操作。

二、Split介绍

想要了解Split的过程，就必须要了解三个类：FileSplit.class、InputFormat.class和LineRecordReader.class

(1)org.apache.hadoop.mapreduce.lib.input.FileSplit

public class FileSplit extends InputSplit implements Writable {
  private Path file;
  private long start;
  private long length;
  private String[] hosts;
  private SplitLocationInfo[] hostInfos;

  public FileSplit() {}
  public FileSplit(Path file, long start, long length, String[] hosts) {
    this.file = file;
    this.start = start;
    this.length = length;
    this.hosts = hosts;
  }
 public FileSplit(Path file, long start, long length, String[] hosts,
     String[] inMemoryHosts) {
   this(file, start, length, hosts);
   hostInfos = new SplitLocationInfo[hosts.length];
   for (int i = 0; i < hosts.length; i++) {
     // because N will be tiny, scanning is probably faster than a HashSet
     boolean inMemory = false;
     for (String inMemoryHost : inMemoryHosts) {
       if (inMemoryHost.equals(hosts[i])) {
         inMemory = true;
         break;
       }
     }
     hostInfos[i] = new SplitLocationInfo(hosts[i], inMemory);
   }
 }
 
  /** The file containing this split's data. */
  public Path getPath() { return file; }
  
  /** The position of the first byte in the file to process. */
  public long getStart() { return start; }
  
  /** The number of bytes in the file to process. */
  @Override
  public long getLength() { return length; }

  @Override
  public String toString() { return file + ":" + start + "+" + length; }

  
  // Writable methods
  
  @Override
  public String[] getLocations() throws IOException {
    if (this.hosts == null) {
      return new String[]{};
    } else {
      return this.hosts;
    }
  }
  
  @Override
  @Evolving
  public SplitLocationInfo[] getLocationInfo() throws IOException {
    return hostInfos;
  }
}

这个类是从Hadoop2.6源代码中复制出来的，并且删除了一部分代码，从该类中我们看出，一个Split分片的内容主要包括Path file、Long start、Long length和String[] hosts。从这四个可以看出Split分片并没有实际的数据，分片其实只是对一个文件进行逻辑上的分片，数据还是按照Block的方式保存在HDFS中，而一个Split分片的主要记录了该分片是从文件的那个位置开始，长度是多少，这些数据的位置在哪里这些信息，在读取分片数据的时候，是根据FileSplit类中的信息去读取相应的Block的数据。这也是为什么分片最好和Block大小相同的原因，如果一个FileSplit的大小大于一个Block的大小，则该分片可能会需要从其他节点的Block读取数据，这样就会造成不必要的网络传输，导致处理时间增长。

(2)org.apache.hadoop.mapreduce.InputFormat

public abstract class InputFormat<K, V> {

  
  public abstract 
    List<InputSplit> getSplits(JobContext context
                               ) throws IOException, InterruptedException;
  
 
  public abstract 
    RecordReader<K,V> createRecordReader(InputSplit split,
                                         TaskAttemptContext context
                                        ) throws IOException, 
                                                 InterruptedException;

 }

我们在实现读取分片的过程就有这两个函数完成，首先我们先介绍getSplit()函数

public InputSplit[] getSplits(JobConf job, int numSplits)
    throws IOException {
    Stopwatch sw = new Stopwatch().start();
    FileStatus[] files = listStatus(job);//获取Job中所有文件是状态信息（filestatus类）
    
    job.setLong(NUM_INPUT_FILES, files.length);   //设置输入文件数量为文件状态的个数
    long totalSize = 0;                           // 计算job中所有文件的总大小
    for (FileStatus file: files) {                // 便利所有的文件，检查是否含有文件
      if (file.isDirectory()) {
        throw new IOException("Not a file: "+ file.getPath());
      }
      totalSize += file.getLen();
    }

    long goalSize = totalSize / (numSplits == 0 ? 1 : numSplits);//根据map数量，计算每个map的处理量
    long minSize = Math.max(job.getLong(org.apache.hadoop.mapreduce.lib.input.
      FileInputFormat.SPLIT_MINSIZE, 1), minSplitSize); //获取分片大小的最大值

    // generate splits
    ArrayList<FileSplit> splits = new ArrayList<FileSplit>(numSplits);//根据map的数量创建分片数组
    NetworkTopology clusterMap = new NetworkTopology();
    for (FileStatus file: files) {    //对每一个文件进行便利
      Path path = file.getPath();    //获取当前文件的路径
      long length = file.getLen();   //获取当前文件的大小
      if (length != 0) {  //在文件不为空的情况下
        FileSystem fs = path.getFileSystem(job);   //创建FileSystem示例
        BlockLocation[] blkLocations;     //创建BlockLocation数组，准备记录该文件所有Block的位置
        if (file instanceof LocatedFileStatus) {   //整个判断语句就是用来获取该文件Block的位置信息
          blkLocations = ((LocatedFileStatus) file).getBlockLocations();
        } else {
          blkLocations = fs.getFileBlockLocations(file, 0, length);
        }
        if (isSplitable(fs, path)) {  //isSplitable为true时，表示一个分片中只允许一个文件，false相反
          long blockSize = file.getBlockSize(); //获取文件的Block的大小
          long splitSize = computeSplitSize(goalSize, minSize, blockSize);//计算一个分片的实际大小

          long bytesRemaining = length; //文件剩余大小
          while (((double) bytesRemaining)/splitSize > SPLIT_SLOP) { //
            String[][] splitHosts = getSplitHostsAndCachedHosts(blkLocations,
                length-bytesRemaining, splitSize, clusterMap); //根据Block位置信息、split大小等信息进行计算split的IP地址
            splits.add(makeSplit(path, length-bytesRemaining, splitSize, //将新产生的Split加入到split的List中
                splitHosts[0], splitHosts[1]));
            bytesRemaining -= splitSize; //减小文件所剩的大小
          }

          if (bytesRemaining != 0) {  //当文件还有剩余量时，经剩余的文件放入到新的分片中
            String[][] splitHosts = getSplitHostsAndCachedHosts(blkLocations, length
                - bytesRemaining, bytesRemaining, clusterMap);
            splits.add(makeSplit(path, length - bytesRemaining, bytesRemaining,
                splitHosts[0], splitHosts[1]));
          }
        } else { //isSplitable==false时，则一个文件可以不被分片，将文件放入到一个split中
          String[][] splitHosts = getSplitHostsAndCachedHosts(blkLocations,0,length,clusterMap);//获取split所在的IP地址
          splits.add(makeSplit(path, 0, length, splitHosts[0], splitHosts[1])); //放入splits的List中
        }
      } else { 
        //Create empty hosts array for zero length files
        splits.add(makeSplit(path, 0, length, new String[0]));
      }
    }
    sw.stop();
    if (LOG.isDebugEnabled()) {
      LOG.debug("Total # of splits generated by getSplits: " + splits.size()
          + ", TimeTaken: " + sw.elapsedMillis());
    }
    return splits.toArray(new FileSplit[splits.size()]);
  }

通过getSplit()函数得到对文件的分片信息后，然后读取分片表示的数据，并生成<key,value>键值对送入到map端，而生成<key,value>键值对则是由createRecordReader()函数完成。

<span style="font-size:18px;"> public RecordReader<LongWritable, Text> 
    createRecordReader(InputSplit split,
                       TaskAttemptContext context) {
    String delimiter = context.getConfiguration().get(
        "textinputformat.record.delimiter"); //读取文件的默认分隔符
    byte[] recordDelimiterBytes = null;
    if (null != delimiter)
      recordDelimiterBytes = delimiter.getBytes(Charsets.UTF_8);
    return new LineRecordReader(recordDelimiterBytes); //调用LineRecordReader方法，生成<key,vavle>键值对
 }</span>

这样通过InputFormat类中的两个方法，我们就完成了对文件的分片和读取<key,value>值。

(3)org.apache.hadoop.mapreduce.lib.input.LineRecordReader

<span style="font-size:18px;">public class LineRecordReader extends RecordReader<LongWritable, Text> {
  private static final Log LOG = LogFactory.getLog(LineRecordReader.class);
  public static final String MAX_LINE_LENGTH = 
    "mapreduce.input.linerecordreader.line.maxlength";

  private long start;
  private long pos;
  private long end;
  private SplitLineReader in;
  private FSDataInputStream fileIn;
  private Seekable filePosition;
  private int maxLineLength;
  private LongWritable key;
  private Text value;
  private boolean isCompressedInput;
  private Decompressor decompressor;
  private byte[] recordDelimiterBytes;

  public LineRecordReader() {
  }

  public LineRecordReader(byte[] recordDelimiter) {
    this.recordDelimiterBytes = recordDelimiter;
  }

  public void initialize(InputSplit genericSplit,
                         TaskAttemptContext context) throws IOException { //初始化信息
    FileSplit split = (FileSplit) genericSplit;
    Configuration job = context.getConfiguration();
    this.maxLineLength = job.getInt(MAX_LINE_LENGTH, Integer.MAX_VALUE);
    start = split.getStart();
    end = start + split.getLength();
    final Path file = split.getPath();

    // open the file and seek to the start of the split
    final FileSystem fs = file.getFileSystem(job);
    fileIn = fs.open(file);
    
    CompressionCodec codec = new CompressionCodecFactory(job).getCodec(file);
    if (null!=codec) {
      isCompressedInput = true;	
      decompressor = CodecPool.getDecompressor(codec);
      if (codec instanceof SplittableCompressionCodec) {
        final SplitCompressionInputStream cIn =
          ((SplittableCompressionCodec)codec).createInputStream(
            fileIn, decompressor, start, end,
            SplittableCompressionCodec.READ_MODE.BYBLOCK);
        in = new CompressedSplitLineReader(cIn, job,
            this.recordDelimiterBytes);
        start = cIn.getAdjustedStart();
        end = cIn.getAdjustedEnd();
        filePosition = cIn;
      } else {
        in = new SplitLineReader(codec.createInputStream(fileIn,
            decompressor), job, this.recordDelimiterBytes);
        filePosition = fileIn;
      }
    } else {
      fileIn.seek(start);
      in = new SplitLineReader(fileIn, job, this.recordDelimiterBytes);
      filePosition = fileIn;
    }
    // If this is not the first split, we always throw away first record
    // because we always (except the last split) read one extra line in
    // next() method.
    if (start != 0) {
      start += in.readLine(new Text(), 0, maxBytesToConsume(start));
    }
    this.pos = start;
  }
 
  public boolean nextKeyValue() throws IOException { //将split分片数据生成<key,value>键值对
    if (key == null) {
      key = new LongWritable();
    }
    key.set(pos); //将key进行赋值，赋值为所在文件的位置
    if (value == null) {
      value = new Text();
    }
    int newSize = 0;
    // We always read one extra line, which lies outside the upper
    // split limit i.e. (end - 1)
    while (getFilePosition() <= end || in.needAdditionalRecordAfterSplit()) { 
      if (pos == 0) {
        newSize = skipUtfByteOrderMark(); //生成一行数据的大小
      } else {
        newSize = in.readLine(value, maxLineLength, maxBytesToConsume(pos)); //读取一行一行数据作为value的值
        pos += newSize; //增加位置信息
      }

      if ((newSize == 0) || (newSize < maxLineLength)) {
        break;
      }

      // line too long. try again
      LOG.info("Skipped line of size " + newSize + " at pos " + 
               (pos - newSize));
    }
    if (newSize == 0) {
      key = null;
      value = null;
      return false;
    } else {
      return true;
    }
  }
}</span>

当LineRecordReader方法每一次读取一行时，便执行一次nextkeyvalue方法，当成功生成一个<key,value>键值对后，nextkeyvalue方法返回true值，这是新得到的key和value存放在LineRecordReader对象中的key和value属性中，就可以进行读取了。当nextkeyvalue()方法将所有的数据读取结束后，就表示一个split中的所有数据被读取到map中。

三 总结

通过上述三个类以及相应的方法，试下了将数据从Block中读取，并生成<key,value>键值对过程：

getSplit()---->splits----->createRecordReader----->nextKeyValue()-----><key,value>键值对

当我们实现自己定制的Split方法时，通常是重写上面三个类中相应的方法，从而实现新的功能