hadoop源码阅读（三）（InputFormat切片源码）

从FileInputFormat类进入，因为在FileInputFormat类中重写了父类InputFormat的getSplits方法，在long minSize = Math.max(getFormatMinSplitSize(), getMinSplitSize(job));处打断点，进入

/**
     * Returns the greater of two {@code int} values. That is, the
     * result is the argument closer to the value of
     * {@link Integer#MAX_VALUE}. If the arguments have the same value,
     * the result is that same value.
     *
     * @param   a   an argument.
     * @param   b   another argument.
     * @return  the larger of {@code a} and {@code b}.
     */
    @IntrinsicCandidate
    public static int max(int a, int b) {
        return (a >= b) ? a : b;
    }

    /**
     * Returns the greater of two {@code long} values. That is, the
     * result is the argument closer to the value of
     * {@link Long#MAX_VALUE}. If the arguments have the same value,
     * the result is that same value.
     *
     * @param   a   an argument.
     * @param   b   another argument.
     * @return  the larger of {@code a} and {@code b}.
     */
    public static long max(long a, long b) {
        return (a >= b) ? a : b;
    }

max方法返回getFormatMinSplitSize()或getMinSplitSize(job)参数中的最大值

//getFormatMinSplitSize()方法固定返回值为1
protected long getFormatMinSplitSize() {
    return 1;
  }

//getMinSplitSize(JobContext job)方法
//如果配置了mapreduce.input.fileinputformat.split.maxsize则返回配置值，否则返回默认值1L
public static long getMinSplitSize(JobContext job) {
    return job.getConfiguration().getLong(SPLIT_MINSIZE, 1L);
  }

public static final String SPLIT_MAXSIZE = 
    "mapreduce.input.fileinputformat.split.maxsize";

同理，在getMaxSplitSize(job)方法中

//如果未配置mapreduce.input.fileinputformat.split.maxsize则返回Long.MAX_VALUE的值,如果配置了就使用配置值
public static long getMaxSplitSize(JobContext context) {
    return context.getConfiguration().getLong(SPLIT_MAXSIZE, 
                                              Long.MAX_VALUE);
  }

//SPLIT_MAXSIZE
public static final String SPLIT_MAXSIZE = 
    "mapreduce.input.fileinputformat.split.maxsize";

//Long.MAX_VALUE值
@Native public static final long MAX_VALUE = 0x7fffffffffffffffL;

具体计算切片大小细节：

//如果未设置minSize默认为1，maxSize默认Long的最大值
//Math.min(maxSize,blockSize)返回的是blockSize，整个计算切片方法返回的是blockSize
//总结: 默认情况下 片大小 = 块大小
//要使片大小 > 块大小，则需要设置minSize>128m
//要使片大小 < 块大小，则需要设置maxSize<128m
protected long computeSplitSize(long blockSize, long minSize,
                                  long maxSize) {
    return Math.max(minSize, Math.min(maxSize, blockSize));
  }

案例：
输入文件
File1 350m
File2 100m

输出文件
File1.split1 0–128m
File2.split2 128m-256m
File3.split3 256m–350m
File4.split4 100m

//若上面那个除后得到的值没有大于1.1走这个方法，将最后一个文件一整个输出(不切)>>可以忽略跨节点问题
if (bytesRemaining != 0) {
            int blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);
            splits.add(makeSplit(path, length-bytesRemaining, bytesRemaining,
                       blkLocations[blkIndex].getHosts(),
                       blkLocations[blkIndex].getCachedHosts()));
          }

走完最终将生成的切片放入到splits集合中.

XMind图后面补