切片源码
从FileInputFormat类进入,因为在FileInputFormat类中重写了父类InputFormat的getSplits方法,在
long minSize = Math.max(getFormatMinSplitSize(), getMinSplitSize(job));
处打断点,进入
/**
* Returns the greater of two {@code int} values. That is, the
* result is the argument closer to the value of
* {@link Integer#MAX_VALUE}. If the arguments have the same value,
* the result is that same value.
*
* @param a an argument.
* @param b another argument.
* @return the larger of {@code a} and {@code b}.
*/
@IntrinsicCandidate
public static int max(int a, int b) {
return (a >= b) ? a : b;
}
/**
* Returns the greater of two {@code long} values. That is, the
* result is the argument closer to the value of
* {@link Long#MAX_VALUE}. If the arguments have the same value,
* the result is that same value.
*
* @param a an argument.
* @param b another argument.
* @return the larger of {@code a} and {@code b}.
*/
public static long max(long a, long b) {
return (a >= b) ? a : b;
}
max方法返回getFormatMinSplitSize()或getMinSplitSize(job)参数中的最大值
//getFormatMinSplitSize()方法固定返回值为1
protected long getFormatMinSplitSize() {
return 1;
}
//getMinSplitSize(JobContext job)方法
//如果配置了mapreduce.input.fileinputformat.split.maxsize则返回配置值,否则返回默认值1L
public static long getMinSplitSize(JobContext job) {
return job.getConfiguration().getLong(SPLIT_MINSIZE, 1L);
}
public static final String SPLIT_MAXSIZE =
"mapreduce.input.fileinputformat.split.maxsize";
同理,在getMaxSplitSize(job)方法中
//如果未配置mapreduce.input.fileinputformat.split.maxsize则返回Long.MAX_VALUE的值,如果配置了就使用配置值
public static long getMaxSplitSize(JobContext context) {
return context.getConfiguration().getLong(SPLIT_MAXSIZE,
Long.MAX_VALUE);
}
//SPLIT_MAXSIZE
public static final String SPLIT_MAXSIZE =
"mapreduce.input.fileinputformat.split.maxsize";
//Long.MAX_VALUE值
@Native public static final long MAX_VALUE = 0x7fffffffffffffffL;
具体计算切片大小细节:
//如果未设置minSize默认为1,maxSize默认Long的最大值
//Math.min(maxSize,blockSize)返回的是blockSize,整个计算切片方法返回的是blockSize
//总结: 默认情况下 片大小 = 块大小
//要使片大小 > 块大小,则需要设置minSize>128m
//要使片大小 < 块大小,则需要设置maxSize<128m
protected long computeSplitSize(long blockSize, long minSize,
long maxSize) {
return Math.max(minSize, Math.min(maxSize, blockSize));
}
案例:
输入文件
File1 350m
File2 100m
输出文件
File1.split1 0–128m
File2.split2 128m-256m
File3.split3 256m–350m
File4.split4 100m
//若上面那个除后得到的值没有大于1.1走这个方法,将最后一个文件一整个输出(不切)>>可以忽略跨节点问题
if (bytesRemaining != 0) {
int blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);
splits.add(makeSplit(path, length-bytesRemaining, bytesRemaining,
blkLocations[blkIndex].getHosts(),
blkLocations[blkIndex].getCachedHosts()));
}
走完最终将生成的切片放入到splits集合中.
XMind图后面补