且具体到底产生多少个分片(split) 因为多少个map 是有关系。(此处是根据新的API来分析,因为新的API 终究要调用到就得API来做具体的动作)
可能会说这个值 是系统根据文件大小 和根据文件分片大小 算出来的,那具体是如何算出来的呢,我们根据源码 一步一步来分析
首先Job.submit()
public void submit() throws IOException, InterruptedException,
ClassNotFoundException {
ensureState(JobState.DEFINE);
setUseNewAPI();
// Connect to the JobTracker and submit the job
connect();
info = jobClient.submitJobInternal(conf); //此处用到JobClient的submitJobInternal 方法 看下面源码2
super.setJobID(info.getID());
state = JobState.RUNNING;
}
源码2 JobClient.submitJobInternal()
我们看 此方法中的 下面一段
// Create the splits for the job
FileSystem fs = submitJobDir.getFileSystem(jobCopy);
LOG.debug("Creating splits at " + fs.makeQualified(submitJobDir));
int maps = writeSplits(context, submitJobDir);//在此处计算 具体有多少个map紧接着看下面源码3
jobCopy.setNumMapTasks(maps);
源码3 writeSplits()
private int writeSplits(org.apache.hadoop.mapreduce.JobContext job,
Path jobSubmitDir) throws IOException,
InterruptedException, ClassNotFoundException {
JobConf jConf = (JobConf)job.getConfiguration();
int maps;
if (jConf.getUseNewMapper()) {//新API
maps = writeNewSplits(job, jobSubmitDir); //见下面源码5
} else {//旧API
maps = writeOldSplits(jConf, jobSubmitDir);
}
return maps;
}
源码5 writeNewSplits()
int writeNewSplits(JobContext job, Path jobSubmitDir) throws IOException,
InterruptedException, ClassNotFoundException {
Configuration conf = job.getConfiguration();
InputFormat<?, ?> input =
ReflectionUtils.newInstance(job.getInputFormatClass(), conf);
List<InputSplit> splits = input.getSplits(job); //我们需要关注的是这一行 调用input 中的getSplits 方法 我们会用FileInputFormat的getSplits方法来做实例 看源码6
T[] array = (T[]) splits.toArray(new InputSplit[splits.size()]);
// sort the splits into order based on size, so that the biggest
// go first
Arrays.sort(array, new SplitComparator());
JobSplitWriter.createSplitFiles(jobSubmitDir, conf,
jobSubmitDir.getFileSystem(conf), array);
return array.length;
}
源码6 FileInputFormat.getSplits(JobConf job, int numSplits)
FileStatus[] files = listStatus(job);
// Save the number of input files in the job-conf
job.setLong(NUM_INPUT_FILES, files.length);
long totalSize = 0; // compute total size
for (FileStatus file: files) { // check we have valid files
if (file.isDir()) {
throw new IOException("Not a file: "+ file.getPath());
}
totalSize += file.getLen();
}
long goalSize = totalSize / (numSplits == 0 ? 1 : numSplits);
long minSize = Math.max(job.getLong("mapred.min.split.size", 1),
minSplitSize);
// generate splits
ArrayList<FileSplit> splits = new ArrayList<FileSplit>(numSplits);
NetworkTopology clusterMap = new NetworkTopology();
for (FileStatus file: files) {
Path path = file.getPath();
FileSystem fs = path.getFileSystem(job);
long length = file.getLen();
BlockLocation[] blkLocations = fs.getFileBlockLocations(file, 0, length);
if ((length != 0) && isSplitable(fs, path)) {
long blockSize = file.getBlockSize();
long splitSize = computeSplitSize(goalSize, minSize, blockSize);//此处计算 split size 从这可以看出来 是根据goal,minSize,blockSize 三个参数来计算的,那需要仔细看看上面三个参数的由来,再继续看源码 7
long bytesRemaining = length;
while (((double) bytesRemaining)/splitSize > SPLIT_SLOP) {
String[] splitHosts = getSplitHosts(blkLocations,
length-bytesRemaining, splitSize, clusterMap);
splits.add(new FileSplit(path, length-bytesRemaining, splitSize,
splitHosts));
bytesRemaining -= splitSize;
}
if (bytesRemaining != 0) {
splits.add(new FileSplit(path, length-bytesRemaining, bytesRemaining,
blkLocations[blkLocations.length-1].getHosts()));
}
} else if (length != 0) {
String[] splitHosts = getSplitHosts(blkLocations,0,length,clusterMap);
splits.add(new FileSplit(path, 0, length, splitHosts));
} else {
//Create empty hosts array for zero length files
splits.add(new FileSplit(path, 0, length, new String[0]));
}
}
LOG.debug("Total # of splits: " + splits.size());
return splits.toArray(new FileSplit[splits.size()]);
源码7 computeSplitSize
protected long computeSplitSize(long goalSize, long minSize,
long blockSize) {
return Math.max(minSize, Math.min(goalSize, blockSize)); 具体计算公式如下,可以看出
}
所以从上面源码的解读,我们可以看出来,这个参数 mapred.map.tasks 的设置对具体的mapreduce 对输入进行分片产生一定的作用,因为具体产生多少分片,多少个map
是根据三个参数来决定的 一个是dfs.block.size 另外一个是mapred.map.tasks 还有一个 就是 mapred.min.split.size 但一般情况下如果不设置 mapred.map.tasks 的情况下 则会根据其它两个参数来决定,但一般情况下 mapred.min.split.size 参数我们也不设置,所以 dfs.block.size 自然就是我们默认的分片大小,如果mapred.min.split.size 大于dfs.block.size 则系统分片就会大于文件系统 块的大小,从而map的个数也会相应的减少。