hadoop-mapreduce中reducetask运行分析

ReduceTask的运行

Reduce处理程序中需要执行三个类型的处理，

1.copy,从各map中copy数据过来

2.sort,对数据进行排序操作。

3.reduce,执行业务逻辑的处理。

ReduceTask的运行也是通过run方法开始，

通过mapreduce.job.reduce.shuffle.consumer.plugin.class配置shuffle的plugin,

默认是Shuffle实现类。实现ShuffleConsumerPlugin接口。

生成Shuffle实例，并执行plugin的init函数进行初始化，

Class<?extendsShuffleConsumerPlugin>clazz =

job.getClass(MRConfig.SHUFFLE_CONSUMER_PLUGIN,Shuffle.class,ShuffleConsumerPlugin.class);

shuffleConsumerPlugin =ReflectionUtils.newInstance(clazz, job);

LOG.info("UsingShuffleConsumerPlugin: " +shuffleConsumerPlugin);

ShuffleConsumerPlugin.ContextshuffleContext =

newShuffleConsumerPlugin.Context(getTaskID(),job, FileSystem.getLocal(job),umbilical,

super.lDirAlloc,reporter, codec,

combinerClass,combineCollector,

spilledRecordsCounter,reduceCombineInputCounter,

shuffledMapsCounter,

reduceShuffleBytes,failedShuffleCounter,

mergedMapOutputsCounter,

taskStatus,copyPhase,sortPhase,this,

mapOutputFile,localMapFiles);

shuffleConsumerPlugin.init(shuffleContext);

执行shuffle的run函数，得到RawKeyValueIterator的实例。

rIter =shuffleConsumerPlugin.run();

Shuffle.run函数定义：

.....................................

inteventsPerReducer = Math.max(MIN_EVENTS_TO_FETCH,

MAX_RPC_OUTSTANDING_EVENTS/ jobConf.getNumReduceTasks());

intmaxEventsToFetch = Math.min(MAX_EVENTS_TO_FETCH,eventsPerReducer);

生成map的完成状态获取线程，并启动此线程，此线程中从am中获取此job中所有完成的map的event

通过ShuffleSchedulerImpl实例把所有的map的完成的map的host,mapid,

等记录到mapLocations容器中。此线程每一秒执行一个获取操作。

//Start the map-completion events fetcher thread

finalEventFetcher<K,V> eventFetcher =

newEventFetcher<K,V>(reduceId,umbilical,scheduler,this,

maxEventsToFetch);

eventFetcher.start();

下面看看EventFetcher.run函数的执行过程：以下代码中我只保留了代码的主体部分。

...................

EventFetcher.run:

publicvoid run() {

intfailures = 0;

........................

intnumNewMaps = getMapCompletionEvents();

..................................

}

......................

}

EventFetcher.getMapCompletionEvents

..................................

MapTaskCompletionEventsUpdateupdate =

umbilical.getMapCompletionEvents(

(org.apache.hadoop.mapred.JobID)reduce.getJobID(),

fromEventIdx,

maxEventsToFetch,

(org.apache.hadoop.mapred.TaskAttemptID)reduce);

events =update.getMapTaskCompletionEvents();

.....................

for(TaskCompletionEvent event : events) {

scheduler.resolve(event);

if(TaskCompletionEvent.Status.SUCCEEDED== event.getTaskStatus()) {

++numNewMaps;

}

}

shecduler是ShuffleShedulerImpl的实例。

ShuffleShedulerImpl.resolve

caseSUCCEEDED:

URI u = getBaseURI(reduceId,event.getTaskTrackerHttp());

addKnownMapOutput(u.getHost() +":"+ u.getPort(),

u.toString(),

event.getTaskAttemptId());

maxMapRuntime= Math.max(maxMapRuntime,event.getTaskRunTime());

break;

.......

ShuffleShedulerImpl.addKnownMapOutput函数：

把mapid与对应的host添加到mapLocations容器中，

MapHost host =mapLocations.get(hostName);

if(host == null){

host = newMapHost(hostName, hostUrl);

mapLocations.put(hostName,host);

}

此时会把host的状设置为PENDING

host.addKnownMap(mapId);

同时把host添加到pendingHosts容器中。notify相关的Fetcher文件copy线程。

//Mark the host as pending

if(host.getState() == State.PENDING){

pendingHosts.add(host);

notifyAll();

}

.....................

回到ReduceTask.run函数中，接着向下执行

//Start the map-output fetcher threads

booleanisLocal = localMapFiles!= null;

通过mapreduce.reduce.shuffle.parallelcopies配置的值，默认为5，生成获取map数据的线程数。

生成Fetcher线程实例，并启动相关的线程。

通过mapreduce.reduce.shuffle.connect.timeout配置连接超时时间。默认180000

通过mapreduce.reduce.shuffle.read.timeout配置读取超时时间，默认为180000

finalintnumFetchers = isLocal ? 1 :

jobConf.getInt(MRJobConfig.SHUFFLE_PARALLEL_COPIES,5);

Fetcher<K,V>[] fetchers =newFetcher[numFetchers];

if(isLocal) {

fetchers[0] = newLocalFetcher<K, V>(jobConf,reduceId,scheduler,

merger,reporter,metrics,this,reduceTask.getShuffleSecret(),

localMapFiles);

fetchers[0].start();

}else{

for(inti=0; i < numFetchers; ++i) {

fetchers[i] = newFetcher<K,V>(jobConf,reduceId,scheduler,merger,

reporter,metrics,this,

reduceTask.getShuffleSecret());

fetchers[i].start();

}

}

.........................

接下来进行Fetcher线程里面，看看Fetcher.run函数运行流程：

..........................

MapHost host = null;

try{

//If merge is on, block

merger.waitForResource();

从ShuffleScheduler中取出一个MapHost实例，

//Get a host to shuffle from

host = scheduler.getHost();

metrics.threadBusy();

执行shuffle操作。

//Shuffle

copyFromHost(host);

} finally{

if(host != null){

scheduler.freeHost(host);

metrics.threadFree();

}

}

接下来看看ShuffleScheduler中的getHost函数：

........

如果pendingHosts的值没有，先wait住，等待EventFetcher线程去获取数据来notify此wait

while(pendingHosts.isEmpty()){

wait();

}

MapHost host = null;

Iterator<MapHost> iter =pendingHosts.iterator();

从pendingHosts中random出一个MapHost，并返回给调用程序。

intnumToPick = random.nextInt(pendingHosts.size());

for(inti=0; i <= numToPick; ++i) {

host = iter.next();

}

pendingHosts.remove(host);

........................

当得到一个MapHost后，执行copyFromHost来进行数据的copy操作。

此时，一个task的host的url样子基本上是这个样子：

host:port/mapOutput?job=xxx&reduce=123(当前reduce的partid值)&map=

copyFromHost的代码部分：

.....

List<TaskAttemptID>maps = scheduler.getMapsForHost(host);

.....

Set<TaskAttemptID>remaining = newHashSet<TaskAttemptID>(maps);

.....

此部分完成后，url样子中map=后面会有很多个mapid，多个用英文的”,”号分开的。

URLurl = getMapOutputURL(host, maps);

此处根据url打开httpconnection,

如果mapreduce.shuffle.ssl.enabled配置为true时，会打开SSL连接。默认为false.

openConnection(url);

.....