hadoop之mapreduce

先看入口  JobClient.runJob(conf);这个函数，之后submitJob，最后调用submitJobInternal方法，接下来看这个方法// jobSubmitClient是JobSubmissionProtocol协议，这个实际上调用的是子类jobTracker的getNewJobId JobID jobId = jobSubmitClient.getNewJobId(); // getSystemDir()得到mapreduce的共享目录，即mapred.system.dir配置的目录 Path submitJobDir = new Path(getSystemDir(), jobId.toString()); Path submitJarFile = new Path(submitJobDir, "job.jar"); Path submitSplitFile = new Path(submitJobDir, "job.split"); // 将一些任务需要的文件jar包传到hdfs上 configureCommandLineOptions(job, submitJobDir, submitJarFile); // 构造任务的配置文件 Path submitJobFile = new Path(submitJobDir, "job.xml"); int reduces = job.getNumReduceTasks(); JobContext context = new JobContext(job, jobId); // 检查输出目录是否存在 if (reduces == 0 ? job.getUseNewMapper() : job.getUseNewReducer()) { org.apache.hadoop.mapreduce.OutputFormat<?, ?> output = ReflectionUtils.newInstance( context.getOutputFormatClass(), job); output.checkOutputSpecs(context); } else { job.getOutputFormat().checkOutputSpecs(fs, job); } // Create the splits for the job LOG.debug("Creating splits at " + fs.makeQualified(submitSplitFile)); int maps; // 写分割文件 if (job.getUseNewMapper()) { maps = writeNewSplits(context, submitSplitFile); } else { maps = writeOldSplits(job, submitSplitFile); } job.set("mapred.job.split.file", submitSplitFile.toString()); job.setNumMapTasks(maps); // Write job file to JobTracker's fs FSDataOutputStream out = FileSystem.create(fs, submitJobFile, new FsPermission(JOB_FILE_PERMISSION)); try { // 写配置文件 job.writeXml(out); } finally { out.close(); } // // Now, actually submit the job (using the submit name) // 发起RPC调用，调用jobTracker的submitJob JobStatus status = jobSubmitClient.submitJob(jobId); if (status != null) { return new NetworkedJob(status); } else { throw new IOException("Could not launch job"); }

接下来就看jobTracker的submitJob方法

首先构造了一个处理中的job，JobInProgress job = new JobInProgress(jobId, this, this.conf);

该构造函数主要是把一些jar包拷到本地，并且设置一些任务属性。接下来是一系列合法性校验

1 任务对应的队列是否存在

2 权限检查

3 内存检查

如果校验不通过则加入到清理队列，销毁该任务，同时抛出异常

接受调用 private synchronized JobStatus addJob(JobID jobId, JobInProgress job) 方法

该方法主要是通过EagerTaskInitializationListener的jobAdded方法

jobInitQueue.add(job); 加入初始化队列
 resortInitQueue(); 根据优先级和创建时间排序

这样客户端的任务就提交成功了

接下来的任务就是怎么把该job拆分成Task

很明显肯定是一个线程去处理这个初始化队列，把job转换成task，接下来具体看

 EagerTaskInitializationListener这个类，它有以下内部类 JobInitManager管理JobInit，InitJob负责job分割成任务

 

class JobInitManager implements Runnable { public void run() { JobInProgress job = null; while (true) { try { synchronized (jobInitQueue) { while (jobInitQueue.isEmpty()) { jobInitQueue.wait(); } job = jobInitQueue.remove(0); } threadPool.execute(new InitJob(job)); //job初始化为任务 } catch (InterruptedException t) { LOG.info("JobInitManagerThread interrupted."); break; } } LOG.info("Shutting down thread pool"); threadPool.shutdownNow(); } } class InitJob implements Runnable { private JobInProgress job; public InitJob(JobInProgress job) { this.job = job; } public void run() { ttm.initJob(job); //ttm是TaskTrackerManager 真正执行的是其子类JobTracker，JobTracker的管理者 }

 

 

 接下来看JobTracker的initJob方法，主要是调用了JobInProgress 的initTasks方法

该方法主要是把job拆分成很多的TaskInProgress 包括map reduce任务

 

回过来看是谁启动了初始化job的线程，看EagerTaskInitializationListener的start方法，以后台线程的形式跑初始化job

public void start() throws IOException { this.jobInitManagerThread = new Thread(jobInitManager, "jobInitManager"); jobInitManagerThread.setDaemon(true); this.jobInitManagerThread.start(); }

 

接下来看最最开始的入口jobTracker的offerService方法

 

public void offerService() throws InterruptedException, IOException { // Prepare for recovery. This is done irrespective of the status of restart // flag. while (true) { try { recoveryManager.updateRestartCount();//记录重启次数 break; } catch (IOException ioe) { LOG.warn("Failed to initialize recovery manager. ", ioe); // wait for some time Thread.sleep(FS_ACCESS_RETRY_PERIOD); LOG.warn("Retrying..."); } } taskScheduler.start(); //启动任务调度器 // Start the recovery after starting the scheduler try { recoveryManager.recover(); //恢复任务 } catch (Throwable t) { LOG.warn("Recovery manager crashed! Ignoring.", t); } this.expireTrackersThread = new Thread(this.expireTrackers, "expireTrackers"); this.expireTrackersThread.start(); //启动检测tacker的线程 this.retireJobsThread = new Thread(this.retireJobs, "retireJobs"); this.retireJobsThread.start(); //启动移除已完成job expireLaunchingTaskThread.start(); //处理任务处理超时 if (completedJobStatusStore.isActive()) { completedJobsStoreThread = new Thread(completedJobStatusStore, "completedjobsStore-housekeeper"); completedJobsStoreThread.start(); } // start the inter-tracker server once the jt is ready this.interTrackerServer.start(); //启动rpc服务 synchronized (this) { state = State.RUNNING; } LOG.info("Starting RUNNING"); this.interTrackerServer.join(); LOG.info("Stopped interTrackerServer"); }

 

接下来主要就是看 taskScheduler.start();了，主要有LimitTasksPerJobTaskScheduler  JobQueueTaskScheduler和TaskScheduler 后者是前者的父类，最后在JobQueueTaskScheduler的start方法中启动了 eagerTaskInitializationListener.start();即启动的分割job的线程