Azkaban的Exec Server分析 29：JobRunner的真正执行过程(前奏)

2019独角兽企业重金招聘Python工程师标准>>>

重点来了，现在看看Job Runner的运行过程！

==========================================================================================

jdb azkaban.execapp.AzkabanExecutorServer  -conf  /root/azkb/azkaban_3.0.0_debug/conf

stop in  azkaban.execapp.JobRunner.run

run

==========================================================================================

之前做了一些校验工作，然后

fireEvent(Event.create(this, Type.JOB_STARTED, null, false));

让我们来看有哪些Listeners

JobRunner-Hello-17[1] print listeners

 listeners = "[

azkaban.execapp.FlowRunner$JobRunnerEventListener@2a32e7d1, 

azkaban.execapp.event.JobCallbackManager@2b274bce, 

azkaban.execapp.jmx.JmxJobMBeanManager@53976f5c

]"

那么，这3个Listener分别做了什么事情呢？

==========================================================================================

第一个，不好意思，这里是start,所以下面的代码都没有执行！

private class JobRunnerEventListener implements EventListener {

public JobRunnerEventListener() {

}

 

@Override

public synchronized void handleEvent(Event event) {

//看到这里了

JobRunner runner = (JobRunner) event.getRunner();

//看到这里了

if (event.getType() == Type.JOB_STATUS_CHANGED) {

updateFlow();

} else if (event.getType() == Type.JOB_FINISHED) {

ExecutableNode node = runner.getNode();

long seconds = (node.getEndTime() - node.getStartTime()) / 1000;

synchronized (mainSyncObj) {

logger.info("Job " + node.getNestedId() + " finished with status " + node.getStatus() + " in "

+ seconds + " seconds");

 

// Cancellation is handled in the main thread, but if the

// flow is

// paused, the main thread is paused too.

// This unpauses the flow for cancellation.

if (flowPaused && node.getStatus() == Status.FAILED && failureAction == FailureAction.CANCEL_ALL) {

flowPaused = false;

}

 

finishedNodes.add(node);

node.getParentFlow().setUpdateTime(System.currentTimeMillis());

interrupt();

fireEventListeners(event);

}

}

//看到这里了

}

==========================================================================================

第二个呢？

@Override

public void handleEvent(Event event) {

// 看到这里了

if (!isInitialized) {

return;

}

 

if (event.getRunner() instanceof JobRunner) {

// 看到这里了

try {

if (event.getType() == Event.Type.JOB_STARTED) {

// 这里开始执行

processJobCallOnStart(event);

} else if (event.getType() == Event.Type.JOB_FINISHED) {

processJobCallOnFinish(event);

}

} catch (Throwable e) {

// Use job runner logger so user can see the issue in their job

// log

JobRunner jobRunner = (JobRunner) event.getRunner();

jobRunner.getLogger().error("Encountered error while hanlding job callback event", e);

}

} else {

logger.warn("((( Got an unsupported runner: " + event.getRunner().getClass().getName() + " )))");

}

 

}

进去debug,发现还是什么都没有执行。。。怪我咯？

==========================================================================================

JmxJobMBeanManager

这个Listener具体做了啥呢？

if (event.getType() == Event.Type.JOB_STARTED) {

// 只是增加了一个计数器的值。。。

runningJobCount.incrementAndGet();

紧接着

loader.uploadExecutableNode(node, props);

这个到底干嘛的？？？

final String INSERT_EXECUTION_NODE = "INSERT INTO execution_jobs "

+ "(exec_id, project_id, version, flow_id, job_id, start_time, "

+ "end_time, status, input_params, attempt) VALUES (?,?,?,?,?,?,?,?,?,?)";

执行了这个语句先！

==========================================================================================

stop in azkaban.execapp.JobRunner.insertJobMetadata

==========================================================================================

 下面会构造一个JVM参数

/**

 * Add useful JVM arguments so it is easier to map a running Java process to

 * a flow, execution id and job

 */

private void insertJVMAargs() {

// 看到这里了

String flowName = node.getParentFlow().getFlowId();//

String jobId = node.getId();

// "-Dazkaban.flowid=World -Dazkaban.execid=17 -Dazkaban.jobid=Hello"

 

String jobJVMArgs = String.format("-Dazkaban.flowid=%s -Dazkaban.execid=%s -Dazkaban.jobid=%s", flowName,

executionId, jobId);

// "-Dazkaban.flowid=World -Dazkaban.execid=17 -Dazkaban.jobid=Hello"

 

String previousJVMArgs = props.get(JavaProcessJob.JVM_PARAMS);

jobJVMArgs += (previousJVMArgs == null) ? "" : " " + previousJVMArgs;

//存起来

logger.info("job JVM args: " + jobJVMArgs);

props.put(JavaProcessJob.JVM_PARAMS, jobJVMArgs);

}

==========================================================================================

然后发现，JobRunner实例，需要构造一个具体的job对象

这是在

job = jobtypeManager.buildJobExecutor(this.jobId, props, logger); 实现的

那么到底怎么构造这个对象呢？

鉴于这部分比较重要，我还是另开一个小节吧。

转载于:https://my.oschina.net/qiangzigege/blog/656937