目录
一、介绍
上一章xxl-job源码分析之---- 搭建项目demo简单介绍了搭建相关的步骤，本章主要介绍xxl-job 的核心jar（xxl-jar-core）,这里用的版本是最新的 xxl-job-2.2.0,更新比较快，不同版本可能逻辑略有不同
二、流程分析(Spring 框架)
在项目里面需要注入XxlJobSpringExecutor.class 这个类, 就从这个类开始分析，首先其类的层次结构如下,
XxlJobSpringExecutor 继承了 XxlJobExecutor，实现了 ApplicationContextAware， DisposableBean，SmartInitializingSingleton
可以看出 ，其实整个逻辑是在类XxlJobExecutor 里面，类XxlJobSpringExecutor 是为了匹配Spring 框架而做的改造
实现ApplicationContextAware 是为了获取上下文
实现DisposableBean是为了重写destroy 方法，用于释放一些资源
实现SmartInitializingSingleton，这个比较关键，也是重点，就是在Bean 创建的生命周期里面，单例 bean 都初始化完成以后，找出带有注解的 job进行注册等一系列操作
2.1 afterSingletonsInstantiated 里面逻辑
afterSingletonsInstantiated 方法里面主要做了三件事：
- 初始化JobHandler ，就是找到 带有注解@XxlJob的 Job ,进行注册
- 刷新GlueFactory
- 调用父类start() 方法，上面说过XxlJobExecutor 里面的才是核心逻辑
public void afterSingletonsInstantiated() {
// 初始化JobHandler ，就是找到 带有注解@XxlJob的 Job ,进行注册
initJobHandlerMethodRepository(applicationContext);
// 刷新GlueFactory, 这里type 为 0/1
// 1就是说明使用的是Spring 框架
GlueFactory.refreshInstance(1);
//启动
try {
super.start();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
2.2 XxlJobSpringExecutor#initJobHandlerMethodRepository
initJobHandlerMethodRepository 方法 主要做了以下几件事：
- 获取到所有的注册的beanName
- 针对每一个bean 进行扫描，获取带有@XxlJob 注解的方法，涉及一个工具类工具类MethodIntrospector
- 对每一个获带有@XxlJob 注解的方法进行校验
这里进行一些校验：
1. job 名字不能为空
2. job 名字不能重复
3. 入参必须要用，而且只能有1个， 还要是String 类型
4. 返回类型 必须要是ReturnT.class - 获取 初始化方法init() 和 销毁方法 destroy()
- 放入Map ,后续进行注册
private void initJobHandlerMethodRepository(ApplicationContext applicationContext) {
if (applicationContext == null) {
return;
}
//获取到所有的注册的beanName
String[] beanDefinitionNames = applicationContext.getBeanNamesForType(Object.class, false, true);
for (String beanDefinitionName : beanDefinitionNames) {
Object bean = applicationContext.getBean(beanDefinitionName);
Map<Method, XxlJob> annotatedMethods = null; // referred to ：org.springframework.context.event.EventListenerMethodProcessor.processBean
// 针对每一个bean 进行扫描，获取带有@XxlJob 注解的方法
try {
annotatedMethods = MethodIntrospector.selectMethods(bean.getClass(),
new MethodIntrospector.MetadataLookup<XxlJob>() {
@Override
public XxlJob inspect(Method method) {
return AnnotatedElementUtils.findMergedAnnotation(method, XxlJob.class);
}
});
} catch (Throwable ex) {
logger.error("xxl-job method-jobhandler resolve error for bean[" + beanDefinitionName + "].", ex);
}
if (annotatedMethods==null || annotatedMethods.isEmpty()) {
continue;
}
// 对获取到的带有@XxlJob 注解的方法进行处理
for (Map.Entry<Method, XxlJob> methodXxlJobEntry : annotatedMethods.entrySet()) {
Method method = methodXxlJobEntry.getKey();
XxlJob xxlJob = methodXxlJobEntry.getValue();
if (xxlJob == null) {
continue;
}
/**
这里进行一些校验，
1. job 名字不能为空
2. job 名字不能重复
3. 入参必须要用，而且只能有1个， 还要是String 类型
4. 返回类型 必须要是ReturnT.class
**/
String name = xxlJob.value();
if (name.trim().length() == 0) {
throw new RuntimeException("xxl-job method-jobhandler name invalid, for[" + bean.getClass() + "#" + method.getName() + "] .");
}
if (loadJobHandler(name) != null) {
throw new RuntimeException("xxl-job jobhandler[" + name + "] naming conflicts.");
}
// execute method
if (!(method.getParameterTypes().length == 1 && method.getParameterTypes()[0].isAssignableFrom(String.class))) {
throw new RuntimeException("xxl-job method-jobhandler param-classtype invalid, for[" + bean.getClass() + "#" + method.getName() + "] , " +
"The correct method format like \" public ReturnT<String> execute(String param) \" .");
}
if (!method.getReturnType().isAssignableFrom(ReturnT.class)) {
throw new RuntimeException("xxl-job method-jobhandler return-classtype invalid, for[" + bean.getClass() + "#" + method.getName() + "] , " +
"The correct method format like \" public ReturnT<String> execute(String param) \" .");
}
method.setAccessible(true);
// 获取需要初始化方法init 和销毁方法destory()
Method initMethod = null;
Method destroyMethod = null;
if (xxlJob.init().trim().length() > 0) {
try {
initMethod = bean.getClass().getDeclaredMethod(xxlJob.init());
initMethod.setAccessible(true);
} catch (NoSuchMethodException e) {
throw new RuntimeException("xxl-job method-jobhandler initMethod invalid, for[" + bean.getClass() + "#" + method.getName() + "] .");
}
}
if (xxlJob.destroy().trim().length() > 0) {
try {
destroyMethod = bean.getClass().getDeclaredMethod(xxlJob.destroy());
destroyMethod.setAccessible(true);
} catch (NoSuchMethodException e) {
throw new RuntimeException("xxl-job method-jobhandler destroyMethod invalid, for[" + bean.getClass() + "#" + method.getName() + "] .");
}
}
// registry jobhandler
// 放入jobHandlerRepository Map
registJobHandler(name, new MethodJobHandler(bean, method, initMethod, destroyMethod));
}
}
}
2.3 XxlJobSpringExecutor#start
start方法 ，有点像Spring 里面的refresh() 方法，将几个主要的大步骤都列出来了，主要做了以下几件事：
- 初始化log路径，创建对应的log路径文件夹
- 初始化 admin-client， 封装成一个或者多个AdminBizClient 对象，放在adminBizList list里面,以便后面的回调，注册，移除等操作
- 启动日志线程，定时清理日志
public void start() throws Exception {
// 初始化log路径，没有给定，默认是 /data/applogs/xxl-job/jobhandler
XxlJobFileAppender.initLogPath(logPath);
/** 初始化invoker, admin-client
这里就是给定的 admin server 端地址，一个或者多个，封装成AdminBizClient 对象
放在adminBizList list里面,以便后面的回调，注册，移除等操作
**/
initAdminBizList(adminAddresses, accessToken);
// 初始化 日志 清理线程，定时清理
JobLogFileCleanThread.getInstance().start(logRetentionDays);
// 初始化回调线程
TriggerCallbackThread.getInstance().start();
// 初始化 executor-server
initEmbedServer(address, ip, port, appname, accessToken);
}
2.3.1 日志路径
log日志默认是在 /data/applogs/xxl-job/jobhandler 目录下面，目录结构如下
2.3.2 日志文件清理线程
日志线程主要作用就是清除超过指定logRetentionDays 天数的日志，此线程主要做了如下事情：
- 判断用户定义的时间， 最小3天 ，小于3天时，没有定时清理日志这个功能, 这个不合理啊， 应该 小于3天给个默认值 3天，不应该不生效啊
- 启动一个守护进程，每隔1天进行一次处理，主要逻辑如下
- 获取日志目录下面的所有的文件和子目录
- 对所有文件和目录进行遍历，去掉文件，和文件夹名字中不带有‘-’ （这个是由于 文件名是用 日期 yyyy-MM-dd 定义的）
- 将文件夹名字上的日期转为 时间，和当前进行对比 ，超过了 最大保留时间，就删除
public void start(final long logRetentionDays){
// limit min value
// 这里的日志最少3天，小于3天还不生效
if (logRetentionDays < 3 ) {
return;
}
localThread = new Thread(new Runnable() {
@Override
public void run() {
while (!toStop) {
try {
//获取日志路径下面的所有文件
File[] childDirs = new File(XxlJobFileAppender.getLogPath()).listFiles();
if (childDirs!=null && childDirs.length>0) {
//获取到当前日期
Calendar todayCal = Calendar.getInstance();
todayCal.set(Calendar.HOUR_OF_DAY,0);
todayCal.set(Calendar.MINUTE,0);
todayCal.set(Calendar.SECOND,0);
todayCal.set(Calendar.MILLISECOND,0);
Date todayDate = todayCal.getTime();
// 对子文件夹进行遍历
for (File childFile: childDirs) {
/**
如果不是 文件夹， 文件夹名字中没有'-' 都跳过
这里都是用日期作为文件夹
**/
if (!childFile.isDirectory()) {
continue;
}
if (childFile.getName().indexOf("-") == -1) {
continue;
}
// file create date
Date logFileCreateDate = null;
try {
SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd");
logFileCreateDate = simpleDateFormat.parse(childFile.getName());
} catch (ParseException e) {
logger.error(e.getMessage(), e);
}
if (logFileCreateDate == null) {
continue;
}
// 这里进行比较， 如果超过了最大的保留时间，就删除
if ((todayDate.getTime()-logFileCreateDate.getTime()) >= logRetentionDays * (24 * 60 * 60 * 1000) ) {
FileUtil.deleteRecursively(childFile);
}
}
}
} catch (Exception e) {
if (!toStop) {
logger.error(e.getMessage(), e);
}
}
try {
// sleep 1天
TimeUnit.DAYS.sleep(1);
} catch (InterruptedException e) {
if (!toStop) {
logger.error(e.getMessage(), e);
}
}
}
logger.info(">>>>>>>>>>> xxl-job, executor JobLogFileCleanThread thread destory.");
}
});
localThread.setDaemon(true);
localThread.setName("xxl-job, executor JobLogFileCleanThread");
localThread.start();
}
2.3.3 callback 线程
TriggerCallbackThread 线程主要逻辑也比较清晰，主要流程如下：
- 启动了一个 触发回调的线程， 通过 take()+ drainTo() 方法 避免高CPU ，每次把需要回到的方法进行回调
- 如果出现停止的情况之后，最后一把把剩余的推送过去
- 这里会再起一个 重试机制的线程，用于处理在回调过程中失败的数据.
public void start() {
// 如果没有配置 admin server 的地址，那就跳过
if (XxlJobExecutor.getAdminBizList() == null) {
logger.warn(">>>>>>>>>>> xxl-job, executor callback config fail, adminAddresses is null.");
return;
}
// callback
triggerCallbackThread = new Thread(new Runnable() {
@Override
public void run() {
// 正常回调
while(!toStop){
try {
/**移除并返回队列的头部， take() 方法是会阻塞的
这里采用了 take()+ drainTo() 方法，既保留了drainTo批量处理数据的高效, 又让其拥有了阻塞效果, 没有数据 时方法不会空循环. 避免CPU占用比较高
**/
HandleCallbackParam callback = getInstance().callBackQueue.take();
if (callback != null) {
// callback list param
List<HandleCallbackParam> callbackParamList = new ArrayList<HandleCallbackParam>();
//调用队列的drainTo 方法，将一次获取所有的数据，drainTo 是不会阻塞的
int drainToNum = getInstance().callBackQueue.drainTo(callbackParamList);
// 加上头部
callbackParamList.add(callback);
// 进行回调， 如果失败会进行重试
if (callbackParamList!=null && callbackParamList.size()>0) {
doCallback(callbackParamList);
}
}
} catch (Exception e) {
if (!toStop) {
logger.error(e.getMessage(), e);
}
}
}
/** 如果程序运行到这里，说明跳出了上面的while 循环， 那就是toStop 为true 了
此时队列中可能还有有遗留没有处理，最后处理一次
通过drainTo 一把捞出，进行处理
**/
try {
List<HandleCallbackParam> callbackParamList = new ArrayList<HandleCallbackParam>();
int drainToNum = getInstance().callBackQueue.drainTo(callbackParamList);
if (callbackParamList!=null && callbackParamList.size()>0) {
doCallback(callbackParamList);
}
} catch (Exception e) {
if (!toStop) {
logger.error(e.getMessage(), e);
}
}
logger.info(">>>>>>>>>>> xxl-job, executor callback thread destory.");
}
});
triggerCallbackThread.setDaemon(true);
triggerCallbackThread.setName("xxl-job, executor TriggerCallbackThread");
triggerCallbackThread.start();
// 这里是 重试机制线程
triggerRetryCallbackThread = new Thread(new Runnable() {
@Override
public void run() {
while(!toStop){
try {
retryFailCallbackFile();
} catch (Exception e) {
if (!toStop) {
logger.error(e.getMessage(), e);
}
}
try {
// 每隔30秒一次
TimeUnit.SECONDS.sleep(RegistryConfig.BEAT_TIMEOUT);
} catch (InterruptedException e) {
if (!toStop) {
logger.error(e.getMessage(), e);
}
}
}
logger.info(">>>>>>>>>>> xxl-job, executor retry callback thread destory.");
}
});
triggerRetryCallbackThread.setDaemon(true);
triggerRetryCallbackThread.start();
}
2.3.3.1 doCallback
doCallback 方法主要是以下逻辑：
1.从 配置的 admin server 地址列表中依次获取，调用对应的callback 方法进行 回调.
2. 如果成功，则记录log 并跳槽， 如果失败，那就 用下一个 admin server 地址进行尝试
3. 如果所有的 admin server 都失败，那就 追加到文件，等重试线程来调用
遍历admin server,这里进行回调，但是如果服务端机器比较多，这里所有的负载都会压在第一台机器上面,
个人感觉可以打乱list,如 Collections.shuffle(list); 这样使负载更加均衡, 同时这里 如果 admin server 动态添加机器（没有使用域名），也是无感知的.
private void doCallback(List<HandleCallbackParam> callbackParamList){
boolean callbackRet = false;
// callback, will retry if error
/** 遍历admin server,这里进行回调，但是如果服务端机器比较多，这里所有的负载都会压在第一台机器上面
如果失败，会用第二台进行测试，以此类推
**/
for (AdminBiz adminBiz: XxlJobExecutor.getAdminBizList()) {
try {
ReturnT<String> callbackResult = adminBiz.callback(callbackParamList);
if (callbackResult!=null && ReturnT.SUCCESS_CODE == callbackResult.getCode()) {
callbackLog(callbackParamList, "<br>----------- xxl-job job callback finish.");
callbackRet = true;
break;
} else {
callbackLog(callbackParamList, "<br>----------- xxl-job job callback fail, callbackResult:" + callbackResult);
}
} catch (Exception e) {
callbackLog(callbackParamList, "<br>----------- xxl-job job callback error, errorMsg:" + e.getMessage());
}
}
// 失败之后重试
if (!callbackRet) {
appendFailCallbackFile(callbackParamList);
}
}
private void callbackLog(List<HandleCallbackParam> callbackParamList, String logContent){
for (HandleCallbackParam callbackParam: callbackParamList) {
// 创建文件夹， 并返回对应的log文件全路径
String logFileName = XxlJobFileAppender.makeLogFileName(new Date(callbackParam.getLogDateTim()), callbackParam.getLogId());
// 存放到threadlocal
XxlJobFileAppender.contextHolder.set(logFileName);
//记录log
XxlJobLogger.log(logContent);
}
}
这里简单介绍一下记录的log 的格式
/// “yyyy-MM-dd HH:mm:ss [ClassName]-[MethodName]-[LineNumber]-[ThreadName] log”;
StackTraceElement[] stackTraceElements = new Throwable().getStackTrace();
StackTraceElement callInfo = stackTraceElements[1];/
这里是使用了new Throwable().getStackTrace() [1] 通过 获取栈信息，把调用的 类-方法-行号-log 打印出来
一般获取到堆栈轨迹的两种方法
Thread.currentThread().getStackTrace()
new Throwable().getStackTrace()
2.3.3.2 appendFailCallbackFile
appendFailCallbackFile 方法就是将失败之后的数据存到文件里面，便于下次重试
private void appendFailCallbackFile(List<HandleCallbackParam> callbackParamList){
// valid
if (callbackParamList==null || callbackParamList.size()==0) {
return;
}
// 这里是将 Object-->byte[]
byte[] callbackParamList_bytes = JdkSerializeTool.serialize(callbackParamList);
// 这里是 存放在 日志路径下面的 /callbacklog/xxl-job-callback-{x}.log
File callbackLogFile = new File(failCallbackFileName.replace("{x}", String.valueOf(System.currentTimeMillis())));
if (callbackLogFile.exists()) {
for (int i = 0; i < 100; i++) {
callbackLogFile = new File(failCallbackFileName.replace("{x}", String.valueOf(System.currentTimeMillis()).concat("-").concat(String.valueOf(i)) ));
if (!callbackLogFile.exists()) {
break;
}
}
}
FileUtil.writeFileContent(callbackLogFile, callbackParamList_bytes);
}
2.3.3.3 重试triggerRetryCallbackThread 线程
重试机制也比较简单，就是 读取 对应目录下面的数据，转换为对应的类型，先将原先文件删除，
然后进行再次调用doCallback 方法
private void retryFailCallbackFile(){
// valid
File callbackLogPath = new File(failCallbackFilePath);
if (!callbackLogPath.exists()) {
return;
}
if (callbackLogPath.isFile()) {
callbackLogPath.delete();
}
if (!(callbackLogPath.isDirectory() && callbackLogPath.list()!=null && callbackLogPath.list().length>0)) {
return;
}
// load and clear file, retry
for (File callbaclLogFile: callbackLogPath.listFiles()) {
byte[] callbackParamList_bytes = FileUtil.readFileContent(callbaclLogFile);
List<HandleCallbackParam> callbackParamList = (List<HandleCallbackParam>) JdkSerializeTool.deserialize(callbackParamList_bytes, List.class);
callbaclLogFile.delete();
doCallback(callbackParamList);
}
}
2.3.3.4 AdminBizClient#callback
在上面doCallback 方法里面 ，通过调用 adminBiz.callback(callbackParamList);进行回调，这里就是通过HttpURLConnection 进行调用,调用 admin server 的API 接口.
@Override
public ReturnT<String> callback(List<HandleCallbackParam> callbackParamList) {
return XxlJobRemotingUtil.postBody(addressUrl+"api/callback", accessToken, timeout, callbackParamList, String.class);
}
@Override
public ReturnT<String> registry(RegistryParam registryParam) {
return XxlJobRemotingUtil.postBody(addressUrl + "api/registry", accessToken, timeout, registryParam, String.class);
}
@Override
public ReturnT<String> registryRemove(RegistryParam registryParam) {
return XxlJobRemotingUtil.postBody(addressUrl + "api/registryRemove", accessToken, timeout, registryParam, String.class);
}
2.3.4 initEmbedServer
initEmbedServer 方法主要逻辑是 ，获取IP 和port ， 调用EmbedServer.start
private void initEmbedServer(String address, String ip, int port, String appname, String accessToken) throws Exception {
/**填充IP和port
**/
port = port>0?port: NetUtil.findAvailablePort(9999);
ip = (ip!=null&&ip.trim().length()>0)?ip: IpUtil.getIp();
// generate address
if (address==null || address.trim().length()==0) {
String ip_port_address = IpUtil.getIpPort(ip, port); // registry-address：default use address to registry , otherwise use ip:port if address is null
address = "http://{ip_port}/".replace("{ip_port}", ip_port_address);
}
// start
embedServer = new EmbedServer();
embedServer.start(address, port, appname, accessToken);
}
2.3.4.1 NetUtil
findAvailablePort 方法主要逻辑就是 获取一个可以使用的端口.
先对Port 进行递增校验， 再递减校验，直到获取可用的端口
public static int findAvailablePort(int defaultPort) {
int portTmp = defaultPort;
while (portTmp < 65535) {
if (!isPortUsed(portTmp)) {
return portTmp;
} else {
portTmp++;
}
}
portTmp = defaultPort--;
while (portTmp > 0) {
if (!isPortUsed(portTmp)) {
return portTmp;
} else {
portTmp--;
}
}
throw new RuntimeException("no available port.");
}
2.3.4.2 EmbedServer
public void start(final String address, final int port, final String appname, final String accessToken) {
executorBiz = new ExecutorBizImpl();
thread = new Thread(new Runnable() {
@Override
public void run() {
// 创建两个线程池组，分别处理客户端的连接和 客户端的读写
EventLoopGroup bossGroup = new NioEventLoopGroup();
EventLoopGroup workerGroup = new NioEventLoopGroup();
// 创建线程池
ThreadPoolExecutor bizThreadPool = new ThreadPoolExecutor(
0,
200,
60L,
TimeUnit.SECONDS,
new LinkedBlockingQueue<Runnable>(2000),
new ThreadFactory() {
@Override
public Thread newThread(Runnable r) {
return new Thread(r, "xxl-rpc, EmbedServer bizThreadPool-" + r.hashCode());
}
},
new RejectedExecutionHandler() {
@Override
public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
throw new RuntimeException("xxl-job, EmbedServer bizThreadPool is EXHAUSTED!");
}
});
try {
// start server
// 创建netty引导类，配置和串联系列组件（设置线程模型，设置通道类型，设置客户端处理器handler，设置绑定端口号）
ServerBootstrap bootstrap = new ServerBootstrap();
bootstrap.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
public void initChannel(SocketChannel channel) throws Exception {
// 配置链式解码器
channel.pipeline()
.addLast(new IdleStateHandler(0, 0, 30 * 3, TimeUnit.SECONDS)) // beat 3N, close if idle
// 解码成HttpRequest
.addLast(new HttpServerCodec())
// 解码成FullHttpRequest
.addLast(new HttpObjectAggregator(5 * 1024 * 1024)) // merge request & reponse to FULL
// 添加处自定义的处理器,这里还是http 协议
.addLast(new EmbedHttpServerHandler(executorBiz, accessToken, bizThreadPool));
}
})
.childOption(ChannelOption.SO_KEEPALIVE, true);
//bind 绑定端口号
ChannelFuture future = bootstrap.bind(port).sync();
logger.info(">>>>>>>>>>> xxl-job remoting server start success, nettype = {}, port = {}", EmbedServer.class, port);
// 进行注册
startRegistry(appname, address);
// wait util stop
future.channel().closeFuture().sync();
} catch (InterruptedException e) {
if (e instanceof InterruptedException) {
logger.info(">>>>>>>>>>> xxl-job remoting server stop.");
} else {
logger.error(">>>>>>>>>>> xxl-job remoting server error.", e);
}
} finally {
// stop
try {
workerGroup.shutdownGracefully();
bossGroup.shutdownGracefully();
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
}
}
});
thread.setDaemon(true); // daemon, service jvm, user thread leave >>> daemon leave >>> jvm leave
thread.start();
}
2.3.4.3 ExecutorRegistryThread
public void start(final String appname, final String address){
// valid
if (appname==null || appname.trim().length()==0) {
logger.warn(">>>>>>>>>>> xxl-job, executor registry config fail, appname is null.");
return;
}
if (XxlJobExecutor.getAdminBizList() == null) {
logger.warn(">>>>>>>>>>> xxl-job, executor registry config fail, adminAddresses is null.");
return;
}
registryThread = new Thread(new Runnable() {
@Override
public void run() {
// 进行注册
while (!toStop) {
try {
RegistryParam registryParam = new RegistryParam(RegistryConfig.RegistType.EXECUTOR.name(), appname, address);
for (AdminBiz adminBiz: XxlJobExecutor.getAdminBizList()) {
try {
// 调用AdminBizClient 的registry 进行注册
ReturnT<String> registryResult = adminBiz.registry(registryParam);
if (registryResult!=null && ReturnT.SUCCESS_CODE == registryResult.getCode()) {
registryResult = ReturnT.SUCCESS;
logger.debug(">>>>>>>>>>> xxl-job registry success, registryParam:{}, registryResult:{}", new Object[]{registryParam, registryResult});
break;
} else {
logger.info(">>>>>>>>>>> xxl-job registry fail, registryParam:{}, registryResult:{}", new Object[]{registryParam, registryResult});
}
} catch (Exception e) {
logger.info(">>>>>>>>>>> xxl-job registry error, registryParam:{}", registryParam, e);
}
}
} catch (Exception e) {
if (!toStop) {
logger.error(e.getMessage(), e);
}
}
// 每隔30s 进行注册一次
try {
if (!toStop) {
TimeUnit.SECONDS.sleep(RegistryConfig.BEAT_TIMEOUT);
}
} catch (InterruptedException e) {
if (!toStop) {
logger.warn(">>>>>>>>>>> xxl-job, executor registry thread interrupted, error msg:{}", e.getMessage());
}
}
}
// 代码到这里时，说明 toStop =true, 进行移除注册
try {
RegistryParam registryParam = new RegistryParam(RegistryConfig.RegistType.EXECUTOR.name(), appname, address);
for (AdminBiz adminBiz: XxlJobExecutor.getAdminBizList()) {
try {
ReturnT<String> registryResult = adminBiz.registryRemove(registryParam);
if (registryResult!=null && ReturnT.SUCCESS_CODE == registryResult.getCode()) {
registryResult = ReturnT.SUCCESS;
logger.info(">>>>>>>>>>> xxl-job registry-remove success, registryParam:{}, registryResult:{}", new Object[]{registryParam, registryResult});
break;
} else {
logger.info(">>>>>>>>>>> xxl-job registry-remove fail, registryParam:{}, registryResult:{}", new Object[]{registryParam, registryResult});
}
} catch (Exception e) {
if (!toStop) {
logger.info(">>>>>>>>>>> xxl-job registry-remove error, registryParam:{}", registryParam, e);
}
}
}
} catch (Exception e) {
if (!toStop) {
logger.error(e.getMessage(), e);
}
}
logger.info(">>>>>>>>>>> xxl-job, executor registry thread destory.");
}
});
registryThread.setDaemon(true);
registryThread.setName("xxl-job, executor ExecutorRegistryThread");
registryThread.start();
}
2.4 EmbedHttpServerHandler
EmbedHttpServerHandler 是继承了SimpleChannelInboundHandler 的自定义处理器，我们看一下channelRead0 里面的逻辑
2.4.1 channelRead0
这里主要如下逻辑，
- 获取 请求的内容，方法类型(post/get)，url 等信息，
- 异步线程调用触发
- 转json
- 返回response.
protected void channelRead0(final ChannelHandlerContext ctx, FullHttpRequest msg) throws Exception {
//final byte[] requestBytes = ByteBufUtil.getBytes(msg.content());
// byteBuf.toString(io.netty.util.CharsetUtil.UTF_8);
// 解析请求，获取 内容、url、获取请求方法类型(post/get)
String requestData = msg.content().toString(CharsetUtil.UTF_8);
String uri = msg.uri();
HttpMethod httpMethod = msg.method();
boolean keepAlive = HttpUtil.isKeepAlive(msg);
String accessTokenReq = msg.headers().get(XxlJobRemotingUtil.XXL_JOB_ACCESS_TOKEN);
// invoke
bizThreadPool.execute(new Runnable() {
@Override
public void run() {
// do invoke
Object responseObj = process(httpMethod, uri, requestData, accessTokenReq);
// to json
String responseJson = GsonTool.toJson(responseObj);
// write response
writeResponse(ctx, keepAlive, responseJson);
}
});
}
2.4.2 EmbedHttpServerHandler#process
process 这里主要做了一些校验
private Object process(HttpMethod httpMethod, String uri, String requestData, String accessTokenReq) {
/**
Check,以下类型直接返回错误信息
1. 如果method 类型不是POST
2. 如果请求方法为null
3. accessToken 和给定的不一样(如果存在)
**/
if (HttpMethod.POST != httpMethod) {
return new ReturnT<String>(ReturnT.FAIL_CODE, "invalid request, HttpMethod not support.");
}
if (uri==null || uri.trim().length()==0) {
return new ReturnT<String>(ReturnT.FAIL_CODE, "invalid request, uri-mapping empty.");
}
if (accessToken!=null
&& accessToken.trim().length()>0
&& !accessToken.equals(accessTokenReq)) {
return new ReturnT<String>(ReturnT.FAIL_CODE, "The access token is wrong.");
}
// 具体的方法，调用对应的services
try {
if ("/beat".equals(uri)) {
return executorBiz.beat();
} else if ("/idleBeat".equals(uri)) {
IdleBeatParam idleBeatParam = GsonTool.fromJson(requestData, IdleBeatParam.class);
return executorBiz.idleBeat(idleBeatParam);
} else if ("/run".equals(uri)) {
TriggerParam triggerParam = GsonTool.fromJson(requestData, TriggerParam.class);
return executorBiz.run(triggerParam);
} else if ("/kill".equals(uri)) {
KillParam killParam = GsonTool.fromJson(requestData, KillParam.class);
return executorBiz.kill(killParam);
} else if ("/log".equals(uri)) {
LogParam logParam = GsonTool.fromJson(requestData, LogParam.class);
return executorBiz.log(logParam);
} else {
return new ReturnT<String>(ReturnT.FAIL_CODE, "invalid request, uri-mapping("+ uri +") not found.");
}
} catch (Exception e) {
logger.error(e.getMessage(), e);
return new ReturnT<String>(ReturnT.FAIL_CODE, "request error:" + ThrowableUtil.toString(e));
}
}
2.4.3 ExecutorBizImpl
2.4.3.1 ExecutorBizImpl#run
这里主要做了以下几件事：
- 判断是否存在老的jobThread，存在说明这个job 之前已经调用过了，如果存在，获取 jobHandler
- 根据不同的运行模式，走不同的分支，获取最新的jobHandler
- 如果当前job 在运行，根据运行的策略（Cover Early / Discard Later / Serial execution ）， 就行调整
- 如果第一次运行此job ，直接运行，如果不是加入队列
public ReturnT<String> run(TriggerParam triggerParam) {
// load old：jobHandler + jobThread
/**
根据jobId,加载老的jobThread,如果存在，那就获取对应的Handler
**/
JobThread jobThread = XxlJobExecutor.loadJobThread(triggerParam.getJobId());
IJobHandler jobHandler = jobThread!=null?jobThread.getHandler():null;
String removeOldReason = null;
/** 根据不同的运行模式执行对应的方法
BEAN, GLUE(Java),GLUE(Shell),GLUE(Python),GLUE(PHP),GLUE(Nodejs),GLUE(PowerShell)
**/
GlueTypeEnum glueTypeEnum = GlueTypeEnum.match(triggerParam.getGlueType());
/**
如果是BEAN 模式，获取对应的jobhandler ，和 jobThread 里面的jobhandler 对比，是否是同一个
**/
if (GlueTypeEnum.BEAN == glueTypeEnum) {
// new jobhandler
IJobHandler newJobHandler = XxlJobExecutor.loadJobHandler(triggerParam.getExecutorHandler());
/** 校验老的jobThread ，这里如果jobThread存在，并且老的jobHandler 和新的不一样，说明发生了变化
将老的jobThread 和 jobHandler 置为空
**/
if (jobThread!=null && jobHandler != newJobHandler) {
// change handler, need kill old thread
removeOldReason = "change jobhandler or glue type, and terminate the old job thread.";
jobThread = null;
jobHandler = null;
}
// 如果老的jobHandler ，赋值为新的，如果都没有找到，则提示错误信息，FAIL_CODE设置500
if (jobHandler == null) {
jobHandler = newJobHandler;
if (jobHandler == null) {
return new ReturnT<String>(ReturnT.FAIL_CODE, "job handler [" + triggerParam.getExecutorHandler() + "] not found.");
}
}
} else if (GlueTypeEnum.GLUE_GROOVY == glueTypeEnum) {
// 校验老的jobThread ，需要JobHandler 为GlueJobHandler类型，并且glueUpdatetime 和传入的要一致
if (jobThread != null &&
!(jobThread.getHandler() instanceof GlueJobHandler
&& ((GlueJobHandler) jobThread.getHandler()).getGlueUpdatetime()==triggerParam.getGlueUpdatetime() )) {
// change handler or gluesource updated, need kill old thread
removeOldReason = "change job source or glue type, and terminate the old job thread.";
// 置空
jobThread = null;
jobHandler = null;
}
// valid handler
if (jobHandler == null) {
try {
IJobHandler originJobHandler = GlueFactory.getInstance().loadNewInstance(triggerParam.getGlueSource());
jobHandler = new GlueJobHandler(originJobHandler, triggerParam.getGlueUpdatetime());
} catch (Exception e) {
logger.error(e.getMessage(), e);
return new ReturnT<String>(ReturnT.FAIL_CODE, e.getMessage());
}
}
} else if (glueTypeEnum!=null && glueTypeEnum.isScript()) {
// valid old jobThread
if (jobThread != null &&
!(jobThread.getHandler() instanceof ScriptJobHandler
&& ((ScriptJobHandler) jobThread.getHandler()).getGlueUpdatetime()==triggerParam.getGlueUpdatetime() )) {
// change script or gluesource updated, need kill old thread
removeOldReason = "change job source or glue type, and terminate the old job thread.";
jobThread = null;
jobHandler = null;
}
// valid handler
if (jobHandler == null) {
jobHandler = new ScriptJobHandler(triggerParam.getJobId(), triggerParam.getGlueUpdatetime(), triggerParam.getGlueSource(), GlueTypeEnum.match(triggerParam.getGlueType()));
}
} else {
return new ReturnT<String>(ReturnT.FAIL_CODE, "glueType[" + triggerParam.getGlueType() + "] is not valid.");
}
// executor block strategy
if (jobThread != null) {
ExecutorBlockStrategyEnum blockStrategy = ExecutorBlockStrategyEnum.match(triggerParam.getExecutorBlockStrategy(), null);
if (ExecutorBlockStrategyEnum.DISCARD_LATER == blockStrategy) {
// 如果是运行时丢弃，直接返回
if (jobThread.isRunningOrHasQueue()) {
return new ReturnT<String>(ReturnT.FAIL_CODE, "block strategy effect："+ExecutorBlockStrategyEnum.DISCARD_LATER.getTitle());
}
} else if (ExecutorBlockStrategyEnum.COVER_EARLY == blockStrategy) {
// 如果是覆盖之前的类型，将之前的置空
if (jobThread.isRunningOrHasQueue()) {
removeOldReason = "block strategy effect：" + ExecutorBlockStrategyEnum.COVER_EARLY.getTitle();
jobThread = null;
}
} else {
// just queue trigger
}
}
//这里是替换旧的线程，或者新建一个新的
if (jobThread == null) {
jobThread = XxlJobExecutor.registJobThread(triggerParam.getJobId(), jobHandler, removeOldReason);
}
// 将数据放入队列
ReturnT<String> pushResult = jobThread.pushTriggerQueue(triggerParam);
return pushResult;
}
public static JobThread registJobThread(int jobId, IJobHandler handler, String removeOldReason){
// 新起一个jobThread,并启动
JobThread newJobThread = new JobThread(jobId, handler);
newJobThread.start();
logger.info(">>>>>>>>>>> xxl-job regist JobThread success, jobId:{}, handler:{}", new Object[]{jobId, handler});
// 返回老的jobThread ，中断
JobThread oldJobThread = jobThreadRepository.put(jobId, newJobThread); // putIfAbsent | oh my god, map's put method return the old value!!!
if (oldJobThread != null) {
oldJobThread.toStop(removeOldReason);
oldJobThread.interrupt();
}
return newJobThread;
}
2.5 JobThread
这里就是具体执行的逻辑，主要看一下 run() 方法里面的东西，主要逻辑也比较简单，流程如下：
- 先运行 init 方法
- 运行具体的job 方法
- 运行 destory() 方法，如果存在
public void run() {
// 如果存在init 方法，调用 init()
try {
handler.init();
} catch (Throwable e) {
logger.error(e.getMessage(), e);
}
//执行具体的逻辑
while(!toStop){
running = false;
idleTimes++;
TriggerParam triggerParam = null;
ReturnT<String> executeResult = null;
try {
/**
这里使用的是 poll(timeout)，主要是为了考虑检查toStop 标志是否发生变成，所以需要循环，所以这里不能使用queue.take()
**/
triggerParam = triggerQueue.poll(3L, TimeUnit.SECONDS);
if (triggerParam!=null) {
running = true;
idleTimes = 0;
triggerLogIdSet.remove(triggerParam.getLogId());
// 记录文件名称，like "logPath/yyyy-MM-dd/9999.log"
String logFileName = XxlJobFileAppender.makeLogFileName(new Date(triggerParam.getLogDateTime()), triggerParam.getLogId());
XxlJobFileAppender.contextHolder.set(logFileName);
ShardingUtil.setShardingVo(new ShardingUtil.ShardingVO(triggerParam.getBroadcastIndex(), triggerParam.getBroadcastTotal()));
// execute
XxlJobLogger.log("<br>----------- xxl-job job execute start -----------<br>----------- Param:" + triggerParam.getExecutorParams());
if (triggerParam.getExecutorTimeout() > 0) {
// limit timeout
Thread futureThread = null;
try {
final TriggerParam triggerParamTmp = triggerParam;
FutureTask<ReturnT<String>> futureTask = new FutureTask<ReturnT<String>>(new Callable<ReturnT<String>>() {
@Override
public ReturnT<String> call() throws Exception {
return handler.execute(triggerParamTmp.getExecutorParams());
}
});
futureThread = new Thread(futureTask);
futureThread.start();
executeResult = futureTask.get(triggerParam.getExecutorTimeout(), TimeUnit.SECONDS);
} catch (TimeoutException e) {
XxlJobLogger.log("<br>----------- xxl-job job execute timeout");
XxlJobLogger.log(e);
executeResult = new ReturnT<String>(IJobHandler.FAIL_TIMEOUT.getCode(), "job execute timeout ");
} finally {
futureThread.interrupt();
}
} else {
// just execute
executeResult = handler.execute(triggerParam.getExecutorParams());
}
if (executeResult == null) {
executeResult = IJobHandler.FAIL;
} else {
executeResult.setMsg(
(executeResult!=null&&executeResult.getMsg()!=null&&executeResult.getMsg().length()>50000)
?executeResult.getMsg().substring(0, 50000).concat("...")
:executeResult.getMsg());
executeResult.setContent(null); // limit obj size
}
XxlJobLogger.log("<br>----------- xxl-job job execute end(finish) -----------<br>----------- ReturnT:" + executeResult);
} else {
if (idleTimes > 30) {
if(triggerQueue.size() == 0) { // avoid concurrent trigger causes jobId-lost
XxlJobExecutor.removeJobThread(jobId, "excutor idel times over limit.");
}
}
}
} catch (Throwable e) {
if (toStop) {
XxlJobLogger.log("<br>----------- JobThread toStop, stopReason:" + stopReason);
}
StringWriter stringWriter = new StringWriter();
e.printStackTrace(new PrintWriter(stringWriter));
String errorMsg = stringWriter.toString();
executeResult = new ReturnT<String>(ReturnT.FAIL_CODE, errorMsg);
XxlJobLogger.log("<br>----------- JobThread Exception:" + errorMsg + "<br>----------- xxl-job job execute end(error) -----------");
} finally {
if(triggerParam != null) {
// callback handler info
if (!toStop) {
// commonm
TriggerCallbackThread.pushCallBack(new HandleCallbackParam(triggerParam.getLogId(), triggerParam.getLogDateTime(), executeResult));
} else {
// is killed
ReturnT<String> stopResult = new ReturnT<String>(ReturnT.FAIL_CODE, stopReason + " [job running, killed]");
TriggerCallbackThread.pushCallBack(new HandleCallbackParam(triggerParam.getLogId(), triggerParam.getLogDateTime(), stopResult));
}
}
}
}
// callback trigger request in queue
while(triggerQueue !=null && triggerQueue.size()>0){
TriggerParam triggerParam = triggerQueue.poll();
if (triggerParam!=null) {
// is killed
ReturnT<String> stopResult = new ReturnT<String>(ReturnT.FAIL_CODE, stopReason + " [job not executed, in the job queue, killed.]");
TriggerCallbackThread.pushCallBack(new HandleCallbackParam(triggerParam.getLogId(), triggerParam.getLogDateTime(), stopResult));
}
}
// destroy
try {
handler.destroy();
} catch (Throwable e) {
logger.error(e.getMessage(), e);
}
logger.info(">>>>>>>>>>> xxl-job JobThread stoped, hashCode:{}", Thread.currentThread());
}
三、小结
本章主要分析了 xxl-job-core 里面的核心代码，整个流程还是相对比较清晰的.
| 支付宝 | 微信 |
|---|---|
 |  |
| 如果有帮助记得打赏哦 | 特别需要您的打赏哦 |
1542
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
