XXL-JOB 微服务改造
调度中心改造
主要改造为适配调度中心添加执行器的手动注册模式，输入对应的微服务名称能正常执行定时任务。
设计
通过 Ribbon + RestTemplate 实现负载均衡发送执行请求。在xxl-job-admin调度中心的启动类中
@Bean
@LoadBalanced
public RestTemplate restTemplate() {
return new RestTemplate();
}
@Bean
public RandomRule randomRule(){
return new RandomRule();
}
调度中心执行定时任务的流程
- 执行请求打到
JobInfoController。
@RequestMapping("/trigger")
@ResponseBody
public ReturnT<String> triggerJob(HttpServletRequest request, int id, String executorParam, String addressList) {
// login user
XxlJobUser loginUser = (XxlJobUser) request.getAttribute(LoginService.LOGIN_IDENTITY_KEY);
// trigger
return xxlJobService.trigger(loginUser, id, executorParam, addressList);
}
- 进行相关权限判断后调用
JobTriggerPoolHelper.trigger()方法。
@Override
public ReturnT<String> trigger(XxlJobUser loginUser, int jobId, String executorParam, String addressList) {
// permission
if (loginUser == null) {
return new ReturnT<String>(ReturnT.FAIL.getCode(), I18nUtil.getString("system_permission_limit"));
}
XxlJobInfo xxlJobInfo = xxlJobInfoDao.loadById(jobId);
if (xxlJobInfo == null) {
return new ReturnT<String>(ReturnT.FAIL.getCode(), I18nUtil.getString("jobinfo_glue_jobid_unvalid"));
}
if (!hasPermission(loginUser, xxlJobInfo.getJobGroup())) {
return new ReturnT<String>(ReturnT.FAIL.getCode(), I18nUtil.getString("system_permission_limit"));
}
// force cover job param
if (executorParam == null) {
executorParam = "";
}
JobTriggerPoolHelper.trigger(jobId, TriggerTypeEnum.MANUAL, -1, null, executorParam, addressList);
return ReturnT.SUCCESS;
}
trigger方法中会调用自身的addTrigger方法，此方法选择在项目启动时初始化好的ThreadPoolExecutor线程池，将接收到的任务交给线程池执行。
public static void trigger(int jobId, TriggerTypeEnum triggerType, int failRetryCount, String executorShardingParam, String executorParam, String addressList) {
helper.addTrigger(jobId, triggerType, failRetryCount, executorShardingParam, executorParam, addressList);
}
- 调用
XxlJobTrigger中的trigger()进行相关参数、策略的判断。
public void addTrigger(final int jobId,
final TriggerTypeEnum triggerType,
final int failRetryCount,
final String executorShardingParam,
final String executorParam,
final String addressList) {
// choose thread pool
ThreadPoolExecutor triggerPool_ = fastTriggerPool;
AtomicInteger jobTimeoutCount = jobTimeoutCountMap.get(jobId);
if (jobTimeoutCount != null && jobTimeoutCount.get() > 10) { // job-timeout 10 times in 1 min
triggerPool_ = slowTriggerPool;
}
// trigger
triggerPool_.execute(new Runnable() {
@Override
public void run() {
long start = System.currentTimeMillis();
try {
// do trigger
XxlJobTrigger.trigger(jobId, triggerType, failRetryCount, executorShardingParam, executorParam, addressList);
} catch (Exception e) {
logger.error(e.getMessage(), e);
} finally {
// check timeout-count-map
long minTim_now = System.currentTimeMillis() / 60000;
if (minTim != minTim_now) {
minTim = minTim_now;
jobTimeoutCountMap.clear();
}
// incr timeout-count-map
long cost = System.currentTimeMillis() - start;
if (cost > 500) { // ob-timeout threshold 500ms
AtomicInteger timeoutCount = jobTimeoutCountMap.putIfAbsent(jobId, new AtomicInteger(1));
if (timeoutCount != null) {
timeoutCount.incrementAndGet();
}
}
}
}
});
}
- 调用
processTrigger()方法，在其中会调用runExecutor()方法。
public static void trigger(int jobId,
TriggerTypeEnum triggerType,
int failRetryCount,
String executorShardingParam,
String executorParam,
String addressList) {
// load data
//根据job id 从 xxl_job_info 获取job ,jobId就是调度中心创建的调度任务
XxlJobInfo jobInfo = XxlJobAdminConfig.getAdminConfig().getXxlJobInfoDao().loadById(jobId);
if (jobInfo == null) {
logger.warn(">>>>>>>>>>>> trigger fail, jobId invalid，jobId={}", jobId);
return;
}
// 执行器，任务参数
if (executorParam != null) {
jobInfo.setExecutorParam(executorParam);
}
// failRetryCount：重试次数
int finalFailRetryCount = failRetryCount >= 0 ? failRetryCount : jobInfo.getExecutorFailRetryCount();
// 根据job id 从 xxl_job_group 获取该任务对应的执行器信息
XxlJobGroup group = XxlJobAdminConfig.getAdminConfig().getXxlJobGroupDao().load(jobInfo.getJobGroup());
// cover addressList
// 执行器地址覆盖，如果重新传入了执行器地址 则进行覆盖；否则取 XxlJobGroup 的执行器地址
//
if (addressList != null && addressList.trim().length() > 0) {
// 如果是外部传入的执行器地址，默认就认为是手动录入
// 执行器地址类型：0=自动注册、1=手动录入
group.setAddressType(1);
group.setAddressList(addressList.trim());
}
// sharding param
// TODO：执行器分片策略，没有看懂
int[] shardingParam = null;
if (executorShardingParam != null) {
String[] shardingArr = executorShardingParam.split("/");
if (shardingArr.length == 2 && isNumeric(shardingArr[0]) && isNumeric(shardingArr[1])) {
shardingParam = new int[2];
shardingParam[0] = Integer.valueOf(shardingArr[0]);
shardingParam[1] = Integer.valueOf(shardingArr[1]);
}
}
// 判断路由策略
// 可以打开 http://dev.crungoo.ecos.xxljob.cn:31000/xxl-job-admin/jobinfo 查看路由策略
// 如果是： 分片广播，则需要为注册的每个执行器都 发送任务
if (ExecutorRouteStrategyEnum.SHARDING_BROADCAST == ExecutorRouteStrategyEnum.match(jobInfo.getExecutorRouteStrategy(), null)
&& group.getRegistryList() != null && !group.getRegistryList().isEmpty()
&& shardingParam == null) {
for (int i = 0; i < group.getRegistryList().size(); i++) {
processTrigger(group, jobInfo, finalFailRetryCount, triggerType, i, group.getRegistryList().size());
}
} else {
if (shardingParam == null) {
shardingParam = new int[]{0, 1};
}
processTrigger(group, jobInfo, finalFailRetryCount, triggerType, shardingParam[0], shardingParam[1]);
}
}
runExecutor()方法会通过address获取ExecutorBiz对象。第一次没有则创建一个ExecutorBiz，否则从缓存取。
private static void processTrigger(XxlJobGroup group, XxlJobInfo jobInfo, int finalFailRetryCount, TriggerTypeEnum triggerType, int index, int total) {
// param
// 阻塞处理策略："SERIAL EXECUTION":单机串行; "DISCARD LATER":丢弃后续调度;"COVER EARLY":覆盖之前调度
ExecutorBlockStrategyEnum blockStrategy = ExecutorBlockStrategyEnum.match(jobInfo.getExecutorBlockStrategy(), ExecutorBlockStrategyEnum.SERIAL_EXECUTION); // block strategy
// 路由策略：太多了，不展示了
ExecutorRouteStrategyEnum executorRouteStrategyEnum = ExecutorRouteStrategyEnum.match(jobInfo.getExecutorRouteStrategy(), null); // route strategy
//
String shardingParam = (ExecutorRouteStrategyEnum.SHARDING_BROADCAST == executorRouteStrategyEnum) ? String.valueOf(index).concat("/").concat(String.valueOf(total)) : null;
// 1、save log-id
XxlJobLog jobLog = new XxlJobLog();
jobLog.setJobGroup(jobInfo.getJobGroup());
jobLog.setJobId(jobInfo.getId());
jobLog.setTriggerTime(new Date());
XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().save(jobLog);
logger.debug(">>>>>>>>>>> xxl-job trigger start, jobId:{}", jobLog.getId());
// 2、init trigger-param
TriggerParam triggerParam = new TriggerParam();
triggerParam.setJobId(jobInfo.getId());
triggerParam.setExecutorHandler(jobInfo.getExecutorHandler());
triggerParam.setExecutorParams(jobInfo.getExecutorParam());
triggerParam.setExecutorBlockStrategy(jobInfo.getExecutorBlockStrategy());
triggerParam.setExecutorTimeout(jobInfo.getExecutorTimeout());
triggerParam.setLogId(jobLog.getId());
triggerParam.setLogDateTime(jobLog.getTriggerTime().getTime());
triggerParam.setGlueType(jobInfo.getGlueType());
triggerParam.setGlueSource(jobInfo.getGlueSource());
triggerParam.setGlueUpdatetime(jobInfo.getGlueUpdatetime().getTime());
triggerParam.setBroadcastIndex(index);
triggerParam.setBroadcastTotal(total);
// 3、init address
// 根据任务的路由策略，从执行器地址中，选择出一个执行器地址，构建路由执行结果
String address = null;
ReturnT<String> routeAddressResult = null;
if (group.getRegistryList() != null && !group.getRegistryList().isEmpty()) {
if (ExecutorRouteStrategyEnum.SHARDING_BROADCAST == executorRouteStrategyEnum) {
if (index < group.getRegistryList().size()) {
address = group.getRegistryList().get(index);
} else {
address = group.getRegistryList().get(0);
}
} else {
routeAddressResult = executorRouteStrategyEnum.getRouter().route(triggerParam, group.getRegistryList());
if (routeAddressResult.getCode() == ReturnT.SUCCESS_CODE) {
address = routeAddressResult.getContent();
}
}
} else {
routeAddressResult = new ReturnT<String>(ReturnT.FAIL_CODE, I18nUtil.getString("jobconf_trigger_address_empty"));
}
// 4、trigger remote executor
// 获取到执行器地址，runExecutor 执行任务
ReturnT<String> triggerResult = null;
if (address != null) {
// eg1：到这里为止还是拿到的是微服务名称
// eg2：IP+端口地址
triggerResult = runExecutor(triggerParam, address);
} else {
triggerResult = new ReturnT<String>(ReturnT.FAIL_CODE, null);
}
// 5、collection trigger info
StringBuffer triggerMsgSb = new StringBuffer();
triggerMsgSb.append(I18nUtil.getString("jobconf_trigger_type")).append("：").append(triggerType.getTitle());
triggerMsgSb.append("<br>").append(I18nUtil.getString("jobconf_trigger_admin_adress")).append("：").append(IpUtil.getIp());
triggerMsgSb.append("<br>").append(I18nUtil.getString("jobconf_trigger_exe_regtype")).append("：")
.append((group.getAddressType() == 0) ? I18nUtil.getString("jobgroup_field_addressType_0") : I18nUtil.getString("jobgroup_field_addressType_1"));
triggerMsgSb.append("<br>").append(I18nUtil.getString("jobconf_trigger_exe_regaddress")).append("：").append(group.getRegistryList());
triggerMsgSb.append("<br>").append(I18nUtil.getString("jobinfo_field_executorRouteStrategy")).append("：").append(executorRouteStrategyEnum.getTitle());
if (shardingParam != null) {
triggerMsgSb.append("(" + shardingParam + ")");
}
triggerMsgSb.append("<br>").append(I18nUtil.getString("jobinfo_field_executorBlockStrategy")).append("：").append(blockStrategy.getTitle());
triggerMsgSb.append("<br>").append(I18nUtil.getString("jobinfo_field_timeout")).append("：").append(jobInfo.getExecutorTimeout());
triggerMsgSb.append("<br>").append(I18nUtil.getString("jobinfo_field_executorFailRetryCount")).append("：").append(finalFailRetryCount);
triggerMsgSb.append("<br><br><span style=\"color:#00c0ef;\" > >>>>>>>>>>>" + I18nUtil.getString("jobconf_trigger_run") + "<<<<<<<<<<< </span><br>")
.append((routeAddressResult != null && routeAddressResult.getMsg() != null) ? routeAddressResult.getMsg() + "<br><br>" : "").append(triggerResult.getMsg() != null ? triggerResult.getMsg() : "");
// 6、save log trigger-info
jobLog.setExecutorAddress(address);
jobLog.setExecutorHandler(jobInfo.getExecutorHandler());
jobLog.setExecutorParam(jobInfo.getExecutorParam());
jobLog.setExecutorShardingParam(shardingParam);
jobLog.setExecutorFailRetryCount(finalFailRetryCount);
//jobLog.setTriggerTime();
jobLog.setTriggerCode(triggerResult.getCode());
jobLog.setTriggerMsg(triggerMsgSb.toString());
XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().updateTriggerInfo(jobLog);
logger.debug(">>>>>>>>>>> xxl-job trigger end, jobId:{}", jobLog.getId());
}
- 调用执行器端进行任务的执行
executorBiz.run(triggerParam)。
public static ReturnT<String> runExecutor(TriggerParam triggerParam, String address) {
ReturnT<String> runResult = null;
try {
// 通过address获取ExecutorBiz对象，第一次没有就创建一个ExecutorBiz，有的话就从缓存取
// TODO：如果是注册的是微服务名称，则需要通过注册中心获取地址
// 把微服务名称转成注册地址(ip+port)
//String accessibleAddress = getAccessibleAddress(address); 我在使用ribbon之前的操作，依旧可以使用微服务名称来访问
ExecutorBiz executorBiz = XxlJobScheduler.getExecutorBiz(address);
// 远程rpc 调用 执行器端，进行任务的执行
runResult = executorBiz.run(triggerParam);
} catch (Exception e) {
logger.error(">>>>>>>>>>> xxl-job trigger error, please check if the executor[{}] is running.", address, e);
runResult = new ReturnT<String>(ReturnT.FAIL_CODE, ThrowableUtil.toString(e));
}
StringBuffer runResultSB = new StringBuffer(I18nUtil.getString("jobconf_trigger_run") + "：");
runResultSB.append("<br>address：").append(address);
runResultSB.append("<br>code：").append(runResult.getCode());
runResultSB.append("<br>msg：").append(runResult.getMsg());
runResult.setMsg(runResultSB.toString());
return runResult;
}
ExecutorBizClient发送执行请求改造：
public class ExecutorBizClient implements ExecutorBiz {
private final RestTemplate restTemplate;
public ExecutorBizClient() {
this.restTemplate = SpringUtils.getBean(RestTemplate.class);
}
public ExecutorBizClient(String addressUrl, String accessToken){
this.addressUrl = "http://"+ addressUrl;
this.accessToken = accessToken;
this.restTemplate = SpringUtils.getBean(RestTemplate.class);
}
private String addressUrl ;
private String accessToken;
// 仅展示执行接口改造，其它接口自行改造
@Override
public ReturnT<String> run(TriggerParam triggerParam) {
//return XxlJobRemotingUtil.postBody(addressUrl + "run", accessToken, timeout, triggerParam, String.class);
// 将 TriggerParam 对象和请求头封装到 HttpEntity 中
HttpEntity<TriggerParam> requestEntity = new HttpEntity<>(triggerParam, getHttpHeaders());
ResponseEntity<String> responseEntity = restTemplate.exchange(
addressUrl+"/run",
HttpMethod.POST,
requestEntity,
String.class
);
return new ReturnT<>(responseEntity.getBody());
}
/**
* 获取通用的请求头
* @return
*/
private HttpHeaders getHttpHeaders() {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
headers.set("Connection", "Keep-Alive");
headers.set("Accept-Charset", "application/json;charset=UTF-8");
headers.set("XXL-JOB-ACCESS-TOKEN", accessToken);
return headers;
}
}
ps:
- 获取可用的执行器地址方法
/**
* 获取可用的执行器地址
* @param address 执行器Ip+port地址/ 执行器微服务名称
* @return
*/
private static String getAccessibleAddress(String address){
if(StringUtils.isEmpty(address)){
return address;
}
// http:// --->通过自动注册直接获取的执行器地址直接返回
if(address.contains("http://")){
return address;
}
// eg:executor-server-address --->为执行器部署的微服务名称 替换为对应的IP+PORT
DiscoveryClient discoveryClient = SpringContextUtil.getBean(DiscoveryClient.class);
// 通过微服务名称获取服务实
List<ServiceInstance> instances = discoveryClient.getInstances(address);
if(instances.size() == 0){
return "";
}
// TODO 未考虑执行策略
return instances.get(0).getUri().toString();
}
改造部分
由于需要适配通过微服
务名称访问具体的执行器，改造主要集中在第6步，runExecutor() 方法中获取 ExecutorBiz 对象时传入的 address 已变更为微服务名称。ExecutorBizClient 发送执行请求时通过开启负载均衡的 RestTemplate 根据微服务名称实现动态调用具体执行器。
执行器改造
主要改造为通过调度中心微服务名称实现注册，并移除原内置的 Netty。新建一个 xxl-job-client，将 Netty 中的接口使用 Controller 封装起来。
设计
将通过调度中心 ip + port 的操作放入其注册线程中执行，确保当 xxl-job 调度中心启动失败时，当前服务也能运行，只是不能执行定时任务。
执行器注册流程
- 具体服务引入
XxlJobConfig的配置类，在其被 Spring 管理时会实例化一个XxlJobSpringExecutor的 bean（具体XxlJobSpringExecutor extends XxlJobExecutor implements ApplicationContextAware, SmartInitializingSingleton, DisposableBean）。
@Bean
public XxlJobSpringExecutor xxlJobExecutor() {
logger.info(">>>>>>>>>>> xxl-job config init.");
XxlJobSpringExecutor xxlJobSpringExecutor = new XxlJobSpringExecutor();
xxlJobSpringExecutor.setAdminAddresses(adminAddresses);
xxlJobSpringExecutor.setAppname(appname);
xxlJobSpringExecutor.setPort(port);
xxlJobSpringExecutor.setAccessToken(accessToken);
xxlJobSpringExecutor.setLogPath(logPath);
xxlJobSpringExecutor.setLogRetentionDays(logRetentionDays);
return xxlJobSpringExecutor;
}
XxlJobSpringExecutor会在重写的afterSingletonsInstantiated方法中调用XxlJobExecutor的start()方法。
@Override
public void afterSingletonsInstantiated() {
// init JobHandler Repository
/*initJobHandlerRepository(applicationContext);*/
// init JobHandler Repository (for method)
initJobHandlerMethodRepository(applicationContext);
// refresh GlueFactory
GlueFactory.refreshInstance(1);
// super start
try {
super.start();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
start()方法中初始化日志路径、initAdminBizList初始化调度中心参数对象AdminBiz、初始化JobLogFileCleanThread和TriggerCallbackThread线程、initEmbedServer原创建内置Netty服务（已移除），将微服务真正的注册到调度中心。
public void start() throws Exception {
// init logpath
XxlJobFileAppender.initLogPath(logPath);
// 通过配置的调度中心的微服务名称，拿到调度中心的服务器地址（IP+PORT）
//String s = loadBalancer(adminAddresses);
// init invoker, admin-client 初始化调度中心参数对象
initAdminBizList(adminAddresses, accessToken);
// init JobLogFileCleanThread
JobLogFileCleanThread.getInstance().start(logRetentionDays);
// init TriggerCallbackThread
TriggerCallbackThread.getInstance().start();
// init executor-server，将微服务真正的注册到调度中心
initEmbedServer("", appname);
}
- 修改在
initAdminBizList中传入的AdminBiz对象中存放调度中心ip + port为调度中心微服务名称。
/**
* 初始化调度中心参数对象
* @param adminAddresses 调度中心微服务名称
* @param accessToken token
* @throws Exception
*/
private void initAdminBizList(String adminAddresses, String accessToken) throws Exception {
if (adminAddresses != null && adminAddresses.trim().length() > 0) {
AdminBiz adminBiz = new AdminBizClient(adminAddresses, accessToken);
if (adminBizList == null) {
adminBizList = new ArrayList<AdminBiz>();
}
adminBizList.add(adminBiz);
}
}
- 修改为获取当前微服务的
nocas中随机的ip + port地址，然后调用embedServer.start方法交给新线程去执行startRegistry()方法。
/**
* 已改造为实现将当前微服务注册通过embedServer.start(address, appname)进行注册
*
* @param address 当前微服务地址
* @param appname 执行器名称
* @throws Exception
*/
private void initEmbedServer(String address, String appname) throws Exception {
// fill ip port
port = port > 0 ? port : NetUtil.findAvailablePort(8080);
String ip = IpUtil.getIp();
// generate address
if (address == null || address.trim().length() == 0) {
String ip_port_address = IpUtil.getIpPort(ip, port); // 当前微服务的地址
address = "http://{ip_port}/".replace("{ip_port}", ip_port_address);
}
// start
embedServer = new EmbedServer();
embedServer.start(address, appname);
}
startRegistry()调用ExecutorRegistryThread执行器注册线程调用adminBiz.registry(registryParam)。
/**
* start registry
*
* @param appname 执行器名称
* @param address 当前执行器（微服务）的IP+PORT
*/
public void startRegistry(final String appname, final String address) {
// start registry
ExecutorRegistryThread.getInstance().start(appname, address);
}
AdminBizClient中发送请求到调度中心完成注册。
@Override
public ReturnT<String> registry(RegistryParam registryParam) {
String addressUrl = loadBalancer(serverName);
if (StringUtils.isEmpty(addressUrl)) {
logger.error(">>>>>>>>>>> xxl-job, registry fail,no service instance");
return ReturnT.FAIL;
}
return XxlJobRemotingUtil.postBody(addressUrl + "/api/registry", accessToken, timeout, registryParam, String.class);
}
改造部分
在发送请求前根据调度中心微服务名称从注册中心获取服务实例：
private String loadBalancer(String serverName) { // 调度器微服务名称
// 从注册中心获取服务实例，
LoadBalancerClient client = XxlJobSpringExecutor.getApplicationContext().getBean(LoadBalancerClient.class);
// 从服务集群中选择一个服务实例
ServiceInstance choose = client.choose(serverName);
if (Objects.isNull(choose)) {
return "";
}
return choose.getUri().toString() + "/xxl-job-admin";
}
移除内置netty服务，将调用定时任务的具体接口放进xxl-job-client中，集成到当前微服务去执行；注册时当前微服务注册通过embedServer.start(address, appname)进行注册
扫一扫
2013
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
