elastic-job个人理解

最新推荐文章于 2024-09-27 00:17:03 发布

zhiyegai

最新推荐文章于 2024-09-27 00:17:03 发布

阅读量1.3w

点赞数 2

分类专栏： elastic-job

本文链接：https://blog.csdn.net/zhiyegai/article/details/81511374

版权

elastic-job 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

--采用elastic-job+数据库+zk(可以用其他注册服务替换) 动态处理定时任务，由于elastic-job是通过向zk获取任务信息，所有任务动态处理是通过监听zk和操作zk来保证数据库的任务数据一致，elastic-job有自带的任务监控系统

1.elastic-job提供了三种类型的作业：Simple类型作业（继承的execute方法内做任务处理）、Dataflow类型作业（继承的fetchData做数据获取处理先执行，继承的processData做任务处理后执行，二者执行完表示任务完成）、Script类型作业（通过脚本执行）；采用的是一个任务对于一个类，暂不支持自定义任务方法，如有需要得使用委托处理

2.任务运行日志如果要使用，需要配置监听，创建任务时加入监听，elastic-job有监听类，需要额外处理可以自定义

3.关联zk,任务会注册到zk,分布式任务主要是通过zk来控制任务

4.java定时器任务类同一继承BaseTask类（解决依赖注入service为空问题）

5.依赖包说明

定时器工程需要除了quartz的相关包外，还需要elastic-job-lite-core，elastic-job-lite-spring相关jar包，如果是要操作和修改任务需要额外引入elastic-job-lite-lifecycle.jar包

6.更新，启用，暂停，删除以及查看任务执行历史记录等操作可以在elastic-job-lite-console.war工程里操作（不支持添加任务）

7.多服务部署，elastic-job的服务器纬度是根据ip地址来判断的，一个ip下如果部署两个任务服务，则只有一条纬度记录，但是有两个实例

8.任务分片原则：

当任务的分片总数=1时（必须大于0），如果部定时任务工程部署在多个服务器上，则任务将会以1主N从的方式执行。一旦本次执行任务的服务器崩溃，其他执行任务的服务器将会在下次作业启动时选择一个替补执行。如果开启了失效转移，那么功能效果更好，可以保证在本次作业执行时崩溃，备机之一立即启动替补执行。

当任务分片总数大于1时，如果定时任务工程部署在多个服务器上，则任务根据分片策略去执行（可以通过策略项输入规则，支持三种，也可以通过实现JobShardingStrategy接口自定义分片策略规则），默认是平均分配（并行在各自的服务器上执行任务调度）。

elastic-job只支持任务分片不支持数据分片，数据分片的实现需要开发人员自行根据分片下标和分片总数以及分片规则来处理，即如果任务分片中数据是查询所有的数据，则根据分片下标和分片总数和分片规则来计算各个分片下获取的数据；如：分片总数大1，有多台任务服务器执行同一个任务，分片规则为平均分配，采用分页获取数据，下面是示例，添加任务，定时分片总数为2 ，两个服务器启动该项目，两个服务任务并行执行（基本时间误差可以忽略不计）的结果如下：

@Component

public class MyTestTask extends BaseTask implements SimpleJob {

@Autowired

private MallHotSaleSyncLogService mallHotSaleSyncLogService;

@Override

public void execute(ShardingContext shardingContext) {

try {

//当前服务器所在的任务分片下标

int shardingItem=shardingContext.getShardingItem();

//总运行的任务分片总数

int total=shardingContext.getShardingTotalCount();

//分片规则,没有设置默认是平均分片,如果设置了可以根据这个规则来处理

// String tt=shardingContext.getShardingParameter();

//数据库表有多少条记录

Long counts= mallHotSaleSyncLogService.count();

//每个分片要查询的数据

List<MallHotSaleSyncLog> listdata=null;

//分页查询条件

MallHotSaleSyncLog mallHotSaleSyncLog=new MallHotSaleSyncLog();

//数据总数要大于分片数据，且分片总数大于1时，采用根据分片下标分页查询，否则分片下标为0的获取所有数据,其他分片不获取数据

if(counts.intValue()>=total && total>1) {

//平均分页多少条数据

int pageSize=new BigDecimal(counts.intValue()/total).setScale(0).intValue();

//当多出的数据加给最后一个分片

if(counts.intValue()%total>0 && shardingItem==(total-1)) {

pageSize=pageSize+counts.intValue()%total;

}

Page<MallHotSaleSyncLog> page=new Page<>(shardingItem+1, pageSize);

mallHotSaleSyncLog.setPage(page);

//每个分片要查询的数据

listdata= mallHotSaleSyncLogService.findList(mallHotSaleSyncLog);

}else {

//分片下标为0的获取所有数据,其他分片不获取数据

listdata= mallHotSaleSyncLogService.findList(mallHotSaleSyncLog);

}

if(null!=listdata && !listdata.isEmpty()) {

System.out.println("-----开始执行时间："+DateTimeUtil.getCurDateTime()+"-----");

System.out.println("数据库数据总数："+counts);

System.out.println(shardingContext.getJobName()+"任务分片下标："+shardingItem);

System.out.println("要处理的数据："+JSON.toJSONString(listdata));

System.out.println("根据查询的数据执行统一的任务处理");

}

}catch(Exception e) {

e.printStackTrace();

}

数据库表数据：

A服务器

-----开始执行时间：2018-08-08 16:08:00-----

数据库数据总数：18

dg_mall_job_1任务分片下标：1

要处理的数据：[{"categoryids":"5_6_4_15_31_36","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{},"id":24,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":22,"sqlMap":{},"syncTime":1532069162000},{"categoryids":"5_6_4_15_31_36","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].global"},"id":26,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":22,"sqlMap":{},"syncTime":1532425717000},{"categoryids":"4_5_6_32_15_3","dbName":"mysql","delFlag":0,"fromShopType":6,"global":{"$ref":"$[0].global"},"id":27,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":16,"sqlMap":{},"syncTime":1532425744000},{"categoryids":"5_6_4_15_31_36","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].global"},"id":28,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":23,"sqlMap":{},"syncTime":1532448122000},{"categoryids":"4_5_6_32_15_3","dbName":"mysql","delFlag":0,"fromShopType":6,"global":{"$ref":"$[0].global"},"id":29,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":17,"sqlMap":{},"syncTime":1532448123000},{"categoryids":"5_6_4_15_31_36","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].global"},"id":30,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":24,"sqlMap":{},"syncTime":1533052920000},{"categoryids":"4_5_6_32_15_3","dbName":"mysql","delFlag":0,"fromShopType":6,"global":{"$ref":"$[0].global"},"id":31,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":18,"sqlMap":{},"syncTime":1533052926000},{"categoryids":"5_6_4_15_31_36","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].global"},"id":32,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":25,"sqlMap":{},"syncTime":1533657720000},{"categoryids":"4_5_6_32_15_3","dbName":"mysql","delFlag":0,"fromShopType":6,"global":{"$ref":"$[0].global"},"id":33,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":19,"sqlMap":{},"syncTime":1533657721000}]

根据查询的数据执行统一的任务处理

B服务器

-----开始执行时间：2018-08-08 16:08:00-----

数据库数据总数：18

dg_mall_job_1任务分片下标：0

要处理的数据：[{"categoryids":"5_6_4_15_53_31","dbName":"mysql","delFlag":0,"fro

mShopType":5,"global":{},"id":6,"isNewRecord":false,"isSucess":1,"page":{"count"

:0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page"

,"funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"n

ext":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPag

e":0,"totalPages":0},"period":16,"sqlMap":{},"syncTime":1527479351000},{"categor

yids":"5_6_4_15_53_31","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"

$ref":"$[0].global"},"id":7,"isNewRecord":false,"isSucess":0,"page":{"count":0,"

disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","fu

ncParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next"

:2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0

,"totalPages":0},"period":16,"sqlMap":{},"syncTime":1527504119000},{"categoryids

":"5_6_4_15_53_31","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref

":"$[0].global"},"id":8,"isNewRecord":false,"isSucess":0,"page":{"count":0,"disa

bled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcPa

ram":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"

notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"to

talPages":0},"period":16,"sqlMap":{},"syncTime":1527504173000},{"categoryids":"5

_6_4_15_53_31","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$

[0].global"},"id":9,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled

":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam"

:"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notC

ount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalP

ages":0},"period":16,"sqlMap":{},"syncTime":1527580467000},{"categoryids":"5_6_4

_15_53_31","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].

global"},"id":10,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":t

rue,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":""

,"html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCoun

t":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPage

s":0},"period":17,"sqlMap":{},"syncTime":1530028922000},{"categoryids":"5_6_4_15

_53_31","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].glo

bal"},"id":11,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true

,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","h

tml":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":

false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":

0},"period":18,"sqlMap":{},"syncTime":1530633723000},{"categoryids":"5_6_4_15_53

_31","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].global

"},"id":12,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"f

irst":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html

":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":fal

se,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},

"period":19,"sqlMap":{},"syncTime":1531238520000},{"categoryids":"5_6_4_15_31","

dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].global"},"id

":14,"isNewRecord":false,"isSucess":0,"page":{"count":0,"disabled":true,"first":

0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","

last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"or

derBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"perio

d":21,"sqlMap":{},"syncTime":1532016122000},{"categoryids":"5_6_4_15_31_36","dbN

ame":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].global"},"id":2

2,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"first":0,"

firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","las

t":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"order

By":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":

22,"sqlMap":{},"syncTime":1532067512000}]

根据查询的数据执行统一的任务处理

对比可知 B服务器分片下标为0 ，查询的数据是id为6、7、8、9、10、11、12、14、22共9条

A服务器分片下标为1，查询的数据是id为24、26、27、28、29、30、31、32、33共9条

当突然断掉B服务器时效果如下：

A服务最开始会继续执行两次相同任务（主要是因为该任务的分片总数值没有改变），自动多线程执行任务，即原先A服务的执行是数据分片（查询的数据是id为6、7、8、9、10、11、12、14、22共9条）和B服务执行的数据分片（查询的数据是id为24、26、27、28、29、30、31、32、33共9条）；

-----开始执行时间：2018-08-08 16:28:00-----

数据库数据总数：18

dg_mall_job_1任务分片下标：0

-----开始执行时间：2018-08-08 16:28:00-----

数据库数据总数：18

dg_mall_job_1任务分片下标：1

根据查询的数据执行统一的任务处理

要处理的数据：[{"categoryids":"5_6_4_15_53_31","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{},"id":6,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":16,"sqlMap":{},"syncTime":1527479351000},{"categoryids":"5_6_4_15_53_31","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].global"},"id":7,"isNewRecord":false,"isSucess":0,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":16,"sqlMap":{},"syncTime":1527504119000},{"categoryids":"5_6_4_15_53_31","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].global"},"id":8,"isNewRecord":false,"isSucess":0,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":16,"sqlMap":{},"syncTime":1527504173000},{"categoryids":"5_6_4_15_53_31","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].global"},"id":9,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":16,"sqlMap":{},"syncTime":1527580467000},{"categoryids":"5_6_4_15_53_31","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].global"},"id":10,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":17,"sqlMap":{},"syncTime":1530028922000},{"categoryids":"5_6_4_15_53_31","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].global"},"id":11,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":18,"sqlMap":{},"syncTime":1530633723000},{"categoryids":"5_6_4_15_53_31","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].global"},"id":12,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":19,"sqlMap":{},"syncTime":1531238520000},{"categoryids":"5_6_4_15_31","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].global"},"id":14,"isNewRecord":false,"isSucess":0,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":21,"sqlMap":{},"syncTime":1532016122000},{"categoryids":"5_6_4_15_31_36","dbName":"mysql","delFlag":0,"fromShopType":5,"global":{"$ref":"$[0].global"},"id":22,"isNewRecord":false,"isSucess":1,"page":{"count":0,"disabled":true,"first":0,"firstPage":false,"firstResult":0,"funcName":"page","funcParam":"","html":"","last":0,"lastPage":false,"list":[],"maxResults":-1,"next":2,"notCount":false,"orderBy":"","pageNo":1,"pageSize":-1,"prev":0,"totalPage":0,"totalPages":0},"period":22,"sqlMap":{},"syncTime":1532067512000}]

根据查询的数据执行统一的任务处理

除非手动改变任务的分片总数为1，这样下次任务时间到了A服务的分片下标为0,分片总数为1，A服务查询的数据id为6、7、8、9、10、11、12、14、22、24、26、27、28、29、30、31、32、33共18条

--目前点购商城的elastic-job 采用php后台管理任务+redis(存储任务信息)+api操作接口+elastic-job任务工程来实现动态操作定时任务

--针对php任务可以使用script类型作业+远程ssh调用来实现（理论上可以，需要实践）