Elastic-Job是一个分布式调度解决方案,2.x版本后由两个相互独立的子项目Elastic-Job-Lite和Elastic-Job-Cloud组成。
Elastic-Job-Lite定位为轻量级无中心化解决方案,使用jar包的形式提供分布式任务的协调服务;Elastic-Job-Cloud采用自研Mesos Framework的解决方案,额外提供资源治理、应用分发以及进程隔离等功能。
注:
http://elasticjob.io/docs/elastic-job-lite/01-start/(官方参考)
https://blog.csdn.net/fanfan_v5/article/details/61310045(原理参考)
实际工作中应用Elastic-Job-Lite就可以解决相关问题,所以此处演示springboot应用Elastic-Job-Lite;
代码演示
搭建环境通过示例代码来演示并具体分析,注意elastic-job是不支持单机多实例的(单机多实例会致使分片运行混乱),通过三台虚拟机来演示实现动态切片,es-job结合java需要依赖zookeeper完成。
环境说明:
JDK1.8 springboot 2.1.9
Zookeeper3.4.12
Elastic-Job-Lite 2.1.4
3台虚拟机 contos6 全部安装zookeeper 关闭防火墙
实现逻辑logic:先从spring初始化SpringJobScheduler,然后创建LiteJobConfiguration,再创建JobCoreConfiguration和XXXJobConfiguration;
1 创建分布式定时任务MyJob实现es-job的接口,任务类型依据实现的接口而定
2 创建JobConfiguration配置MyJob的参数属性,初始化时要init一个SpringJobScheduler
3 SpringJobScheduler创建过程中要创建LiteJobConfiguration获取配置
4 LiteJobConfiguration创建过程中需要JobCoreConfiguration和XXXJobConfiguration(任务类型)的配置
step1. maven引入jar包-pom.xml
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.9.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.cc</groupId>
<artifactId>es-job</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>es-job</name>
<description>es-job</description>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
<elastic-job.version>2.1.4</elastic-job.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<!-- elastic-job dependency -->
<dependency>
<groupId>com.dangdang</groupId>
<artifactId>elastic-job-lite-core</artifactId>
<version>${elastic-job.version}</version>
</dependency>
<dependency>
<groupId>com.dangdang</groupId>
<artifactId>elastic-job-lite-spring</artifactId>
<version>${elastic-job.version}</version>
</dependency>
<!-- spring boot dependency -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
</dependency>
<!--
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
-->
</dependencies>
<build>
<finalName>es-job</finalName>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
step2. 定义配置-application.properties
springboot配置文件目前不支持es-job的配置参数,通过zookeeper配置注册中心进行自定义配置,然后通过解析zookeeper创建es-job配置属性;
server.port=8881
#配置zookeeper地址
zookeeper.address=192.168.85.133:2181,192.168.85.138:2181,192.168.85.139:2181
#es-job中提供命名空间配置(通俗释义:可以配置多个,不同的项目可以应用不同空间)
zookeeper.namespace=elastic-job
#配置zookeeper超时时间
zookeeper.connectionTimeout=10000
zookeeper.sessionTimeout=10000
#配置es-job最大重试次数(独有配置项)
zookeeper.maxRetries=3
step3. 装载zookeeper注册中心-RegistryCenterConfig.java
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
/**
* @Configuration标示此类为配置文件属性
* @ConditionalOnExpression当满足参数条件时解析
* 此处设置条件为当配置zookeeper才进行解析,并生成RegistryCenterConfig对象
*/
import com.dangdang.ddframe.job.reg.zookeeper.ZookeeperConfiguration;
import com.dangdang.ddframe.job.reg.zookeeper.ZookeeperRegistryCenter;
@Configuration
@ConditionalOnExpression("'${zookeeper.address}'.length() > 0")
public class RegistryCenterConfig {
/**ZookeeperRegistryCenter为es-job提供的创建对象,定义方法定义初始化方式
* 传入配置文件中自定义的参数,然后通过springboot的@Value进行获取并赋值
* zookeeper.address=192.168.85.133:2181,192.168.85.138:2181,192.168.85.139:2181
* zookeeper.namespace=elastic-job
* zookeeper.connectionTimeout=10000
* zookeeper.sessionTimeout=10000
* zookeeper.maxRetries=3
*
* 最终返回ZookeeperRegistryCenter,将注册中心加载到spring容器中
*/
@Bean(initMethod = "init")
public ZookeeperRegistryCenter registryCenter(@Value("${zookeeper.address}") final String serverLists,
@Value("${zookeeper.namespace}") final String namespace,
@Value("${zookeeper.connectionTimeout}") final int connectionTimeout,
@Value("${zookeeper.sessionTimeout}") final int sessionTimeout,
@Value("${zookeeper.maxRetries}") final int maxRetries) {
ZookeeperConfiguration zookeeperConfiguration = new ZookeeperConfiguration(serverLists, namespace);
zookeeperConfiguration.setConnectionTimeoutMilliseconds(connectionTimeout);
zookeeperConfiguration.setSessionTimeoutMilliseconds(sessionTimeout);
zookeeperConfiguration.setMaxRetries(maxRetries);
return new ZookeeperRegistryCenter(zookeeperConfiguration);
}
}
step4.创建定时任务-MySimpleJob.java
按照逻辑此处要创建获取定时任务及执行定时任务类,在解读具体作业场景为ShardingContext配置任务参数;
import com.dangdang.ddframe.job.api.ShardingContext;
import com.dangdang.ddframe.job.api.simple.SimpleJob;
/**
* 创建分布式任务MySimpleJob实现es-job的SimpleJob接口及其方法
* ShardingContext是分片的上下文对象,从中获取配置参数,然后执行任务
*
*/
public class MySimpleJob implements SimpleJob{
@Override
public void execute(ShardingContext shardingContext) {
/*执行任务需要从ShardingContext中获取配置的任务:先要配置ShardingContext*/
}
}
step5.配置作业(ShardingContext配置参照官方文档)
配置方式
Elastic-Job配置分为3个层级,分别是Core, Type和Root。每个层级使用相似于装饰者模式的方式装配。
Core对应JobCoreConfiguration,用于提供作业核心配置信息,如:作业名称、分片总数、CRON表达式等。
Type对应JobTypeConfiguration,有3个子类分别对应SIMPLE, DATAFLOW和SCRIPT类型作业,提供3种作业需要的不同配置,如:DATAFLOW类型是否流式处理或SCRIPT类型的命令行等。
Root对应JobRootConfiguration,有2个子类分别对应Lite和Cloud部署类型,提供不同部署类型所需的配置,如:Lite类型的是否需要覆盖本地配置或Cloud占用CPU或Memory数量等。
配置参数
第一级配置:细节配置项,配置实现分布式任务的具体细节,例如,分片数、传递参数、执行周期、转移开关、重试等;’
第二级配置:构建配置器注入第一级JobCoreConfiguration,通过反射获取到SimpleJob字节码文件;
第三级配置:重点为分片策略jobShardingStrategyClass的配置,参考:http://elasticjob.io/docs/elastic-job-lite/02-guide/job-sharding-strategy/;
step6.实现作业配置:es-job的配置未能集成到springboot,所以同样需要自定义配置,通过解析的方式来实现;
application.properties添加配置:
#es-job基本作业配置
simpleJob.cron=0/10 * * * * ?
simpleJob.shardingTotalCount=5
simpleJob.shardingItemParameters=0=beijing,1=shanghai,2=changchun,3=changsha,4=hangzhou
simpleJob.jobParameter=source1=public,source2=private
simpleJob.failover=true
simpleJob.monitorExecution=true
simpleJob.monitorPort=8889
simpleJob.maxTimeDiffSeconds=-1
simpleJob.jobShardingStrategyClass=com.dangdang.ddframe.job.lite.api.strategy.impl.AverageAllocationJobShardingStrategy
创建MySimpleJobConfig:
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import com.cc.es.listener.SimpleJobListener;
import com.cc.es.task.MySimpleJob;
import com.dangdang.ddframe.job.api.simple.SimpleJob;
import com.dangdang.ddframe.job.config.JobCoreConfiguration;
import com.dangdang.ddframe.job.config.JobRootConfiguration;
import com.dangdang.ddframe.job.config.simple.SimpleJobConfiguration;
import com.dangdang.ddframe.job.lite.api.JobScheduler;
import com.dangdang.ddframe.job.lite.config.LiteJobConfiguration;
import com.dangdang.ddframe.job.lite.spring.api.SpringJobScheduler;
import com.dangdang.ddframe.job.reg.zookeeper.ZookeeperRegistryCenter;
@Configuration
public class MySimpleJobConfig {
/*
* 将注册中心注入
*/
@Autowired
private ZookeeperRegistryCenter registryCenter;
/*
* 以注入的方式返回一个自定义的MySimpleJob()对象
* 此做法代表MySimpleJobConfig配置类是用来配置MySimpleJob对象
*/
@Bean
public SimpleJob simpleJob() {
return new MySimpleJob();
}
/*
* spet1:定义方法返回JobScheduler实现注入配置属性,进行包装
* 参数:final SimpleJob simpleJob,默认必须与返回MySimpleJob的方法名统一
*/
@Bean(initMethod = "init")
public JobScheduler simpleJobScheduler(final SimpleJob simpleJob,
@Value("${simpleJob.cron}") final String cron,
@Value("${simpleJob.shardingTotalCount}") final int shardingTotalCount,
@Value("${simpleJob.shardingItemParameters}") final String shardingItemParameters,
@Value("${simpleJob.jobParameter}") final String jobParameter,
@Value("${simpleJob.failover}") final boolean failover,
@Value("${simpleJob.monitorExecution}") final boolean monitorExecution,
@Value("${simpleJob.monitorPort}") final int monitorPort,
@Value("${simpleJob.maxTimeDiffSeconds}") final int maxTimeDiffSeconds,
@Value("${simpleJob.jobShardingStrategyClass}") final String jobShardingStrategyClass
) {
return new SpringJobScheduler(simpleJob,
registryCenter,
getLiteJobConfiguration(simpleJob.getClass(),
cron,
shardingTotalCount,
shardingItemParameters,
jobParameter,
failover,
monitorExecution,
monitorPort,
maxTimeDiffSeconds,
jobShardingStrategyClass
),
new SimpleJobListener());//作业监听器,下面构建
}
/*
* spet2:定义方法返回LiteJobConfiguration实现属性配置,参数配置
*/
private LiteJobConfiguration getLiteJobConfiguration(Class<? extends SimpleJob> jobClass, String cron,
int shardingTotalCount, String shardingItemParameters, String jobParameter, boolean failover,
boolean monitorExecution, int monitorPort, int maxTimeDiffSeconds, String jobShardingStrategyClass) {
/**按照官方给出方案进行参数配置
* // 定义作业核心配置
JobCoreConfiguration simpleCoreConfig = JobCoreConfiguration.newBuilder("demoSimpleJob", "0/15 * * * * ?", 10).build();
// 定义SIMPLE类型配置
SimpleJobConfiguration simpleJobConfig = new SimpleJobConfiguration(simpleCoreConfig, SimpleDemoJob.class.getCanonicalName());
// 定义Lite作业根配置
JobRootConfiguration simpleJobRootConfig = LiteJobConfiguration.newBuilder(simpleJobConfig).build();
*/
JobCoreConfiguration jobCoreConfiguration = JobCoreConfiguration.newBuilder(
jobClass.getName(), cron, shardingTotalCount)
.misfire(true)
.failover(failover)
.jobParameter(jobParameter)
.shardingItemParameters(shardingItemParameters)
.build();
SimpleJobConfiguration simpleJobConfiguration = new SimpleJobConfiguration(jobCoreConfiguration, jobClass.getCanonicalName());
//JobRootConfiguration jobRootConfiguration = LiteJobConfiguration.newBuilder(simpleJobConfiguration).build();
LiteJobConfiguration liteJobConfiguration = LiteJobConfiguration.newBuilder(simpleJobConfiguration)
.jobShardingStrategyClass(jobShardingStrategyClass)
.monitorExecution(monitorExecution)
.monitorPort(monitorPort)
.maxTimeDiffSeconds(maxTimeDiffSeconds)
/* overwrite默认为false,代表每次启动运行都已注册中心(zookeeper)配置为准,
* 推荐使用默认,es-job提供的监控管理平台可以对配置的一些维度参数进行修改,可以合理调整配置;
* 如果overwrite(true)那么就是本地的配置会覆盖配置中心,管理平台就失去意义,而且不能动态调整。
*/
.overwrite(true)
.build();
return liteJobConfiguration;
}
}
创建es-job提供的作业监听器SimpleJobListener:
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.dangdang.ddframe.job.executor.ShardingContexts;
import com.dangdang.ddframe.job.lite.api.listener.ElasticJobListener;
public class SimpleJobListener implements ElasticJobListener{
private static Logger LOGGER = LoggerFactory.getLogger(SimpleJobListener.class);
@Override
public void beforeJobExecuted(ShardingContexts shardingContexts) {
LOGGER.info("-------------执行任务前:{}",shardingContexts);
}
@Override
public void afterJobExecuted(ShardingContexts shardingContexts) {
LOGGER.info("=============执行任务后:{}",shardingContexts);
}
}
step7.完善MySimpleJob自定义的作业任务,运行测试
import com.dangdang.ddframe.job.api.ShardingContext;
import com.dangdang.ddframe.job.api.simple.SimpleJob;
/**
* 创建分布式任务MySimpleJob实现es-job的SimpleJob接口及其方法
* ShardingContext是分片的上下文对象,从中获取配置参数,然后执行任务
*
*/
public class MySimpleJob implements SimpleJob{
@Override
public void execute(ShardingContext shardingContext) {
System.out.println("-----开始任务-----");
System.out.println(shardingContext.getJobName());
System.out.println(shardingContext.getJobParameter());
System.out.println(shardingContext.getShardingItem());
System.out.println(shardingContext.getShardingParameter());
System.out.println(shardingContext.getShardingTotalCount());
System.out.println("当前线程:-----"+Thread.currentThread().getName());
System.out.println("-----任务执行结束-----");
}
}
运行测试<项目结构>:运行Application.java
输出结果:
间隔10m后再次执行、以此类推。