SpringBoot定时任务 - 什么是ElasticJob？如何集成ElasticJob实现分布式任务调度？

🚀 优质资源分享 🚀

学习路线指引（点击解锁）	知识定位	人群定位
🧡 Python实战微信订餐小程序 🧡	进阶级	本课程是python flask+微信小程序的完美结合，从项目搭建到腾讯云部署上线，打造一个全栈订餐系统。
💛Python量化交易实战💛	入门级	手把手带你打造一个易扩展、更安全、效率更高的量化交易系统
作者：@pdai本文为作者原创，转载请注明出处：https://blog.csdn.net/pengdai/p/16542459.html

---

内容目录

• 知识准备
  • 什么是ElasticJob
• 实现案例
  • POM依赖
  • 基础Entity和Dao
  • Job定义
  • 装载配置
  • 测试触发
  • 简单测试
• 运维控制台
  • 启动运行
  • 全局配置
  • 作业操作
  • 作业历史
• 参考资料
• 更多内容

---

前文展示quartz实现基于数据库的分布式任务管理和job生命周期的控制，那在分布式场景下如何解决弹性调度、资源管控、以及作业治理等呢？针对这些功能前当当团队开发了ElasticJob，2020 年 5 月 28 日ElasticJob成为 Apache ShardingSphere 的子项目；本文介绍ElasticJob以及SpringBoot的集成。@pdai

• SpringBoot定时任务 - 分布式elastic-job方式
  • 知识准备
    • 分布式任务知识体系
    • 什么是ElasticJob
  • 实现案例
    • POM依赖
    • 基础Entity和Dao
    • Job定义
    • 装载配置
    • 测试触发
    • 简单测试
  • 运维控制台
    • 启动运行
    • 全局配置
    • 作业操作
    • 作业历史
  • 参考资料

知识准备

需要对分布式任务的知识体系和ElasticJob有基本的理解。@pdai

什么是ElasticJob

ElasticJob 是面向互联网生态和海量任务的分布式调度解决方案，由两个相互独立的子项目 ElasticJob-Lite 和 ElasticJob-Cloud 组成。 它通过弹性调度、资源管控、以及作业治理的功能，打造一个适用于互联网场景的分布式调度解决方案，并通过开放的架构设计，提供多元化的作业生态。 它的各个产品使用统一的作业 API，开发者仅需一次开发，即可随意部署。ElasticJob 已于 2020 年 5 月 28 日成为 Apache ShardingSphere 的子项目。

使用 ElasticJob 能够让开发工程师不再担心任务的线性吞吐量提升等非功能需求，使他们能够更加专注于面向业务编码设计； 同时，它也能够解放运维工程师，使他们不必再担心任务的可用性和相关管理需求，只通过轻松的增加服务节点即可达到自动化运维的目的。

ElasticJob-Lite: 定位为轻量级无中心化解决方案，使用 jar 的形式提供分布式任务的协调服务。

Elasticjob-lite的案例- SpringBoot集成定时任务 - 分布式Elasticjob-lite方式

ElasticJob-Cloud： 采用自研 Mesos Framework 的解决方案，额外提供资源治理、应用分发以及进程隔离等功能。

ElasticJob-Lite和ElasticJob-Cloud的区别

	ElasticJob-Lite	ElasticJob-Cloud
无中心化	是	否
资源分配	不支持	支持
作业模式	常驻	常驻 + 瞬时
部署依赖	ZooKeeper	ZooKeeper + Mesos

实现案例

本例将展示ElasticJob-Lite集成Springboot的案例，案例参考自ElasticJob的官网，同时做了一些调整和issue修复。

POM依赖

ElaticJob针对SpringBoot集成的starter依赖，针对错误通知的依赖elasticjob-error-handler-xxx（如果需要的话）

对任务的记录和追踪是存放在DB的，所以需要配置JPA和MySQL/H2等。

<dependency>
    <groupId>org.springframework.bootgroupId>
    <artifactId>spring-boot-starter-webartifactId>
dependency>
<dependency>
    <groupId>org.springframework.bootgroupId>
    <artifactId>spring-boot-starter-data-jpaartifactId>
dependency>

<dependency>
    <groupId>org.apache.shardingsphere.elasticjobgroupId>
    <artifactId>elasticjob-lite-spring-boot-starterartifactId>
    <version>3.0.1version>
dependency>
<dependency>
    <groupId>org.apache.shardingsphere.elasticjobgroupId>
    <artifactId>elasticjob-error-handler-dingtalkartifactId>
    <version>3.0.1version>
dependency>
<dependency>
    <groupId>org.apache.shardingsphere.elasticjobgroupId>
    <artifactId>elasticjob-error-handler-wechatartifactId>
    <version>3.0.1version>
dependency>
<dependency>
    <groupId>org.apache.shardingsphere.elasticjobgroupId>
    <artifactId>elasticjob-error-handler-emailartifactId>
    <version>3.0.1version>
dependency>
<dependency>
    <groupId>org.apache.curatorgroupId>
    <artifactId>curator-testartifactId>
    <version>5.2.0version>
dependency>

<dependency>
    <groupId>mysqlgroupId>
    <artifactId>mysql-connector-javaartifactId>
    <version>5.1.42version>
    <scope>runtimescope>
dependency>
<dependency>
    <groupId>com.h2databasegroupId>
    <artifactId>h2artifactId>
    <scope>runtimescope>
dependency>
<dependency>
    <groupId>org.springframework.bootgroupId>
    <artifactId>spring-boot-starter-testartifactId>
    <scope>testscope>
dependency>

基础Entity和Dao

Foo

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package tech.pdai.springboot.elasticjob.lite.entity;

import java.io.Serializable;

public final class Foo implements Serializable {

    private static final long serialVersionUID = 2706842871078949451L;

    private final long id;

    private final String location;

    private Status status;

    public Foo(final long id, final String location, final Status status) {
        this.id = id;
        this.location = location;
        this.status = status;
    }

    public long getId() {
        return id;
    }

    public String getLocation() {
        return location;
    }

    public Status getStatus() {
        return status;
    }

    public void setStatus(final Status status) {
        this.status = status;
    }

    @Override
    public String toString() {
        return String.format("id: %s, location: %s, status: %s", id, location, status);
    }

    public enum Status {
        TODO,
        COMPLETED
    }
}
折叠 

dao

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package tech.pdai.springboot.elasticjob.lite.repository;

import org.springframework.stereotype.Repository;
import tech.pdai.springboot.elasticjob.lite.entity.Foo;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

@Repository
public class FooRepository {
    
    private final Map data = new ConcurrentHashMap<>(300, 1);
 
 public FooRepository() {
 init();
 }
 
 private void init() {
 addData(0L, 100L, "Beijing");
 addData(100L, 200L, "Shanghai");
 addData(200L, 300L, "Guangzhou");
 }
 
 private void addData(final long idFrom, final long idTo, final String location) {
 for (long i = idFrom; i < idTo; i++) {
 data.put(i, new Foo(i, location, Foo.Status.TODO));
 }
 }
 
 public List findTodoData(final String location, final int limit) {
 List result = new ArrayList<>(limit);
 int count = 0;
 for (Map.Entry each : data.entrySet()) {
 Foo foo = each.getValue();
 if (foo.getLocation().equals(location) && foo.getStatus() == Foo.Status.TODO) {
 result.add(foo);
 count++;
 if (count == limit) {
 break;
 }
 }
 }
 return result;
 }
 
 public void setCompleted(final long id) {
 data.get(id).setStatus(Foo.Status.COMPLETED);
 }
}
折叠 

Job定义

• 基本的Job, 实现SimpleJob接口

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package tech.pdai.springboot.elasticjob.lite.job;

import java.time.LocalDateTime;
import java.util.List;

import org.apache.shardingsphere.elasticjob.api.ShardingContext;
import org.apache.shardingsphere.elasticjob.simple.job.SimpleJob;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import tech.pdai.springboot.elasticjob.lite.entity.Foo;
import tech.pdai.springboot.elasticjob.lite.repository.FooRepository;

@Component
public class SpringBootSimpleJob implements SimpleJob {

    private final Logger logger = LoggerFactory.getLogger(SpringBootSimpleJob.class);

    @Autowired
    private FooRepository fooRepository;

    @Override
    public void execute(final ShardingContext shardingContext) {
        logger.info("Item: {} | Time: {} | Thread: {} | {}",
                shardingContext.getShardingItem(), LocalDateTime.now(), Thread.currentThread().getId(), "SIMPLE");
        List data = fooRepository.findTodoData(shardingContext.getShardingParameter(), 10);
 for (Foo each : data) {
 fooRepository.setCompleted(each.getId());
 }
 }
}

• 数据流处理Job, 实现DataflowJob接口

包含两个主要方法，一个是获取数据的方法fetchData， 一个是处理数据的方法processData

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package tech.pdai.springboot.elasticjob.lite.job;

import java.time.LocalDateTime;
import java.util.List;

import javax.annotation.Resource;

import org.apache.shardingsphere.elasticjob.api.ShardingContext;
import org.apache.shardingsphere.elasticjob.dataflow.job.DataflowJob;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
import tech.pdai.springboot.elasticjob.lite.entity.Foo;
import tech.pdai.springboot.elasticjob.lite.repository.FooRepository;

@Component
public class SpringBootDataflowJob implements DataflowJob {

 private final Logger logger = LoggerFactory.getLogger(SpringBootDataflowJob.class);

 @Resource
 private FooRepository fooRepository;

 @Override
 public List fetchData(final ShardingContext shardingContext) {
 logger.info("Item: {} | Time: {} | Thread: {} | {}",
 shardingContext.getShardingItem(), LocalDateTime.now(), Thread.currentThread().getId(), "DATAFLOW FETCH");
 return fooRepository.findTodoData(shardingContext.getShardingParameter(), 10);
 }

 @Override
 public void processData(final ShardingContext shardingContext, final List data) {
 logger.info("Item: {} | Time: {} | Thread: {} | {}",
 shardingContext.getShardingItem(), LocalDateTime.now(), Thread.currentThread().getId(), "DATAFLOW PROCESS");
 for (Foo each : data) {
 fooRepository.setCompleted(each.getId());
 }
 }
}
折叠 

• 错误通知处理 - Email

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package tech.pdai.springboot.elasticjob.lite.job;

import org.apache.shardingsphere.elasticjob.api.ShardingContext;
import org.apache.shardingsphere.elasticjob.simple.job.SimpleJob;
import org.springframework.stereotype.Component;

@Component
public class SpringBootOccurErrorNoticeEmailJob implements SimpleJob {
    
    @Override
    public void execute(final ShardingContext shardingContext) {
        throw new RuntimeException(String.format("An exception has occurred in Job, The parameter is %s", shardingContext.getShardingParameter()));
    }
}

• 错误通知处理 - Wechat

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package tech.pdai.springboot.elasticjob.lite.job;

import org.apache.shardingsphere.elasticjob.api.ShardingContext;
import org.apache.shardingsphere.elasticjob.simple.job.SimpleJob;
import org.springframework.stereotype.Component;

@Component
public class SpringBootOccurErro