springbatch学习

已于 2022-10-14 10:30:00 修改
阅读量292 收藏
点赞数
分类专栏： java 文章标签： mysql spring java
于 2022-07-25 22:33:00 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/Proxbj/article/details/125984850
版权

一、sql建表语句

/*
 Navicat Premium Data Transfer

 Source Server         : 120.79.157.16
 Source Server Type    : MySQL
 Source Server Version : 80029
 Source Host           : 120.79.157.16:3307
 Source Schema         : spring_batch

 Target Server Type    : MySQL
 Target Server Version : 80029
 File Encoding         : 65001

 Date: 13/10/2022 21:04:19
*/

SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;

-- ----------------------------
-- Table structure for BATCH_JOB_EXECUTION
-- ----------------------------
DROP TABLE IF EXISTS `BATCH_JOB_EXECUTION`;
CREATE TABLE `BATCH_JOB_EXECUTION` (
  `JOB_EXECUTION_ID` bigint NOT NULL,
  `VERSION` bigint DEFAULT NULL,
  `JOB_INSTANCE_ID` bigint DEFAULT NULL,
  `CREATE_TIME` datetime DEFAULT NULL,
  `START_TIME` datetime DEFAULT NULL,
  `END_TIME` datetime DEFAULT NULL,
  `STATUS` varchar(10) COLLATE utf8mb4_general_ci DEFAULT NULL,
  `EXIT_CODE` varchar(2500) COLLATE utf8mb4_general_ci DEFAULT NULL,
  `EXIT_MESSAGE` varchar(2500) COLLATE utf8mb4_general_ci DEFAULT NULL,
  `LAST_UPDATED` datetime(6) DEFAULT NULL,
  `JOB_CONFIGURATION_LOCATION` varchar(255) COLLATE utf8mb4_general_ci DEFAULT NULL,
  PRIMARY KEY (`JOB_EXECUTION_ID`),
  KEY `JOB_INST_EXEC_FK` (`JOB_INSTANCE_ID`),
  CONSTRAINT `JOB_INST_EXEC_FK` FOREIGN KEY (`JOB_INSTANCE_ID`) REFERENCES `BATCH_JOB_INSTANCE` (`JOB_INSTANCE_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;

-- ----------------------------
-- Table structure for BATCH_JOB_EXECUTION_CONTEXT
-- ----------------------------
DROP TABLE IF EXISTS `BATCH_JOB_EXECUTION_CONTEXT`;
CREATE TABLE `BATCH_JOB_EXECUTION_CONTEXT` (
  `JOB_EXECUTION_ID` bigint NOT NULL,
  `SHORT_CONTEXT` varchar(2500) COLLATE utf8mb4_general_ci NOT NULL,
  `SERIALIZED_CONTEXT` text COLLATE utf8mb4_general_ci,
  PRIMARY KEY (`JOB_EXECUTION_ID`),
  CONSTRAINT `JOB_EXEC_CTX_FK` FOREIGN KEY (`JOB_EXECUTION_ID`) REFERENCES `BATCH_JOB_EXECUTION` (`JOB_EXECUTION_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;

-- ----------------------------
-- Table structure for BATCH_JOB_EXECUTION_PARAMS
-- ----------------------------
DROP TABLE IF EXISTS `BATCH_JOB_EXECUTION_PARAMS`;
CREATE TABLE `BATCH_JOB_EXECUTION_PARAMS` (
  `JOB_EXECUTION_ID` bigint NOT NULL,
  `TYPE_CD` varchar(6) COLLATE utf8mb4_general_ci NOT NULL,
  `KEY_NAME` varchar(100) COLLATE utf8mb4_general_ci NOT NULL,
  `STRING_VAL` varchar(250) COLLATE utf8mb4_general_ci DEFAULT NULL,
  `DATE_VAL` datetime(6) DEFAULT NULL,
  `LONG_VAL` bigint DEFAULT NULL,
  `DOUBLE_VAL` double DEFAULT NULL,
  `IDENTIFYING` char(1) COLLATE utf8mb4_general_ci NOT NULL,
  KEY `JOB_EXEC_PARAMS_FK` (`JOB_EXECUTION_ID`),
  CONSTRAINT `JOB_EXEC_PARAMS_FK` FOREIGN KEY (`JOB_EXECUTION_ID`) REFERENCES `BATCH_JOB_EXECUTION` (`JOB_EXECUTION_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;

-- ----------------------------
-- Table structure for BATCH_JOB_EXECUTION_SEQ
-- ----------------------------
DROP TABLE IF EXISTS `BATCH_JOB_EXECUTION_SEQ`;
CREATE TABLE `BATCH_JOB_EXECUTION_SEQ` (
  `ID` bigint NOT NULL,
  `UNIQUE_KEY` char(1) COLLATE utf8mb4_general_ci NOT NULL,
  UNIQUE KEY `UNIQUE_KEY_UN` (`UNIQUE_KEY`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;

-- ----------------------------
-- Table structure for BATCH_JOB_INSTANCE
-- ----------------------------
DROP TABLE IF EXISTS `BATCH_JOB_INSTANCE`;
CREATE TABLE `BATCH_JOB_INSTANCE` (
  `JOB_INSTANCE_ID` bigint NOT NULL,
  `VERSION` bigint DEFAULT NULL,
  `JOB_NAME` varchar(100) COLLATE utf8mb4_general_ci NOT NULL,
  `JOB_KEY` varchar(32) COLLATE utf8mb4_general_ci NOT NULL,
  PRIMARY KEY (`JOB_INSTANCE_ID`),
  UNIQUE KEY `JOB_INST_UN` (`JOB_NAME`,`JOB_KEY`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;

-- ----------------------------
-- Table structure for BATCH_JOB_SEQ
-- ----------------------------
DROP TABLE IF EXISTS `BATCH_JOB_SEQ`;
CREATE TABLE `BATCH_JOB_SEQ` (
  `ID` bigint NOT NULL,
  `UNIQUE_KEY` char(1) COLLATE utf8mb4_general_ci NOT NULL,
  UNIQUE KEY `UNIQUE_KEY_UN` (`UNIQUE_KEY`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;

-- ----------------------------
-- Table structure for BATCH_STEP_EXECUTION
-- ----------------------------
DROP TABLE IF EXISTS `BATCH_STEP_EXECUTION`;
CREATE TABLE `BATCH_STEP_EXECUTION` (
  `STEP_EXECUTION_ID` bigint NOT NULL,
  `VERSION` bigint NOT NULL,
  `STEP_NAME` varchar(100) COLLATE utf8mb4_general_ci NOT NULL,
  `JOB_EXECUTION_ID` bigint NOT NULL,
  `CREATE_TIME` datetime(3) DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP(3),
  `START_TIME` datetime(3) DEFAULT NULL,
  `END_TIME` datetime(3) DEFAULT NULL,
  `STATUS` varchar(10) COLLATE utf8mb4_general_ci DEFAULT NULL,
  `COMMIT_COUNT` bigint DEFAULT NULL,
  `READ_COUNT` bigint DEFAULT NULL,
  `FILTER_COUNT` bigint DEFAULT NULL,
  `WRITE_COUNT` bigint DEFAULT NULL,
  `READ_SKIP_COUNT` bigint DEFAULT NULL,
  `WRITE_SKIP_COUNT` bigint DEFAULT NULL,
  `PROCESS_SKIP_COUNT` bigint DEFAULT NULL,
  `ROLLBACK_COUNT` bigint DEFAULT NULL,
  `EXIT_CODE` varchar(2500) COLLATE utf8mb4_general_ci DEFAULT NULL,
  `EXIT_MESSAGE` varchar(2500) COLLATE utf8mb4_general_ci DEFAULT NULL,
  `LAST_UPDATED` datetime(6) DEFAULT NULL,
  PRIMARY KEY (`STEP_EXECUTION_ID`),
  KEY `JOB_EXEC_STEP_FK` (`JOB_EXECUTION_ID`),
  CONSTRAINT `JOB_EXEC_STEP_FK` FOREIGN KEY (`JOB_EXECUTION_ID`) REFERENCES `BATCH_JOB_EXECUTION` (`JOB_EXECUTION_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;

-- ----------------------------
-- Table structure for BATCH_STEP_EXECUTION_CONTEXT
-- ----------------------------
DROP TABLE IF EXISTS `BATCH_STEP_EXECUTION_CONTEXT`;
CREATE TABLE `BATCH_STEP_EXECUTION_CONTEXT` (
  `STEP_EXECUTION_ID` bigint NOT NULL,
  `SHORT_CONTEXT` varchar(2500) COLLATE utf8mb4_general_ci NOT NULL,
  `SERIALIZED_CONTEXT` text COLLATE utf8mb4_general_ci,
  PRIMARY KEY (`STEP_EXECUTION_ID`),
  CONSTRAINT `STEP_EXEC_CTX_FK` FOREIGN KEY (`STEP_EXECUTION_ID`) REFERENCES `BATCH_STEP_EXECUTION` (`STEP_EXECUTION_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;

-- ----------------------------
-- Table structure for BATCH_STEP_EXECUTION_SEQ
-- ----------------------------
DROP TABLE IF EXISTS `BATCH_STEP_EXECUTION_SEQ`;
CREATE TABLE `BATCH_STEP_EXECUTION_SEQ` (
  `ID` bigint NOT NULL,
  `UNIQUE_KEY` char(1) COLLATE utf8mb4_general_ci NOT NULL,
  UNIQUE KEY `UNIQUE_KEY_UN` (`UNIQUE_KEY`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;

-- ----------------------------
-- Table structure for tb_student
-- ----------------------------
DROP TABLE IF EXISTS `tb_student`;
CREATE TABLE `tb_student` (
  `id` bigint NOT NULL AUTO_INCREMENT COMMENT 'id',
  `age` int DEFAULT NULL COMMENT '年龄',
  `name` varchar(11) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '名字',
  `birthday` date DEFAULT NULL COMMENT '生日',
  PRIMARY KEY (`id`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=119511 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci ROW_FORMAT=DYNAMIC;

-- ----------------------------
-- Procedure structure for rand_data
-- ----------------------------
DROP PROCEDURE IF EXISTS `rand_data`;
delimiter ;;
CREATE PROCEDURE `rand_data`(in num int)
begin
declare str char(62) default 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789'; 
declare str2 char(2);
declare i int default 0;
while i<num do
set str2=concat(substring(str,1+floor(rand()*61),1),substring(str,1+floor(rand()*61),1));
set i=i+1;
insert into tb_student(name,age,birthday) values (str2,floor(rand()*num),CURDATE());
end while;
 end
;;
delimiter ;

SET FOREIGN_KEY_CHECKS = 1;


INSERT INTO BATCH_JOB_SEQ (ID, UNIQUE_KEY) select * from (select 0 as ID, '0' as UNIQUE_KEY) as tmp where not exists(select * from BATCH_JOB_SEQ);

INSERT INTO BATCH_JOB_EXECUTION_SEQ (ID, UNIQUE_KEY) select * from (select 0 as ID, '0' as UNIQUE_KEY) as tmp where not exists(select * from BATCH_JOB_EXECUTION_SEQ);

INSERT INTO BATCH_STEP_EXECUTION_SEQ (ID, UNIQUE_KEY) select * from (select 0 as ID, '0' as UNIQUE_KEY) as tmp where not exists(select * from BATCH_STEP_EXECUTION_SEQ);

二、基本概念

  • 架构图
    在这里插入图片描述

在这里插入图片描述

三、基本概念

  1. Job，创建Job实例。包括创建父任务、子任务
    Job的实现类有SimpleJob和flowJob，Job
  2. JobInstance,Job的实例
    JobInstance = Job + JobParameters
  • Step，创建Step（Job的实现步骤）
  • Flow，多个Step的组合，可以在其他job中共用
  • Split，多线程处理job
  • Decider，决策器，代表不同步骤执行完之后，需要收到什么信息才能继续往下面执行
  • ChunkListener，Step的块监听器
  • JobListener，Job的监听器
  • StepListener,也是Step的监听器
Proxbj
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
使用spring batch需要在数据库建立的几个表——建表语句
07-31
使用spring batch需要在数据库建立的几个表——建表语句（BATCH_JOB_INSTANCE、BATCH_JOB_EXECUTION、BATCH_JOB_EXECUTION_CONTEXT、`BATCH_JOB_EXECUTION_PARAMS` 、`BATCH_JOB_EXECUTION_SEQ` 、`BATCH_JOB_SEQ` 、`BATCH_STEP_EXECUTION` 、`BATCH_STEP_EXECUTION_CONTEXT` 、`BATCH_STEP_EXECUTION_SEQ` ）
springBatch 组件学习教程源码
04-04
springBatch 学习教程源码全套 对应博客解析：https://blog.csdn.net/wuzhiwei549/article/details/85392128
Spring Batch学习demo项目源码
10-30
Spring Batch学习过程中练习的一个简单完整的demo项目源码
史上最全的SpringBatch学习教程
12-14
写在前面： 我是「境里婆娑」。我还是从前那个少年，没有一丝丝改变，时间只不过是考验，种在心中信念丝毫未减，眼前这个少年，还是最初那张脸，面前再多艰险不退却。 写博客的目的就是分享给大家一起学习交流，如果您对 Java感兴趣，可以关注我，我们一起学习 相关文章： 今天我把写过的SpringBatch系列的文章全部整理在此，方便大家学习查看！ SpringBatch读单个文件(FlatFileItemReader)和写单个文件(FlatFileItemWriter)(一) SpringBatch顺序读取多文件(MultiResourceItemReader)和顺序写文件(MultiResource
SpringBatch学习入门
01-07
IDEA，Springbatch的入门学习资源，包括对账文件，一起学习指正，邮箱：furui-2008@163.com
详解Spring batch 入门学习教程(附源码)
08-28
本篇文章主要介绍了Spring batch 入门学习教程(附源码)，小编觉得挺不错的，现在分享给大家，也给大家做个参考。一起跟随小编过来看看吧
spring batch学习
dzl84394的专栏
02-21 1112
分类 检查文件是否存在 ftp拉文件（复制文件） 删除文件 读取文件 写文件 写数据库 读取数据库 统计后写数据库 校验数据是否存在
学习Spring-batch(一)-------入门
梧桐树的博客
03-12 725
就是将数据分批次进行处理的过程。比如：银行对账逻辑，跨系统数据同步等。系统A从数据库中导出数据到文件，系统B读取文件数据并写入到数据库典型批处理特点：自动执行，根据系统设定的工作步骤自动完成数据量大，少则百万，多则上千万甚至上亿。(如果是10亿，100亿那只能上大数据了)定时执行，比如：每天，每周，每月执行。
Spring batch 学习笔记
notMoonHeart的博客
04-25 713
一、Spring batch 简介 Spring batch 是Spring系列处理批量数据的框架。 主要构成如图所示： JobRepository用来注册job的容器，用来存储 Job 在运行过程中的状态信息，如果失败了，可以重失败的地方重新发起。 JobLauncher 用来启动Job的接口，可以不指定。会在项目启动时，会直接启动批量逻辑，可以再YML文件里配置。 Jo...
Spring Batch入门学习
z2431435的博客
08-29 520
dblink（Database Link，数据库的链接像电话线一样是一个通道，要跨本地数据库访问另外一个数据库表中的数据时，本地数据库中就必须要创建远程数据库的dblink,通过dblink本地数据库可以像访问本地数据库一样访问远程数据库表中的数据。）要转换成文件加载方式，所以要改用spring batch。Spring Batch 是一个轻量级的、完善的批处理框架,旨在帮助企业建立健壮、高效的批处理应用。Spring Batch 提供了大量可重用的组件。...
spring batch 学习多种场景练习demo项目源码
11-09
spring batch 学习多种场景练习demo项目源码，包括: 1.读取文件--处理--写入文件; 2.读取文件--处理--写入数据库 ; 3.读数据库--处理--写入数据库; 4.读数据库--处理--写入文件; 文件类型包括：CSV,TXT,JSON,XML; 数据库读写操作包括：JDBC 游标,JDBC分页,JDBC存储过程； Hibernate 游标，Hibernate分页，ibatis分页，jpa分页。
pro spring batch
06-05
Since its release, Spring Framework has transformed virtually every aspect of Java development including web applications, security, aspect-oriented programming, persistence, and messaging. Spring Batch, one of its newer additions, now brings the same familiar Spring idioms to batch processing. Spring Batch addresses the needs of any batch process, from the complex calculations performed in the biggest financial institutions to simple data migrations that occur with many software development projects. Pro Spring Batch is intended to answer three questions: *What? What is batch processing? What does it entail? What makes it different from the other applications we are developing? What are the challenges inherent in the development of a batch process? *Why? Why do batch processing? Why can't we just process things as we get them? Why do we do batch processing differently than the web applications that we currently work on? *How? How to implement a robust, scalable, distributed batch processing system using open-source frameworks Pro Spring Batch gives concrete examples of how each piece of functionality is used and why it would be used in a real-world application. This includes providing tips that the 'school of hard knocks' has taught author Michel Minella during his experience with Spring Batch. Pro Spring Batch includes examples of I/O options that are not mentioned in the official user's guide, as well as performance tips on things like how to limit the impact of maintaining the state of your jobs. The author also walks you through, from end to end, the design and implementation of a batch process based upon a theoretical real-world example. This includes basic project setup, implementation, testing, tuning and scaling for large volumes. What you'll learn * Batch concepts and how they relate to the Spring Batch framework * How to use declarative I/O using the Spring Batch readers/writers * Data integrity techniques used by Spring Batch, including transactions and job state/restartability * How to scale batch jobs via distributed batch processing * How to handle testing batch processes (Unit and functional) Who this book is for * Java developers with Spring experience. * Java Architects designing batch solutions More specifically, this book is intended for those who have a solid foundation in the core Java platform. Batch processing covers a wide spectrum of topics, not all of which are covered in detail in this book. Concepts in Java which the reader should be comfortable with include file I/O, JDBC, and transactions. Given that Spring Batch is a framework built upon the open-source IoC container Spring, which will not be covered in this book, it is expected that the reader will be familiar with its concepts and conventions. With that in mind, the reader is not expected to have any prior exposure to the Spring Batch framework. All concepts related to it will be explained in detail, with working examples. Table of Contents * Batch and Spring * Spring Batch 101 * Sample Job * Understanding Jobs and Steps * Job Repository and Metadata * Running a Job * Readers * Item Processors * Item Writers * Sample Application * Scaling and Tuning * Testing Batch Processes
Pro Spring Batch
10-09
Java Spring Batch pdf
SpringBatch数据库建表语句.txt
07-02
SpringBatch数据库建表语句，存储springBatch批处理过程中需要保存的数据和步骤信息
pro spring batch 源码
09-21
英文版、可用来学习和了解spring batch使用、包括扩展、并行、分布式处理等
非常强，批处理框架 Spring Batch 就该这么用！（场景实战）
weixin_37097680的博客
07-18 818
点击上方“果汁简历”，选择“置顶公众号”来源：https://urlify.cn/NFb6Vr前言概念词就不多说了，我简单地介绍下 ， spring batch 是一个 方便使用的 较健全的 批处理 框架。为什么说是方便使用的，因为这是 基于spring的一个框架，接入简单、易理解、流程分明。为什么说是较健全的， 因为它提供了往常我们在对大批量数据进行处理时需要考虑到的 日志跟踪、事务粒度调配、可...
Spring Batch 批处理框架优化实践，效率嘎嘎高！
最新发布
Java知音
10-16 345
一、Spring Batch简介1 框架概述2 核心概念和组件二、批处理优化实践4.1 使用Spring Boot Actuator进行监控4.2 使用管理控制台来监控3.1 批处理启动前校验3.2 读写校验2.1 多进程处理2.2 多线程处理1.1 分页处理数据1.2 使用读写缓存1.3 行级别写操作1 减少读写次数2 并发处理任务3 提高数据校验准确性4 监控批处理任务三、实践示例3.1 修改...
Spring Batch元数据表不是自动创建的吗？
一名可爱的技术搬运工
05-24 2069
如果jobRepository与创建MapJobRepositoryFactoryBean （在内存中的元数据），Spring批处理作业运行成功。 spring-config.xml <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobReposi...
Spring Batch 批处理数据表
langfeiyes的博客
02-04 2004
了解返回状态ExitStatus状态类后，接下来一起学习一下Spring Batch 批处理核心的数据表
Spring batch学习资料
05-10
以下是一些 Spring Batch 学习资料： 1. 官方文档：https://docs.spring.io/spring-batch/docs/current/reference/html/index.html 2. Spring Batch 入门指南：https://www.baeldung.com/spring-batch 3. Spring ...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值