Spring Batch 3.0 知识点总结

来自官网总结
https://docs.spring.io/spring-batch/3.0.x/reference/htmlsingle/

Spring Batch架构

Spring Batch组件

• 运行模型 JobLauncher Job Step
• 重试机制 RetryTemplate
• 输入输出 ItemReader ItemWriter

批处理实现类型

1. Normal processing in a batch window 普通批处理
2. Concurrent batch / on-line processing 在线实时批处理
3. Parallel Processing 并行批处理
4. Partitioning 分区批处理
   

分区机制

1. Fixed and Even Break-Up of Record Set固定值分解
2. Breakup by a Key Column 关键字
3. Assigned to a batch instance via a partitioning table 分区表
4. Assigned to a batch instance by a portion of the value 依赖关键字取值的部分来拆分
5. Breakup by Views 视图分解
6. Addition of a Processing Indicator 额外指定的处理策略
7. Extract Table to a Flat File 导出表数据为分布式文件
8. Use of a Hashing Column 使用hash列

样例

JOB样例

JOB运行状态三种 STARTED FAILED COMPLETED

<job id="footballJob" job-repository="specialRepository">
 
    <step id="playerload"          parent="s1" next="gameLoad"/>
    <step id="gameLoad"            parent="s3" next="playerSummarization"/>
    <step id="playerSummarization" parent="s3"/>
  <listeners>
 
        <listener ref="sampleListener"/>
 
  </listeners>
</job>

step样例

    <job id="ioSampleJob">
        <step id="step1">
            <tasklet>
                <chunk reader="itemReader" writer="itemWriter" commit-interval="2"/>
            </tasklet>
        </step>
    </job>

JobRepository 样例以及事务控制

<job-repository id="jobRepository"
    data-source="dataSource"
    transaction-manager="transactionManager"
    isolation-level-for-create="SERIALIZABLE" REPEATABLE_READ
    table-prefix="BATCH_"<!-- 可以改 !!!-->
max-varchar-length="1000"/>

–事务

<aop:config>
    <aop:advisor
           pointcut="execution(* org.springframework.batch.core..*Repository+.*(..))"/>
    <advice-ref="txAdvice" />
</aop:config>

<tx:advice id="txAdvice" transaction-manager="transactionManager">
    <tx:attributes>
        <tx:method name="*" />
    </tx:attributes>
</tx:advice>

更改项目的数据库类型

<bean id="jobRepository" class="org...JobRepositoryFactoryBean">
    <property name="databaseType" *value="db2"*/>
    <property name="dataSource" ref="dataSource"/>
</bean>

同步模型

<bean id="jobLauncher"
      class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
    <property name="jobRepository" ref="jobRepository" />
</bean>

异步模型

<bean id="jobLauncher"
      class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
    <property name="jobRepository" ref="jobRepository" />
    <property name="taskExecutor">
        <bean class="org.springframework.core.task.SimpleAsyncTaskExecutor" />
    </property>
</bean>

JobExplorer JOB阅读器

查询已经存在的execution p:tablePrefix可指定前缀

<bean id="jobExplorer" class="org.spr...JobExplorerFactoryBean"
      p:dataSource-ref="dataSource" ***p:tablePrefix="BATCH_"*** />

RepeatTemplate 循环执行

RepeatOperations 实现类RepeatTemplate
RepeatContext RepeatStatus CONTINUABLE/ FINISHED
并行处理 TaskExecutorRepeatTemplate SynchronousTaskExecutor RepeatCallback

多线程

略

并行

一种方案是 使用flow属性

<job id="job1">
	<split id="split1" task-executor="taskExecutor" next="step4">
		<flow>
			<step id="step1" parent="s1" next="step2"/>
			<step id="step2" parent="s2"/>
		</flow>
		<flow>
			<step id="step3" parent="s3"/>
		</flow>
	</split>
	<step id="step4" parent="s4"/>
</job>
<beans:bean id="taskExecutor" class="org.spr...SimpleAsyncTaskExecutor"/>

重试机制

RetryOperations 接口实现类 RetryTemplate
重试的策略4中：

RetryPolicy ：SimpleRetryPolicy TimeoutRetryPolicy ExceptionClassifierRetryPolicy BackoffPolicy   

开启重试

<aop:config>
    <aop:pointcut id="transactional"
        expression="execution(* com..*Service.remoteCall(..))" />
    <aop:advisor pointcut-ref="transactional"
        advice-ref="retryAdvice" order="-1"/>
</aop:config>
<bean id="retryAdvice"
class="org.springframework.batch.retry.interceptor.RetryOperationsInterceptor"/>

测试

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "/simple-job-launcher-context.xml",
                                     "/jobs/skipSampleJob.xml" })
@TestExecutionListeners( { DependencyInjectionTestExecutionListener.class,
     StepScopeTestExecutionListener.class })
public class SkipSampleFunctionalTests { ... }

##一般batch模式

<step id="simpleStep">
    <tasklet>
        <chunk reader="reader" writer="writer" commit-interval="10"
               chunk-completion-policy="completionPolicy"/>
 
    </tasklet>
</step>

<bean id="completionPolicy" class="org.example...SpecialCompletionPolicy"/>

其他小技巧

Footer

可用于每个文件处理完成之后增加总结性的 例如记录数 行数等。

//写文件 Footer 
<bean id="itemWriter" class="org.spr...FlatFileItemWriter">
    <property name="resource" ref="outputResource" />
    <property name="lineAggregator" ref="lineAggregator"/>
    <property name="headerCallback" ref="headerCallback" />
    <property name="footerCallback" ref="footerCallback" />
</bean>

例如

public void writeFooter(Writer writer) throws IOException {
 
        writer.write("Total Amount Processed: " + totalAmount);
 
    }

执行cmd命令

<bean class="org.springframework.batch.core.step.tasklet.SystemCommandTasklet">
    <property name="command" value="echo hello" />
    <!-- 5 second timeout for the command to complete -->
    <property name="timeout" value="5000" />
</bean>

对JSR-352的支持

主要有两个功能 Partitioned 和 Batchlet

• Split - Running multiple steps in parallel. Multiple threads
• Executing a single step via multiple threads. Partitioning
• Dividing the data up for parallel processing (master/slave). Remote Chunking
• Executing the processor piece of logic remotely
  直接写代码可参考

JobOperator operator = BatchRuntime.getJobOperator();

jobOperator.start("myJob", new Properties());

文件流写入batch

<int:channel id="inboundFileChannel"/>
<int:channel id="outboundJobRequestChannel"/>
<int:channel id="jobLaunchReplyChannel"/>

<int-file:inbound-channel-adapter id="filePoller"
    channel="inboundFileChannel"
    directory="file:/tmp/myfiles/"
    filename-pattern="*.csv">
  <int:poller fixed-rate="1000"/>
</int-file:inbound-channel-adapter>

<int:transformer input-channel="inboundFileChannel"
    output-channel="outboundJobRequestChannel">
  <bean class="io.spring.sbi.FileMessageToJobRequest">
    <property name="job" ref="personJob"/>
    <property name="fileParameterName" value="input.file.name"/>
  </bean>
</int:transformer>

<batch-int:job-launching-gateway request-channel="outboundJobRequestChannel"
    reply-channel="jobLaunchReplyChannel"/>

<int:logging-channel-adapter channel="jobLaunchReplyChannel"/>

附1 使用Batch的注解

@BatchProperty and @Inject 两个一起才起作用
@EnableBatchConfiguration
@EnableBatchProcessing

附2 Batch内部注解

spring batch中主要由job step trunk组成，trunk可以看做是更细粒度的step，对于三大模块，都建立了监听器 JobExecutionListener、 StepExecutionListener 、ChunkListener。
其中 ChunkListener、StepExecutionListener 的父类是StepListener

附3 最后的彩蛋

因为用到了DB-DB的方式，看了下源码，目前还不支持直接配置多数据源 但是可以自己继承 AbstractBatchConfiguration 来实现自己的多数据源配置

	protected BatchConfigurer getConfigurer(Collection<BatchConfigurer> configurers) throws Exception {
		if (this.configurer != null) {
			return this.configurer;
		}
		if (configurers == null || configurers.isEmpty()) {
			if (dataSources == null || dataSources.isEmpty()) {
				DefaultBatchConfigurer configurer = new DefaultBatchConfigurer();
				configurer.initialize();
				this.configurer = configurer;
				return configurer;
			} else if(dataSources != null && dataSources.size() == 1) {
				DataSource dataSource = dataSources.iterator().next();
				DefaultBatchConfigurer configurer = new DefaultBatchConfigurer(dataSource);
				configurer.initialize();
				this.configurer = configurer;
				return configurer;
			} else {
				throw new IllegalStateException("To use the default BatchConfigurer the context must contain no more than" +
														"one DataSource, found " + dataSources.size());
			}
		}
		if (configurers.size() > 1) {
			throw new IllegalStateException(
					"To use a custom BatchConfigurer the context must contain precisely one, found "
							+ configurers.size());
		}
		this.configurer = configurers.iterator().next();
		return this.configurer;
	}

虽然定义的是 Collection 但未给出调用方式 并且

if (configurers.size() > 1)

这是要逗我吗？
给出重写的思路：重写的时候，把if (configurers.size() > 1)这一段去掉，再加上调用的配置prefix 应该就可以实现多数据源了。