在LT eip里,springbatch是常用到的批处理框架。小批量简单数据结构可以直接用其加上rowmap做批量同步,大数据量复杂数据结构转换同步可以用batch+mq(分发到多个服务处理)+smooks。
以下是以erp600中海关3个基础资料(成品电子账册、料件电子账册、账册备案信息)批量同步到k3cloud系统为例
1、创建batch的job(任务)配置文件,参考RESTEipClient\src\main\resources\META-INF\spring\batch\customs.xml(因为erp600是LT自己开发的erp系统,命名规范是按拼音首字母,虽然不是正规的命名方式,但在此也如此处理,方便后期开发维护人员对照)
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:batch="http://www.springframework.org/schema/batch" xsi:schemaLocation=" http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd"> <!-- JOB --> <!-- 海关3个基础资料的任务,按顺序执行 --> <batch:job id="customsJob"> <!--成品电子账册 --> <batch:step id="cpdzzc" next="ljdzzc"> <batch:tasklet> <batch:chunk reader="CpdzzcReader" writer="CpdzzcTempWriter" commit-interval="3000" /> <!-- 3000条一次commit --> <batch:listeners> <batch:listener ref="cpdzzcListener" /> </batch:listeners> </batch:tasklet> </batch:step> <!--料件电子账册 --> <batch:step id="ljdzzc" next="zcbaxx"> <batch:tasklet> <batch:chunk reader="LjdzzcReader" writer="ljdzzcTempWriter" commit-interval="3000" /> <!-- 3000条一次commit --> <batch:listeners> <batch:listener ref="ljdzzcListener" /> </batch:listeners> </batch:tasklet> </batch:step> <!--账册 备案信息--> <batch:step id="zcbaxx"> <batch:tasklet> <batch:chunk reader="ZcbaxxReader" writer="zcbaxxTempWriter" commit-interval="3000" /> <!-- 3000条一次commit --> <batch:listeners> <batch:listener ref="zcbaxxListener" /> </batch:listeners> </batch:tasklet> </batch:step> </batch:job> <!--ERP600 READER --> <!--erp600_xsgl.dbo.xsgl_cpdzzc 成品电子账册 --> <bean id="CpdzzcReader" class="org.springframework.batch.item.database.JdbcCursorItemReader"> <property name="dataSource" ref="erp600xsglDataSource" /> <property name="rowMapper" ref="cpdzzcTempRowMapper" /> <property name="sql" value="select * from erp600_xsgl.dbo.xsgl_cpdzzc " /> </bean> <!--erp600_xsgl.dbo.xsgl_ljdzzc 料件电子账册 --> <bean id="LjdzzcReader" class="org.springframework.batch.item.database.JdbcCursorItemReader"> <property name="dataSource" ref="erp600xsglDataSource" /> <property name="rowMapper" ref="ljdzzcTempRowMapper" /> <property name="sql" value="select * from erp600_xsgl.dbo.xsgl_ljdzzc " /> </bean> <!--erp600_xsgl.dbo.xsgl_zcbaxx 账册备案 --> <bean id="ZcbaxxReader" class="org.springframework.batch.item.database.JdbcCursorItemReader"> <property name="dataSource" ref="erp600xsglDataSource" /> <property name="rowMapper" ref="zcbaxxTempRowMapper" /> <property name="sql" value="select * from erp600_xsgl.dbo.xsgl_zcbaxx " /> </bean> </beans>
以cpdzzc(成品电子账册)为例
1 <!--成品电子账册 --> 2 <batch:step id="cpdzzc" next="ljdzzc"> <!--执行完step"cpdzzc"后再执行step"ljdzzc"--> 3 <batch:tasklet> 4 <batch:chunk reader="CpdzzcReader" writer="CpdzzcTempWriter" commit-interval="3000" /> 5 <!--批处理的读取执行者为"CpdzzcReader",写入者为"CpdzzcTempWriter",3000条做一次commit --> 6 <batch:listeners> 7 <!--step"cpdzzc"执行过程中的监听者,可以做事前事后的处理--> 8 <batch:listener ref="cpdzzcListener" /> 9 </batch:listeners> 10 </batch:tasklet> 11 </batch:step> 12 13 …… 14 15 <!--erp600_xsgl.dbo.xsgl_cpdzzc 成品电子账册 --> 16 <!--Spring Batch对DB基于游标的读取数据操作,是由其核心类JdbcCursorItemReader来实现的,如有复杂的sql,可以自己继承 17 JdbcCursorItemReader,重写setSql方法;与读取数据对应的,Spring Batch有JdbcBatchItemWriter类来执行写数据 18 --> 19 <bean id="CpdzzcReader" 20 class="org.springframework.batch.item.database.JdbcCursorItemReader"> 21 <property name="dataSource" ref="erp600xsglDataSource" /> 22 <property name="rowMapper" ref="cpdzzcTempRowMapper" /> 23 <property name="sql" value="select * from erp600_xsgl.dbo.xsgl_cpdzzc " /> 24 </bean>
读取数据结果映射类
@Component("cpdzzcTempRowMapper") public class CpdzzcTempRowMapper implements RowMapper{ public Object mapRow(ResultSet rs, int rowNum) throws SQLException { // TODO Auto-generated method stub CpdzzcTemp temp=new CpdzzcTemp(); temp.setSrcid(rs.getInt("id")); temp.setFhandbookid(processString(rs.getString("scxh"))); temp.setFcancel(processString(rs.getString("zfr"))); temp.setFcanceldate(rs.getDate("zfrq")); temp.setFhsid(processString(rs.getString("hsspbm"))); temp.setFproductname(processString(rs.getString("cppm"))); temp.setFtype(processString(rs.getString("ggxh"))); temp.setFunit(processString(rs.getString("zcdw"))); temp.setFremark(processString(rs.getString("bz"))); temp.setFcreator(processString(rs.getString("bzr"))); temp.setFcreatedate(rs.getDate("bzrq")); temp.setFmodifydate(rs.getDate("zhxgrq")); temp.setFproducttype(processString(rs.getString("cplx"))); temp.setFtesttype(processString(rs.getString("jylx"))); return temp; } public String processString( String value) { value=null!=value&&value.length()>0?value:" "; return value; } }
cpDzzcListener需要实现org.springframework.batch.core.StepExecutionListener接口,接口有2个方法
public interface StepExecutionListener extends StepListener { //执行前的处理 void beforeStep(StepExecution stepExecution); //执行后的处理 ExitStatus afterStep(StepExecution stepExecution); }
写入执行类
@Component("CpdzzcTempWriter") public class CpdzzcTempWriter implements ItemWriter<CpdzzcTemp> { @Autowired private OrclDaoImpl dao; public void write(List<? extends CpdzzcTemp> items) throws Exception { for(CpdzzcTemp item:items){ dao.saveCpdzzc(item); } } }
如果想程序调用,则如下,其中行7的原因是Spring Batch Job同一个job instance,成功执行后是不允许重新执行的【失败后是否允许重跑,可通过配置Job的restartable参数来控制,默认是true】,如果需要重新执行,可以变通处理,
添加一个JobParameters构建类,以当前时间作为参数,保证其他参数相同的情况下却是不同的job instance
1 public String trunsCustoms(){ 2 3 try { 4 JobLauncher jobLauncher=(JobLauncher)SpringContextUtil.getBean("jobLauncher"); 5 Job customsJob=(Job)SpringContextUtil.getBean("customsJob"); 6 Map<String,JobParameter> parameters = new HashMap<String,JobParameter>(); 7 parameters.put(UUID.randomUUID().toString(), new JobParameter(System.currentTimeMillis())); 8 jobLauncher.run(customsJob, new JobParameters(parameters)); 9 return "true"; 10 11 } catch (JobExecutionAlreadyRunningException e) { 12 e.printStackTrace(); 13 } catch (JobRestartException e) { 14 e.printStackTrace(); 15 } catch (JobInstanceAlreadyCompleteException e) { 16 e.printStackTrace(); 17 } catch (JobParametersInvalidException e) { 18 e.printStackTrace(); 19 } 20 return "false"; 21 }
<!-- JOB REPOSITORY - WE USE IN-MEMORY REPOSITORY FOR OUR EXAMPLE --> <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"> <property name="transactionManager" ref="transactionManager" /> </bean> <!-- batch config --> <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher"> <property name="jobRepository" ref="jobRepository" /> </bean> <!-- Spring Batch Job同一个job instance,成功执行后是不允许重新执行的【失败后是否允许重跑,可通过配置Job的restartable参数来控制,默认是true】,如果需要重新执行,可以变通处理, 添加一个JobParameters构建类,以当前时间作为参数,保证其他参数相同的情况下却是不同的job instance --> <bean id="jobParameterBulider" class="org.springframework.batch.core.JobParametersBuilder" />
如果想用quartz定时调用,则如下
1、定义一个QuartzJob
1 @Component("quartzCustomsJob") 2 public class QuartzSaleLogisJob { 3 4 @Autowired 5 private JobLauncher launcher; 6 7 @Autowired 8 private Job customsJob; 9 10 @Autowired 11 JobParametersBuilder builder; 12 13 public void execute() throws Exception{ 14 15 builder.addDate("date", new Date()); 16 launcher.run(customsJob, builder.toJobParameters()); 17 } 18 19 }
配置文件增加quartz配置信息
<bean id="customsJobDetail" class="org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean"> <property name="targetObject"> <!-- 定时执行的类 --> <ref bean="quartzCustomsJob" /> </property> <property name="targetMethod"> <!-- 定时执行的类方法 --> <value>execute</value> </property> </bean> <bean id="customsCronTrigger" class="org.springframework.scheduling.quartz.CronTriggerBean" > <!-- 这里不可以直接在属性jobDetail中引用taskJob,因为他要求的是一个jobDetail类型的对象,所以我们得通过MethodInvokingJobDetailFactoryBean来转一下 --> <property name="jobDetail" > <ref bean="customsJobDetail" /> </property> <property name="cronExpression" > <value>0 0/30 7-19 * * ?</value> <!-- 每天从7点至19点 每30分钟--> </property> </bean> <!-- 触发器工厂,将所有的定时任务都注入工厂--> <bean name="startQuartz" class="org.springframework.scheduling.quartz.SchedulerFactoryBean"> <property name="triggers"> <list> <ref local="customsCronTrigger" /> </list> </property> </bean>