在本教程中,我们将向您展示如何配置Spring Batch作业以将数据从XML文件( XStream库)读取到无SQL数据库( MongoDB )中。 另外,创建一个单元测试用例以启动和测试批处理作业。
使用的工具和库
- Maven 3
- Eclipse 4.2
- JDK 1.6
- Spring Core 3.2.2。发布
- Spring Batch 2.2.0。发布
- Spring Batch测试2.2.0.RELEASE
- Spring OXM 3.2.2。发布
- MongoDB Java驱动程序2.11.2
- MongoDB 2.2.3
- 的jUnit 4.11
- 测试NG 6.8.5
PS此示例– XML文件(读取器)– MongoDB(写入器)。
1.简单的Java项目
1.使用Maven创建一个快速入门Java Project,将其转换并导入Eclipse IDE。
$ mvn archetype:generate -DgroupId=com.mkyong -DartifactId=SpringBatchExample2
-DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false
$ cd SpringBatchExample/
$ mvn eclipse:eclipse
2.项目依赖
在pom.xml
声明所有项目依赖项
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mkyong</groupId>
<artifactId>SpringBatchExample</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<name>SpringBatchExample</name>
<url>http://maven.apache.org</url>
<properties>
<jdk.version>1.6</jdk.version>
<spring.version>3.2.2.RELEASE</spring.version>
<spring.batch.version>2.2.0.RELEASE</spring.batch.version>
<spring.data.version>1.2.1.RELEASE</spring.data.version>
<mongodb.driver.version>2.11.2</mongodb.driver.version>
</properties>
<dependencies>
<!-- Spring Core -->
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>${spring.version}</version>
</dependency>
<!-- Spring XML to/back object -->
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-oxm</artifactId>
<version>${spring.version}</version>
</dependency>
<!-- Spring Batch dependencies -->
<dependency>
<groupId>org.springframework.batch</groupId>
<artifactId>spring-batch-core</artifactId>
<version>${spring.batch.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.batch</groupId>
<artifactId>spring-batch-infrastructure</artifactId>
<version>${spring.batch.version}</version>
</dependency>
<!-- Spring Batch unit test -->
<dependency>
<groupId>org.springframework.batch</groupId>
<artifactId>spring-batch-test</artifactId>
<version>${spring.batch.version}</version>
</dependency>
<!-- MongoDB database driver -->
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>${mongodb.driver.version}</version>
</dependency>
<!-- Spring data mongodb -->
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-mongodb</artifactId>
<version>${spring.data.version}</version>
</dependency>
<!-- Junit -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<!-- Testng -->
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.8.5</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<finalName>spring-batch</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-eclipse-plugin</artifactId>
<version>2.9</version>
<configuration>
<downloadSources>true</downloadSources>
<downloadJavadocs>false</downloadJavadocs>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>${jdk.version}</source>
<target>${jdk.version}</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
3.项目目录结构
查看最终的项目结构,获得下一步的概览。
4. XML文件
这是资源文件夹中的XML文件。
<?xml version="1.0" encoding="UTF-8" ?>
<report>
<record id="1">
<date>6/1/2013</date>
<impression>139,237</impression>
<clicks>40</clicks>
<earning>220.90</earning>
</record>
<record id="2">
<date>6/2/2013</date>
<impression>339,100</impression>
<clicks>60</clicks>
<earning>320.88</earning>
</record>
<record id="3">
<date>6/3/2013</date>
<impression>431,436</impression>
<clicks>76</clicks>
<earning>270.80</earning>
</record>
</report>
5.读取XML文件
在Spring批处理中,我们可以使用StaxEventItemReader
读取XML文件,并使用XStreamMarshaller
将XML值和属性映射到对象。
<!-- ...... -->
<bean id="xmlItemReader"
class="org.springframework.batch.item.xml.StaxEventItemReader">
<property name="fragmentRootElementName" value="record" />
<property name="resource" value="classpath:xml/report.xml" />
<property name="unmarshaller" ref="reportUnmarshaller" />
</bean>
<bean id="reportUnmarshaller"
class="org.springframework.oxm.xstream.XStreamMarshaller">
<property name="aliases">
<util:map id="aliases">
<entry key="record" value="com.mkyong.model.Report" />
</util:map>
</property>
<property name="converters">
<array>
<ref bean="reportConverter" />
</array>
</property>
</bean>
<bean id="reportConverter" class="com.mkyong.converter.ReportConverter" />
package com.mkyong.model;
import java.math.BigDecimal;
import java.text.SimpleDateFormat;
import java.util.Date;
public class Report {
private int id;
private Date date;
private long impression;
private int clicks;
private BigDecimal earning;
//getter and setter methods
}
要将XML值映射到Date
和BigDecimal
类的“复杂”数据类型,您需要附加一个自定义converter
来手动转换和映射该值。
package com.mkyong.converter;
import java.math.BigDecimal;
import java.text.NumberFormat;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Locale;
import com.mkyong.model.Report;
import com.thoughtworks.xstream.converters.Converter;
import com.thoughtworks.xstream.converters.MarshallingContext;
import com.thoughtworks.xstream.converters.UnmarshallingContext;
import com.thoughtworks.xstream.io.HierarchicalStreamReader;
import com.thoughtworks.xstream.io.HierarchicalStreamWriter;
public class ReportConverter implements Converter {
@Override
public boolean canConvert(Class type) {
//we only need "Report" object
return type.equals(Report.class);
}
@Override
public void marshal(Object source,
HierarchicalStreamWriter writer, MarshallingContext context) {
//do nothing
}
@Override
public Object unmarshal(
HierarchicalStreamReader reader, UnmarshallingContext context) {
Report obj = new Report();
//get attribute
obj.setId(Integer.valueOf(reader.getAttribute("id")));
reader.moveDown(); //get date
Date date = null;
try {
date = new SimpleDateFormat("M/d/yyyy").parse(reader.getValue());
} catch (ParseException e) {
e.printStackTrace();
}
obj.setDate(date);
reader.moveUp();
reader.moveDown(); //get impression
String impression = reader.getValue();
NumberFormat format = NumberFormat.getInstance(Locale.US);
Number number = 0;
try {
number = format.parse(impression);
} catch (ParseException e) {
e.printStackTrace();
}
obj.setImpression(number.longValue());
reader.moveUp();
reader.moveDown(); //get click
obj.setClicks(Integer.valueOf(reader.getValue()));
reader.moveUp();
reader.moveDown(); //get earning
obj.setEarning(new BigDecimal(reader.getValue()));
reader.moveUp();
return obj;
}
}
6. MongoDB数据库
定义一个mongodb实例,以及一个mongoTemplate
。
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:mongo="http://www.springframework.org/schema/data/mongo"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.2.xsd
http://www.springframework.org/schema/data/mongo
http://www.springframework.org/schema/data/mongo/spring-mongo-1.0.xsd">
<!-- connect to mongodb -->
<mongo:mongo host="127.0.0.1" port="27017" />
<mongo:db-factory dbname="yourdb" />
<bean id="mongoTemplate"
class="org.springframework.data.mongodb.core.MongoTemplate">
<constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
</bean>
</beans>
7. Spring Batch核心设置
定义jobRepository
和jobLauncher
。
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.2.xsd">
<!-- stored job-meta in memory -->
<bean id="jobRepository"
class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">
<property name="transactionManager" ref="transactionManager" />
</bean>
<bean id="transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
<bean id="jobLauncher"
class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
</beans>
8.Spring批处理作业
Spring批处理作业,读取report.xml
文件,映射到Report
对象,并将其写入MongoDB
。 阅读评论,它应该是不言自明的。
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:util="http://www.springframework.org/schema/util"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/batch
http://www.springframework.org/schema/batch/spring-batch-2.2.xsd
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.2.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util-3.2.xsd">
<batch:job id="reportJob">
<batch:step id="step1">
<batch:tasklet>
<batch:chunk reader="xmlItemReader" writer="mongodbItemWriter"
commit-interval="1">
</batch:chunk>
</batch:tasklet>
</batch:step>
</batch:job>
<!-- Read XML file -->
<bean id="xmlItemReader"
class="org.springframework.batch.item.xml.StaxEventItemReader">
<property name="fragmentRootElementName" value="record" />
<property name="resource" value="classpath:xml/report.xml" />
<property name="unmarshaller" ref="reportUnmarshaller" />
</bean>
<!-- Maps XML values to Object -->
<bean id="reportUnmarshaller"
class="org.springframework.oxm.xstream.XStreamMarshaller">
<property name="aliases">
<util:map id="aliases">
<entry key="record" value="com.mkyong.model.Report" />
</util:map>
</property>
<!-- attach a custom converter -->
<property name="converters">
<array>
<ref bean="reportConverter" />
</array>
</property>
</bean>
<bean id="reportConverter" class="com.mkyong.converter.ReportConverter" />
//write it to MongoDB, 'report' collection (table)
<bean id="mongodbItemWriter"
class="org.springframework.batch.item.data.MongoItemWriter">
<property name="template" ref="mongoTemplate" />
<property name="collection" value="report" />
</bean>
</beans>
9.单元测试
使用jUnit或TestNG框架对其进行单元测试。首先,必须手动声明JobLauncherTestUtils
。
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.2.xsd">
<!-- this bean should auto load -->
<bean class="org.springframework.batch.test.JobLauncherTestUtils"/>
</beans>
jUnit示例
package com.mkyong;
import static org.junit.Assert.assertEquals;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.batch.core.BatchStatus;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.test.JobLauncherTestUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {
"classpath:spring/batch/jobs/job-report.xml",
"classpath:spring/batch/config/context.xml",
"classpath:spring/batch/config/database.xml",
"classpath:spring/batch/config/test-context.xml"})
public class AppTest {
@Autowired
private JobLauncherTestUtils jobLauncherTestUtils;
@Test
public void launchJob() throws Exception {
//JobExecution jobExecution = jobLauncherTestUtils.launchJob();
JobExecution jobExecution = jobLauncherTestUtils.launchStep("step1");
assertEquals(BatchStatus.COMPLETED, jobExecution.getStatus());
}
}
TestNG示例
package com.mkyong;
import org.springframework.batch.core.BatchStatus;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.test.JobLauncherTestUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.testng.AbstractTestNGSpringContextTests;
import org.testng.Assert;
import org.testng.annotations.Test;
@ContextConfiguration(locations = {
"classpath:spring/batch/jobs/job-report.xml",
"classpath:spring/batch/config/context.xml",
"classpath:spring/batch/config/database.xml",
"classpath:spring/batch/config/test-context.xml"})
public class AppTest2 extends AbstractTestNGSpringContextTests {
@Autowired
private JobLauncherTestUtils jobLauncherTestUtils;
@Test
public void launchJob() throws Exception {
JobExecution jobExecution = jobLauncherTestUtils.launchJob();
Assert.assertEquals(jobExecution.getStatus(), BatchStatus.COMPLETED);
}
}
输出。 XML值将插入到MongoDB中。
mongo
MongoDB shell version: 2.2.3
connecting to: test
> use yourdb
switched to db yourdb
> show collections
report
system.indexes
> db.report.find()
{ "_id" : 1, "_class" : "com.mkyong.model.Report",
"date" : ISODate("2013-05-31T16:00:00Z"), "impression" : NumberLong(139237),
"clicks" : 40, "earning" : "220.90" }
{ "_id" : 2, "_class" : "com.mkyong.model.Report",
"date" : ISODate("2013-06-01T16:00:00Z"), "impression" : NumberLong(339100),
"clicks" : 60, "earning" : "320.88" }
{ "_id" : 3, "_class" : "com.mkyong.model.Report",
"date" : ISODate("2013-06-02T16:00:00Z"), "impression" : NumberLong(431436),
"clicks" : 76, "earning" : "270.80" }
>
10.作业元数据如何?
抱歉,我还没有解决方案。 据我所知,工作元数据需要关系数据库,以确保工作的可重启性和回滚。 根据设计,MongoDB没有“可靠”的交易管理。
解决方案1:创建另一个关系数据库来存储作业元数据,嗯……听起来很愚蠢,但可以。 您有更好的主意吗?
解决方案2:等待Spring的团队为此提出解决方案。
下载源代码
下载它– SpringBatch-XML-MongoDB-Example.zip (81 kb)
参考文献
翻译自: https://mkyong.com/spring-batch/spring-batch-example-xml-file-to-database/