Spring Batch示例– XML文件到MongoDB数据库

这篇教程展示了如何使用Spring Batch配置作业,从XML文件(借助XStream库)读取数据并将其导入到MongoDB数据库。教程涵盖了项目设置、依赖、XML文件解析、MongoDB连接、批处理核心配置、单元测试以及作业元数据的处理问题。
摘要由CSDN通过智能技术生成

在本教程中,我们将向您展示如何配置Spring Batch作业以将数据从XML文件( XStream库)读取到无SQL数据库( MongoDB )中。 另外,创建一个单元测试用例以启动和测试批处理作业。

使用的工具和库

  1. Maven 3
  2. Eclipse 4.2
  3. JDK 1.6
  4. Spring Core 3.2.2。发布
  5. Spring Batch 2.2.0。发布
  6. Spring Batch测试2.2.0.RELEASE
  7. Spring OXM 3.2.2。发布
  8. MongoDB Java驱动程序2.11.2
  9. MongoDB 2.2.3
  10. 的jUnit 4.11
  11. 测试NG 6.8.5

PS此示例– XML文件(读取器)– MongoDB(写入器)。

1.简单的Java项目

1.使用Maven创建一个快速入门Java Project,将其转换并导入Eclipse IDE。

$ mvn archetype:generate -DgroupId=com.mkyong -DartifactId=SpringBatchExample2 
  -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false
$ cd SpringBatchExample/
$ mvn eclipse:eclipse

2.项目依赖

pom.xml声明所有项目依赖项

pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" 
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
	http://maven.apache.org/maven-v4_0_0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.mkyong</groupId>
	<artifactId>SpringBatchExample</artifactId>
	<packaging>jar</packaging>
	<version>1.0-SNAPSHOT</version>
	<name>SpringBatchExample</name>
	<url>http://maven.apache.org</url>

	<properties>
		<jdk.version>1.6</jdk.version>
		<spring.version>3.2.2.RELEASE</spring.version>
		<spring.batch.version>2.2.0.RELEASE</spring.batch.version>
		<spring.data.version>1.2.1.RELEASE</spring.data.version>
		<mongodb.driver.version>2.11.2</mongodb.driver.version>
	</properties>

	<dependencies>

		<!-- Spring Core -->
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-core</artifactId>
			<version>${spring.version}</version>
		</dependency>

		<!-- Spring XML to/back object -->
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-oxm</artifactId>
			<version>${spring.version}</version>
		</dependency>

		<!-- Spring Batch dependencies -->
		<dependency>
			<groupId>org.springframework.batch</groupId>
			<artifactId>spring-batch-core</artifactId>
			<version>${spring.batch.version}</version>
		</dependency>
		<dependency>
			<groupId>org.springframework.batch</groupId>
			<artifactId>spring-batch-infrastructure</artifactId>
			<version>${spring.batch.version}</version>
		</dependency>

		<!-- Spring Batch unit test -->
		<dependency>
			<groupId>org.springframework.batch</groupId>
			<artifactId>spring-batch-test</artifactId>
			<version>${spring.batch.version}</version>
		</dependency>

		<!-- MongoDB database driver -->
		<dependency>
			<groupId>org.mongodb</groupId>
			<artifactId>mongo-java-driver</artifactId>
			<version>${mongodb.driver.version}</version>
		</dependency>

		<!-- Spring data mongodb -->
		<dependency>
			<groupId>org.springframework.data</groupId>
			<artifactId>spring-data-mongodb</artifactId>
			<version>${spring.data.version}</version>
		</dependency>

		<!-- Junit -->
		<dependency>
			<groupId>junit</groupId>
			<artifactId>junit</artifactId>
			<version>4.11</version>
			<scope>test</scope>
		</dependency>

		<!-- Testng -->
		<dependency>
			<groupId>org.testng</groupId>
			<artifactId>testng</artifactId>
			<version>6.8.5</version>
			<scope>test</scope>
		</dependency>

	</dependencies>
	<build>
	    <finalName>spring-batch</finalName>
	    <plugins>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-eclipse-plugin</artifactId>
			<version>2.9</version>
			<configuration>
				<downloadSources>true</downloadSources>
				<downloadJavadocs>false</downloadJavadocs>
			</configuration>
		</plugin>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-compiler-plugin</artifactId>
			<version>2.3.2</version>
			<configuration>
				<source>${jdk.version}</source>
				<target>${jdk.version}</target>
			</configuration>
		</plugin>
	    </plugins>
	</build>

</project>

3.项目目录结构

查看最终的项目结构,获得下一步的概览。

Spring批处理-xml到mongodb

4. XML文件

这是资源文件夹中的XML文件。

resources/xml/report.xml
<?xml version="1.0" encoding="UTF-8" ?>
<report>
    <record id="1">
        <date>6/1/2013</date>
        <impression>139,237</impression>
        <clicks>40</clicks>
        <earning>220.90</earning>
    </record>
    <record id="2">
        <date>6/2/2013</date>
        <impression>339,100</impression>
        <clicks>60</clicks>
        <earning>320.88</earning>
    </record>
    <record id="3">
        <date>6/3/2013</date>
        <impression>431,436</impression>
        <clicks>76</clicks>
        <earning>270.80</earning>
    </record>
</report>

5.读取XML文件

在Spring批处理中,我们可以使用StaxEventItemReader读取XML文件,并使用XStreamMarshaller将XML值和属性映射到对象。

resources/spring/batch/jobs/job-report.xml
<!-- ...... -->
    <bean id="xmlItemReader" 
           class="org.springframework.batch.item.xml.StaxEventItemReader">
	<property name="fragmentRootElementName" value="record" />
	<property name="resource" value="classpath:xml/report.xml" />
	<property name="unmarshaller" ref="reportUnmarshaller" />
    </bean>

    <bean id="reportUnmarshaller" 
           class="org.springframework.oxm.xstream.XStreamMarshaller">

	<property name="aliases">
	    <util:map id="aliases">
		<entry key="record" value="com.mkyong.model.Report" />
	    </util:map>
	</property>
	<property name="converters">
	    <array>
		<ref bean="reportConverter" />
	    </array>
	</property>
				
    </bean>
	
    <bean id="reportConverter" class="com.mkyong.converter.ReportConverter" />
Report.java
package com.mkyong.model;

import java.math.BigDecimal;
import java.text.SimpleDateFormat;
import java.util.Date;

public class Report {

	private int id;
	private Date date;
	private long impression;
	private int clicks;
	private BigDecimal earning;

	//getter and setter methods

}

要将XML值映射到DateBigDecimal类的“复杂”数据类型,您需要附加一个自定义converter来手动转换和映射该值。

ReportConverter.java
package com.mkyong.converter;

import java.math.BigDecimal;
import java.text.NumberFormat;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Locale;

import com.mkyong.model.Report;
import com.thoughtworks.xstream.converters.Converter;
import com.thoughtworks.xstream.converters.MarshallingContext;
import com.thoughtworks.xstream.converters.UnmarshallingContext;
import com.thoughtworks.xstream.io.HierarchicalStreamReader;
import com.thoughtworks.xstream.io.HierarchicalStreamWriter;

public class ReportConverter implements Converter {

	@Override
	public boolean canConvert(Class type) {
		//we only need "Report" object
		return type.equals(Report.class);
	}

	@Override
	public void marshal(Object source, 
            HierarchicalStreamWriter writer, MarshallingContext context) {
	    //do nothing
	}

	@Override
	public Object unmarshal(
            HierarchicalStreamReader reader, UnmarshallingContext context) {
		
		Report obj = new Report();
		
		//get attribute
		obj.setId(Integer.valueOf(reader.getAttribute("id")));
		reader.moveDown(); //get date
		
		Date date = null;
		try {
			date = new SimpleDateFormat("M/d/yyyy").parse(reader.getValue());
		} catch (ParseException e) {
			e.printStackTrace();
		}
		obj.setDate(date);
		reader.moveUp();
		
		reader.moveDown(); //get impression
		
		String impression = reader.getValue();
		NumberFormat format = NumberFormat.getInstance(Locale.US);
                Number number = 0;
		try {
			number = format.parse(impression);
		} catch (ParseException e) {
			e.printStackTrace();
		}
		obj.setImpression(number.longValue());
		
		reader.moveUp();
		
		reader.moveDown(); //get click
		obj.setClicks(Integer.valueOf(reader.getValue()));
		reader.moveUp();
		
		reader.moveDown(); //get earning
		obj.setEarning(new BigDecimal(reader.getValue()));
                reader.moveUp();
        
                return obj;
        
	}
}

6. MongoDB数据库

定义一个mongodb实例,以及一个mongoTemplate

resources/spring/batch/config/database.xml
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:mongo="http://www.springframework.org/schema/data/mongo" 
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://www.springframework.org/schema/beans 
		http://www.springframework.org/schema/beans/spring-beans-3.2.xsd
		http://www.springframework.org/schema/data/mongo
        http://www.springframework.org/schema/data/mongo/spring-mongo-1.0.xsd">

        <!-- connect to mongodb -->
	<mongo:mongo host="127.0.0.1" port="27017" />
	<mongo:db-factory dbname="yourdb" />
 
	<bean id="mongoTemplate" 
                class="org.springframework.data.mongodb.core.MongoTemplate">
		<constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
	</bean>
	  
</beans>

7. Spring Batch核心设置

定义jobRepositoryjobLauncher

resources/spring/batch/config/context.xml
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="
	http://www.springframework.org/schema/beans 
	http://www.springframework.org/schema/beans/spring-beans-3.2.xsd">

    <!-- stored job-meta in memory --> 
    <bean id="jobRepository"
	class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">
	<property name="transactionManager" ref="transactionManager" />
    </bean>
 	
    <bean id="transactionManager"
	class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
	
 
    <bean id="jobLauncher"
	class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
	<property name="jobRepository" ref="jobRepository" />
    </bean>

</beans>

8.Spring批处理作业

Spring批处理作业,读取report.xml文件,映射到Report对象,并将其写入MongoDB 。 阅读评论,它应该是不言自明的。

resources/spring/batch/jobs/job-report.xml
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:batch="http://www.springframework.org/schema/batch" 
	xmlns:util="http://www.springframework.org/schema/util" 
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://www.springframework.org/schema/batch
		http://www.springframework.org/schema/batch/spring-batch-2.2.xsd
		http://www.springframework.org/schema/beans 
		http://www.springframework.org/schema/beans/spring-beans-3.2.xsd
		http://www.springframework.org/schema/util 
		http://www.springframework.org/schema/util/spring-util-3.2.xsd">

    <batch:job id="reportJob">
	<batch:step id="step1">
	  <batch:tasklet>
		<batch:chunk reader="xmlItemReader" writer="mongodbItemWriter"
			commit-interval="1">
		</batch:chunk>
	  </batch:tasklet>
	</batch:step>
    </batch:job>

    <!-- Read XML file -->
    <bean id="xmlItemReader" 
        class="org.springframework.batch.item.xml.StaxEventItemReader">
	<property name="fragmentRootElementName" value="record" />
	<property name="resource" value="classpath:xml/report.xml" />
	<property name="unmarshaller" ref="reportUnmarshaller" />
    </bean>

    <!-- Maps XML values to Object -->
    <bean id="reportUnmarshaller" 
        class="org.springframework.oxm.xstream.XStreamMarshaller">
	<property name="aliases">
	  <util:map id="aliases">
		<entry key="record" value="com.mkyong.model.Report" />
	  </util:map>
	</property>

        <!-- attach a custom converter -->
	<property name="converters">
	  <array>
		<ref bean="reportConverter" />
	  </array>
	</property>

    </bean>

    <bean id="reportConverter" class="com.mkyong.converter.ReportConverter" />

    //write it to MongoDB, 'report' collection (table)
    <bean id="mongodbItemWriter" 
        class="org.springframework.batch.item.data.MongoItemWriter">
	<property name="template" ref="mongoTemplate" />
	<property name="collection" value="report" />
    </bean>
	
</beans>

9.单元测试

使用jUnit或TestNG框架对其进行单元测试。首先,必须手动声明JobLauncherTestUtils

test/resources/spring/batch/config/test-context.xml
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="
	http://www.springframework.org/schema/beans 
	http://www.springframework.org/schema/beans/spring-beans-3.2.xsd">
 
    <!-- this bean should auto load -->
    <bean class="org.springframework.batch.test.JobLauncherTestUtils"/>
    
</beans>

jUnit示例

AppTest.java
package com.mkyong;

import static org.junit.Assert.assertEquals;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.batch.core.BatchStatus;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.test.JobLauncherTestUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {
    "classpath:spring/batch/jobs/job-report.xml",
    "classpath:spring/batch/config/context.xml",
    "classpath:spring/batch/config/database.xml",
    "classpath:spring/batch/config/test-context.xml"})
public class AppTest {

    @Autowired
    private JobLauncherTestUtils jobLauncherTestUtils;

    @Test
    public void launchJob() throws Exception {

        //JobExecution jobExecution = jobLauncherTestUtils.launchJob();
        
        JobExecution jobExecution = jobLauncherTestUtils.launchStep("step1");
        
        assertEquals(BatchStatus.COMPLETED, jobExecution.getStatus());
        
    }
}

TestNG示例

AppTest2.java
package com.mkyong;

import org.springframework.batch.core.BatchStatus;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.test.JobLauncherTestUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.testng.AbstractTestNGSpringContextTests;
import org.testng.Assert;
import org.testng.annotations.Test;

@ContextConfiguration(locations = {
    "classpath:spring/batch/jobs/job-report.xml",
    "classpath:spring/batch/config/context.xml",
    "classpath:spring/batch/config/database.xml",
    "classpath:spring/batch/config/test-context.xml"})
public class AppTest2 extends AbstractTestNGSpringContextTests {

    @Autowired
    private JobLauncherTestUtils jobLauncherTestUtils;

    @Test
    public void launchJob() throws Exception {

        JobExecution jobExecution = jobLauncherTestUtils.launchJob();
        Assert.assertEquals(jobExecution.getStatus(), BatchStatus.COMPLETED);
        
    }
}

输出。 XML值将插入到MongoDB中。

mongo
MongoDB shell version: 2.2.3
connecting to: test

> use yourdb
switched to db yourdb
> show collections
report
system.indexes

> db.report.find()
{ "_id" : 1, "_class" : "com.mkyong.model.Report", 
"date" : ISODate("2013-05-31T16:00:00Z"), "impression" : NumberLong(139237), 
"clicks" : 40, "earning" : "220.90" }

{ "_id" : 2, "_class" : "com.mkyong.model.Report", 
"date" : ISODate("2013-06-01T16:00:00Z"), "impression" : NumberLong(339100), 
"clicks" : 60, "earning" : "320.88" }

{ "_id" : 3, "_class" : "com.mkyong.model.Report", 
"date" : ISODate("2013-06-02T16:00:00Z"), "impression" : NumberLong(431436), 
"clicks" : 76, "earning" : "270.80" }
>

10.作业元数据如何?

抱歉,我还没有解决方案。 据我所知,工作元数据需要关系数据库,以确保工作的可重启性和回滚。 根据设计,MongoDB没有“可靠”的交易管理。

解决方案1:创建另一个关系数据库来存储作业元数据,嗯……听起来很愚蠢,但可以。 您有更好的主意吗?

解决方案2:等待Spring的团队为此提出解决方案。

下载源代码

下载它– SpringBatch-XML-MongoDB-Example.zip (81 kb)

参考文献

  1. XStream转换器教程
  2. 非RDBMS的Spring Batch支持
  3. Spring Batch XML项目读取器和写入器
  4. Spring Batch –配置和运行作业
  5. 使用Maven创建Java项目

翻译自: https://mkyong.com/spring-batch/spring-batch-example-xml-file-to-database/

  • 0
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Spring Batch是一个用于大规模批量处理的开源框架,可以帮助我们简化和规范化批量作业的开发。在Spring Batch中获取多个文件的示例可以通过以下步骤来完成: 首先,我们需要配置Spring Batch的Job和Step。在Job配置中,我们可以定义多个Step,每个Step可以处理一个文件。在Step配置中,我们可以定义ItemReader、ItemProcessor和ItemWriter等组件,来完成文件读取、数据处理和结果写入等功能。 其次,我们需要编写ItemReader的实现类来读取文件。在多文件的情况下,我们可以通过Loop资源配置来遍历不同的文件。例如,可以使用MultiResourceItemReader来设置多个文件路径,并通过设置同一个Delegate对象来处理每个文件的读取。 然后,我们可以编写ItemProcessor的实现类来处理读取到的数据。在多文件的情况下,我们可以使用文件名称作为数据的一部分,以便在后续的处理中对数据进行分类或区分。 最后,我们可以编写ItemWriter的实现类来将处理后的结果写入到目标文件中。在多文件的情况下,我们可以使用不同的文件名称来区分不同的结果文件。 综上所述,通过以上配置和编码,我们可以实现Spring Batch获取多个文件的示例。通过配置Job和Step,使用MultiResourceItemReader处理多个文件的读取,使用ItemProcessor处理数据,最后使用ItemWriter将结果写入到相应的文件中。这样,我们就可以实现在Spring Batch中处理多个文件的任务。
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值