Java实时处理 - Spring Integration - MQ Message

目录

Introduction

Spring Integration Components

Message

Channel

EndPoint

Channel Adapter

Messaging Gateway 

Service Activator

Router 

 Splitter

 Aggregator

Messaging

Integrating systems

Get Started with Spring Integration

公司的案例

其他资源

---

1. 官方文档： (1) https://spring.io/projects/spring-integration

               （有PDF：https://docs.spring.io/spring-integration/docs/current/reference/pdf/spring-integration-reference.pdf)

                 (2) Get Started with Spring Integration: https://spring.io/guides/gs/integration/

2. Github: https://github.com/spring-projects/spring-integration

3. 书籍推荐：

《Spring实战》(即Spring in action 第5版/第6版 第9/10章Spring集成)

《Spring Integration in Action》

4. 视频推荐:

Oreilly: Understanding Spring Integration [Video]

Introduction

Spring Integration provides an extension of the Spring programming model to support the well-known enterprise integration patterns. It enables lightweight messaging within Spring-based applications and supports integration with external systems via declarative adapters. Those adapters provide a higher level of abstraction over Spring’s support for remoting, messaging, and scheduling.

使⽤Spring Integration实现通⽤的集成模式。Spring Integration是众多集成模式的现成实现，这些模式在Gregor Hohpe和Bobby Woolf编写的《企业集成模式》（Enterprise Integration Patterns，Addison-Wesley，2003）中进⾏了归类。每个模式都实现为⼀个组件，消息会通过该组件在管道中传递数据。借助Spring配置，可以将这些组件组装成⼀个管道，数据可以通过这个管道来流动。

声明⼀个简单的集成流
通常来讲，在使⽤Spring Integration创建集成流时，是通过声明⼀个应⽤程序能够接收或发送哪些数据到应⽤程序之外的资源来实现的。应⽤程序可能集成的资源之⼀就是⽂件系统。因此，Spring Integration的很多组件都有读⼊和写⼊⽂件的通道适配器（channel adapter）。

为了熟悉Spring Integration，我们将会创建⼀个集成流，这个流会写⼊数据到⽂件系统中。⾸先，我们需要添加Spring Integration到项⽬的构建⽂件中。对于Maven构建来讲，必要的依赖如下：

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-integration</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.integration</groupId>
			<artifactId>spring-integration-file</artifactId>
		</dependency>

第⼀项依赖是Spring Integration的Spring Boot starter。不管我们与哪种流进⾏交互，对于Spring Integration流的开发来讲，这个依赖都是必需的。与所有的Spring Boot starter⼀样，在Initializr表单中，这个依赖也可以通过复选框进⾏选择。
第⼆项依赖是Spring Integration的⽂件端点模块。这个模块是与外部系统集成的⼆⼗多个模块之⼀。⽂件端点模块提供了将⽂件从⽂件系统导⼊集成流和/或将流中的数据写⼊⽂件系统的能⼒。

其他技术细节可以参考文章开头的书籍推荐……

ARCHITECTURE

• Lightweight message-driven architecture
• Passes messages using "pipes and filters"
• No direct tie in to an enterprise service bus
• Leverages task schedulers and task executers
• Adapters facilitate communication between systems

Two areas of focus for Spring Integration: lightweight intra-application messaging and flexible interapplication integration

The figure contains several boxes, and those boxes are connected via pipes. Now substitute “filters” for boxes, and you have the classic pipes-and-filters architectural style.

Anyone familiar with a UNIX-based operating system can appreciate the pipes-and-filters style: it provides the foundation of such operating systems. Consider a basic example:

$> echo foo | sed s/foo/bar/
bar

You can see that it’s literally the pipe symbol being used to connect two commands (the filters). It’s easy to swap out different processing steps or to extend the chain to accomplish more complex tasks while still using these same building blocks (returns elided):

$> cat /usr/share/dict/words | grep ^foo | head -9 | sed s/foo/bar/
bar bard barder bardful bardless bardlessness bardstuff bardy barfaraw

Adapters are used to map the content from outbound messages into the format that some external system expects to receive and to map inbound content from those external systems into messages. 

AVAILABLE ADAPTERS

• Data stores (JDBC, MongoDB, JPA)
• File stores (File system, FTP, SFTP)
• HTTP
• Mail (POP3 or IMAP for receiving, SMTP for sending)
• Messaging (AMQP, JMS)
• Twitter
• Web Services (SOAP, REST, JSON) 
• User Datagram Protocol (UDP)
• Transmission Control Protocol (TCP)
• Java Management Extensions (JMX)
• Remote Method Invocation (RMI)
• Really Simple Syndication (RSS) feeds
• Extensible Messaging and Presence Protocol (XMPP)

附：

JDBC: Java Database Connectivity。

SFTP: Secured File Transfer Protocol。

AMQP: Advanced Message Queuing Protocol，一个提供统一消息服务的应用层标准高级消息队列协议，是应用层协议的一个开放标准，为面向消息的中间件设计。

SOAP: Simple Object Access Protocol，简单对象访问协议，是交换数据的一种协议规范。

REST: Representational State Transfer, 表述性状态传递，REST API 也称为 RESTful API，是遵循 REST 架构规范的应用编程接口（API 或 Web API），支持与 RESTful Web 服务进行交互。

JSON: JavaScript Object Notation，Javascript 对象表示法。

除了Spring Integration，Spring还提供其他Options：

详情请参考：

https://spring.io/event-driven

Spring Integration Components

集成流是由⼀个或多个如下介绍的组件组成的。

• 通道（channel）：将消息从⼀个元素传递到另⼀个元素。
• 过滤器（filter）：基于某些断⾔，条件化地允许某些消息通过流。
• 转换器（transformer）：改变消息的值和/或将消息载荷从⼀种类型转换成另⼀种类型。
• 路由器（router）：将消息路由⾄⼀个或多个通道，通常会基于消息的头信息进⾏路由。
• 切分器（splitter）：将传⼊的消息切割成两个或更多的消息，然后将每个消息发送⾄不同的通道；
• 聚合器（aggregator）：切分器的反向操作，将来⾃不同通道的多个消息合并成⼀个消息。
• 服务激活器（service activator）：将消息传递给某个Java⽅法来进⾏处理，并将返回值发布到输出通道上。
• 通道适配器（channel adapter）：将通道连接到某些外部系统或传输⽅式，可以接受输⼊，也可以写出到外部系统。
• ⽹关（gateway）：通过接⼝将数据传递到集成流中。

Message

 

Each message consists of headers and a payload. The header contains data that’s relevant to the messaging system, such as the Return Address or Correlation ID. The payload contains the actual data to be accessed or processed by the receiver. Messages can have different functions. For example, a Command Message tells the receiver to do something, an Event Message notifies the receiver that something has happened, and a Document Message transfers some data from the sender to the receiver. 

The message is a representation of the contract between the sender and receiver. In some applications the message might be fine to send a reference to an object over the channel, but in others it might be necessary to use a more interoperable representation like an identifier or a serialized version of the original data.

Channel

Channels can be categorized based on two dimensions: type of handoff and type of delivery. The handoff can be either synchronous or asynchronous, and the delivery can be either point-to-point or publish-subscribe. 

EndPoint

Message endpoints can be as simple as routing to another channel or as complicated as splitting the message into multiple parts or aggregating the parts back together. Connections to the application or the outside world are also endpoints, and these connections take the form of channel adapters, messaging gateways, or service activators. 

Channel Adapter

Messaging Gateway 

 

Service Activator

 

Router 

 Splitter

 

 Aggregator

Messaging

• Messages and channels

• Message Endpoints

• Splitting and aggregating message

• Routing and filtering

Integrating systems

• Handling messages with XML payloads

// Transforming between XML and Java
// Transforming XML using XSLT 
// Routing and splitting with XPath 
// Validating XML messages

• Spring Integration and the Java Message Service (JMS)

• Email-based integration

• Filesystem integration

• Spring Integration and web services

Get Started with Spring Integration

(Spring官网)

Using Spring Integration to create a simple application that retrieves data from an RSS Feed (Spring Blog), manipulates the data, and then writes it to a file. This guide uses traditional Spring Integration XML configuration. Other guides show how to use Java Configuration and DSL with and without JDK 8 Lambda expressions.

下载https://spring.io/guides/gs/integration/官网的zip包打开是这样的：

在新窗口单独打开"complete"这个project：

主要文件：

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>2.6.3</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>
	<groupId>com.example</groupId>
	<artifactId>integration-complete</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<name>integration-complete</name>
	<description>Demo project for Spring Boot</description>

	<properties>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-integration</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.integration</groupId>
			<artifactId>spring-integration-feed</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.integration</groupId>
			<artifactId>spring-integration-file</artifactId>
		</dependency>

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.integration</groupId>
			<artifactId>spring-integration-test</artifactId>
			<scope>test</scope>
		</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>

</project>

IntegrationApplication.java

package com.example.integration;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.annotation.ImportResource;

@SpringBootApplication
@ImportResource("/integration/integration.xml")
public class IntegrationApplication {
	public static void main(String[] args) throws Exception {
		ConfigurableApplicationContext ctx = new SpringApplication(IntegrationApplication.class).run(args);
		System.out.println("Hit Enter to terminate");
		System.in.read();
		ctx.close();
	}

}

integration.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns:int="http://www.springframework.org/schema/integration"
	xmlns:file="http://www.springframework.org/schema/integration/file"
	xmlns:feed="http://www.springframework.org/schema/integration/feed"
	xsi:schemaLocation="http://www.springframework.org/schema/integration/feed https://www.springframework.org/schema/integration/feed/spring-integration-feed.xsd
		http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd
		http://www.springframework.org/schema/integration/file https://www.springframework.org/schema/integration/file/spring-integration-file.xsd
		http://www.springframework.org/schema/integration https://www.springframework.org/schema/integration/spring-integration.xsd">

    <feed:inbound-channel-adapter id="news" url="https://spring.io/blog.atom" auto-startup="${auto.startup:true}">
        <int:poller fixed-rate="5000"/>
    </feed:inbound-channel-adapter>

    <int:transformer
            input-channel="news"
            expression="payload.title + ' @ ' + payload.link + '#{systemProperties['line.separator']}'"
            output-channel="file"/>

    <file:outbound-channel-adapter id="file"
            mode="APPEND"
            charset="UTF-8"
            directory="/tmp/si"
            filename-generator-expression="'${feed.file.name:SpringBlog}'"/>

</beans>

Three integration elements are in play here:

• <feed:inbound-channel-adapter>: An inbound adapter that retrieves the posts, one per poll. As configured here, it polls every five seconds. The posts are placed into a channel named news (corresponding to the adapter’s ID).

• <int:transformer>: Transforms entries (com.rometools.rome.feed.synd.SyndEntry) in the news channel, extracting the entry’s title (payload.title) and link (payload.link) and concatenating them into a readable String (and adding a newline). The String is then sent to the output channel named file.

• <file:outbound-channel-adapter>: An outbound channel adapter that writes content from its channel (named file) to a file. Specifically, as configured here, it appends anything in the file channel to a file at /tmp/si/SpringBlog.

FlowTests.java

package com.example.integration;

import static org.assertj.core.api.Assertions.assertThat;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;

import org.junit.jupiter.api.Test;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.integration.endpoint.SourcePollingChannelAdapter;
import org.springframework.integration.support.MessageBuilder;
import org.springframework.messaging.MessageChannel;

import com.rometools.rome.feed.synd.SyndEntryImpl;

@SpringBootTest({ "auto.startup=false",	  // we don't want to start the real feed
				  "feed.file.name=Test" })   // use a different file
public class FlowTests {

	@Autowired
	private SourcePollingChannelAdapter newsAdapter;

	@Autowired
	private MessageChannel news;

	@Test
	public void test() throws Exception {
		assertThat(this.newsAdapter.isRunning()).isFalse();
		SyndEntryImpl syndEntry = new SyndEntryImpl();
		syndEntry.setTitle("Test Title");
		syndEntry.setLink("http://characters/frodo");
		File out = new File("/tmp/si/Test");
		out.delete();
		assertThat(out.exists()).isFalse();
		this.news.send(MessageBuilder.withPayload(syndEntry).build());
		assertThat(out.exists()).isTrue();
		BufferedReader br = new BufferedReader(new FileReader(out));
		String line = br.readLine();
		assertThat(line).isEqualTo("Test Title @ http://characters/frodo");
		br.close();
		out.delete();
	}

}

Run the application:

If you use Gradle, you can run the application by using ./gradlew bootRun. Alternatively, you can build the JAR file by using ./gradlew build and then run the JAR file, as follows:

java -jar build/libs/gs-integration-0.1.0.jar

If you use Maven, you can run the application by using ./mvnw spring-boot:run. Alternatively, you can build the JAR file with ./mvnw clean package and then run the JAR file, as follows:

java -jar target/gs-integration-0.1.0.jar

Verification: 

tail -f /tmp/si/SpringBlog

see the following sample output (though the actual news will differ): 

 Testing the application: (passed) 

公司的案例

（后续整理）

其他资源

JMS Provider：ActiveMQ, Kafka, IBM WebSphere MQ, Solace，RabbitMQ, RocketMQ等

JMS，ActiveMQ，Solace和RxJava记录_Beth_Chan的博客-CSDN博客

Kafka记录_Beth_Chan的博客-CSDN博客

IBM WebSphere MQ_Beth_Chan的博客-CSDN博客