Spring5学习（二）-spring projects之Spring Cloud Data Flow

Spring Cloud Data Flow

Spring Cloud Data Flow is a toolkit（工具包） for building data integration（数据集成） and real-time data processing（实时数据处理） pipelines（管道）.

Pipelines consist of（包括） Spring Boot apps, built using the Spring Cloud Stream or Spring Cloud Task microservice frameworks. This makes Spring Cloud Data Flow suitable（ 适合）for a range of（一系列的） data processing use cases, from import/export to event streaming（事件流） and predictive analytics（预测分析）.

Overview（概述）

The Spring Cloud Data Flow server uses Spring Cloud Deployer, to deploy pipelines onto（到…之上;对…了解;映射到…上） modern runtimes such as Cloud Foundry（云计算）, Kubernetes, Apache Mesos or Apache YARN.

A selection（选择;被选中者;可供选择的事物） of pre-built stream（预建的流） and task/batch starter apps for various（各式各样的） data integration and processing（ 处理;整理;）scenarios（场景） facilitate（促进） learning and experimentation（实验）.

Custom stream and task applications, targeting different middleware（针对不同的中间件） or data services, can be built using the familiar Spring Boot style programming model.

A simple stream pipeline DSL makes it easy to specify which apps to deploy and how to connect outputs and inputs. A new composed task DSL was added in v1.2.

The dashboard offers a graphical editor(图形编辑器) for building new pipelines interactively互动;交互式的, as well as(既…又…;除…之外（也）;此外) views of deployable(可展开的) apps and running apps with metrics(度量).

The Spring Could Data Flow server exposes（暴露） a REST API for composing and deploying data pipelines. A separate（单独的;不同的;分开的，分离的;不相关的） shell makes it easy to work with the API from the command line.

Platform Implementations

An easy way to get started on Spring Cloud Data Flow would be to follow the platform-specific implementation links from the table below. Each of the implementations evolves（ 演变）in isolation（孤立地） with independent（独立的） release cadences（节奏）. It is highly recommended to（强烈推荐） review the platform-specific reference docs to learn more about the feature capabilities（能力）.

Server Type	Stable Release	Milestone/Snapshot Release
Local Server	1.2.3.RELEASE[docs]	1.3.0.M2[docs]
Cloud Foundry Server	1.2.4.RELEASE[docs]	1.3.0.M2[docs]
Kubernetes Server	1.2.2.RELEASE[docs]	1.3.0.M2[docs]
Apache YARN Server	1.2.2.RELEASE[docs]	1.2.3.BUILD-SNAPSHOT[docs]
Apache Mesos Server	1.0.0.RELEASE[docs]	1.1.0.BUILD-SNAPSHOT[docs]

Community Implementations

Spring Cloud Data Flow for HashiCorp Nomad
Spring Cloud Data Flow for Red Hat OpenShift

Quick Start

Step 1 - Download the Spring Cloud Data Flow Local-Server.
Mac users can use 'curl -O' instead of wget.

wget http://repo.spring.io/release/org/springframework/cloud/spring-cloud-dataflow-server-local/1.2.3.RELEASE/spring-cloud-dataflow-server-local-1.2.3.RELEASE.jar

Step 2 - Launch the Local-Server.

java -jar spring-cloud-dataflow-server-local-1.2.3.RELEASE.jar

Step 3 - Download and Start Apache Kafka 0.10 as messaging middleware.
Mac homebrew users can 'brew install kafka'

Step 4 - Open the dashboard at http://localhost:9393/dashboard.

Step 5 - Bulk（散装） Import out-of-the-box（拆盒即可使用的，开箱即用的） stream applications for Apache Kafka using the URI 'http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven'.
(latest bit.ly links here)

Step 6 - Use 'Create Stream' under STREAMS to define and deploy a stream time | log called 'ticktock'.

Once the ‘ticktock’ stream is deployed, two running stream apps will appear（出现，显现） under RUNTIME. Click on 'ticktock.log' to determine the location of the stdout（标准输出） log file.

Step 7 - Verify that events are being written to the ticktock（滴答声） log every second.

tail -f /var/folders/ ... /ticktock.log/stdout_0.log

Building Blocks（阻塞） of Spring Cloud Data Flow

Spring Cloud Data Flow builds upon several projects（在几个项目） and the top-level building blocks of the ecosystem（生态系统） are listed in the following visual representation（表现;陈述;表现…的事物;）. Each project represents a core capability and they evolve in isolation（孤立发展）, with separate release cadences（有独立的发布节奏） - follow the links to find more details about each project.

Spring Cloud Data Flow Local Server

Spring Cloud Data Flow Cloud Foundry Server

Spring Cloud Data Flow Kubernetes Server

Spring Cloud Data Flow Apache Yarn Server

Spring Cloud Data Flow Apache Mesos Server

---

REST-APIs / Shell / DSL	Dashboard	Spring Flo	Spring Cloud Data Flow Metrics Collector
Spring Cloud Data Flow - Core

↓     Uses     ↓

Spring Cloud Deployer - Service Provider Interface (SPI)

↑     Implements     ↑

Spring Cloud Deployer Local

Spring Cloud Deployer Cloud Foundry

Spring Cloud Deployer Kubernetes

Spring Cloud Deployer Yarn

Spring Cloud Deployer Mesos

↓     Deploys     ↓

Spring Cloud Stream App Starters

Spring Cloud Task App Starters

Spring Cloud Stream

Spring Cloud Task

↓     Uses     ↓

Spring Integration

Spring Boot

Spring Batch

说明：

1. Kubernetes：

Kubernetes是Google开源的容器集群管理系统，其提供应用部署、维护、 扩展机制等功能，利用Kubernetes能方便地管理跨机器运行容器化的应用，其主要功能如下：
1) 使用Docker对应用程序包装(package)、实例化(instantiate)、运行(run)。
2) 以集群的方式运行、管理跨机器的容器。
3) 解决Docker跨机器容器之间的通讯问题。
4) Kubernetes的自我修复机制使得容器集群总是运行在用户期望的状态。

（来自：InfoQ）

2. Apache Mesos：见Spring5学习（一）

3. Apache YARN：见Spring5学习（一）

4. DSL：DSL的中文名是数字用户线路，是以电话线为传输介质的传输技术组合。DSL技术在传递公用电话网络的用户环路上支持对称和非对称传输模式，解决了经常发生在网络服务供应商和最终用户间的“最后一公里”的传输瓶颈问题。（来自：百度百科）