- SpringCloud DataFlow — 0. 本地部署部署
- SpringCloud DataFlow — 1. 自定义Processor
- SpringCloud DataFlow — 2. 自定义Sink
- SpringCloud DataFlow — 3. 暴露properties
- SpringCloud DataFlow — 4. Prometheus + Grafana 监控
- SpringCloud DataFlow — 5. 多分支负载
Processor
# 参考文档
https://dataflow.spring.io/docs/stream-developer-guides/streams/standalone-stream-kafka/#usagecostprocessor-processor
# Spring Initialzr
https://start.spring.io/
新建Maven项目
1. 基本信息
Group: etl.dmt.quick
Artiface: unpack-processor-kafka
2. 添加依赖
kafka | cloud stream | actuator | cloud connectors
3. 生成项目, 解压缩, 导入IDE
4. 创建UnpackProcessor.java
package etl.dmt.quick.unpackprocessorkafka;
import etl.dmt.quick.dto.InputDataDTO;
import etl.dmt.quick.dto.OutputDataDTO;
import org.springframework.cloud.stream.annotation.EnableBinding;
import org.springframework.cloud.stream.annotation.StreamListener;
import org.springframework.cloud.stream.messaging.Processor;
import org.springframework.messaging.support.MessageBuilder;
@EnableBinding(Processor.class)
public class UnpackProcessor {
private Processor processor;
public CsvUnpack(Processor processor) {
this.processor = processor;
}
@StreamListener(Processor.INPUT)
public void unpack(InputDataDTO inputData) {
OutputDataDTO outputData = new OutputDataDTO();
for(int i=0; i<5; i++) {
this.processor.output().send(MessageBuilder.withPayload(outputData).build());
}
}
}
4.1 也可以这样写input和output
package etl.dmt.quick.unpackprocessorkafka;
import etl.dmt.quick.dto.InputDataDTO;
import etl.dmt.quick.dto.OutputDataDTO;
import org.springframework.cloud.stream.annotation.EnableBinding;
import org.springframework.cloud.stream.annotation.StreamListener;
import org.springframework.cloud.stream.messaging.Processor;
import org.springframework.messaging.handler.annotation.SendTo;
@EnableBinding(Processor.class)
public class UnpackProcessor {
@StreamListener(Processor.INPUT)
@SendTo(Processor.OUTPUT)
public OutputDataDTO unpack(InputDataDTO inputData) {
OutputDataDTO outputData = new OutputDataDTO();
return outputData;
}
}
5. 打包到本地maven仓库中
./mvnw clean install -Dmaven.test.skip=true