0 准备
准备测试数据:
sensor_1,1624006065247,43.92789292115926
sensor_2,1624006065247,97.45845640790921
sensor_3,1624006065247,41.35949935067326
sensor_4,1624006065247,86.68115422056633
sensor_5,1624006065247,52.53673229860578
sensor_6,1624006065247,56.6603508147016
sensor_7,1624006065247,80.31827896634314
sensor_8,1624006065247,85.2968397027334
sensor_9,1624006065247,67.08038287401958
sensor_10,1624006065247,58.008757044788
sensor_1,1624006065353,43.49476762604196
// 定义样例类,传感器id,时间戳,温度
case class SensorReading(id: String, timestamp: Long, temperature: Double)
1 split分流
分流demo
import org.apache.flink.streaming.api.scala._
object transformerSensor {
def main(args: Array[String]): Unit = {
val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment
val dataStream: DataStream[String] = env.readTextFile("E:\\bigdata\\Flink2\\src\\main\\resources\\sensor.txt")
val dStreamSensor: DataStream[SensorReading] = dataStream.map(
line => {
val splits = line.split(",")
SensorReading(splits(0), splits(1).toLong, splits(2).toDouble)
}
)
//分流
val splitDStream: SplitStream[SensorReading] = dStreamSensor.split(sensorData => {
if (sensorData.temperature < 50) Seq("low") else Seq("high")
})
//取出对应的流
val lowDStream: DataStream[SensorReading] = splitDStream.select("low")
val highDStream: DataStream[SensorReading] = splitDStream.select("high")
val allDStream: DataStream[SensorReading] = splitDStream.select("low","high")
lowDStream.print().setParallelism(1)
env.execute("transformer lowDStream test ")
}
可以看到将数据流分为温度大于50的high流和低于50的low流,以及high和low组成的整个数据流。输出low流结果,只有低于50度的数据才会显示
2 合流操作
2.1 connect合流
//合流
val mapLowDStream: DataStream[(String, Double, String)] = lowDStream.map(x => (x.id, x.temperature, "Normal"))
val maphighLowDStream: DataStream[(String, Double, String)] = highDStream.map(x => (x.id, x.temperature, "Warning"))
val coDStream: ConnectedStreams[(String, Double, String), (String, Double, String)] = mapLowDStream.connect(maphighLowDStream)
val result: DataStream[(String, Double, String)] = coDStream.map(
lowData => (lowData._1, lowData._2, "healthy"),
warningData => (warningData._1, warningData._2, "warning")
)
result.print().setParallelism(1)
env.execute("transformer connect test")
2.2 union合流
val unionDStream: DataStream[(String, Double, String)] = mapLowDStream.union(maphighLowDStream)
unionDStream.print().setParallelism(1)
env.execute("transformer union test")
2.3 两者区别
1. Union之前两个流的类型必须是一样,Connect可以不一样,在之后的coMap中再去调整成为一样的。
2. Connect只能操作两个流,Union可以操作多个。