Flink流式计算介绍之Transformer

0 准备

准备测试数据：
sensor_1,1624006065247,43.92789292115926
sensor_2,1624006065247,97.45845640790921
sensor_3,1624006065247,41.35949935067326
sensor_4,1624006065247,86.68115422056633
sensor_5,1624006065247,52.53673229860578
sensor_6,1624006065247,56.6603508147016
sensor_7,1624006065247,80.31827896634314
sensor_8,1624006065247,85.2968397027334
sensor_9,1624006065247,67.08038287401958
sensor_10,1624006065247,58.008757044788
sensor_1,1624006065353,43.49476762604196
// 定义样例类，传感器id，时间戳，温度

case class SensorReading(id: String, timestamp: Long, temperature: Double)

1 split分流

分流demo

import org.apache.flink.streaming.api.scala._
object transformerSensor {

  def main(args: Array[String]): Unit = {

    val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment
    val dataStream: DataStream[String] = env.readTextFile("E:\\bigdata\\Flink2\\src\\main\\resources\\sensor.txt")
    val dStreamSensor: DataStream[SensorReading] = dataStream.map(
      line => {
        val splits = line.split(",")
        SensorReading(splits(0), splits(1).toLong, splits(2).toDouble)
      }
    )

    //分流
    val splitDStream: SplitStream[SensorReading] = dStreamSensor.split(sensorData => {
      if (sensorData.temperature < 50) Seq("low") else Seq("high")
    })

    //取出对应的流
    val lowDStream: DataStream[SensorReading] = splitDStream.select("low")
    val highDStream: DataStream[SensorReading] = splitDStream.select("high")
    val allDStream: DataStream[SensorReading] = splitDStream.select("low","high")

    lowDStream.print().setParallelism(1)
    env.execute("transformer lowDStream test ")
｝

可以看到将数据流分为温度大于50的high流和低于50的low流，以及high和low组成的整个数据流。输出low流结果，只有低于50度的数据才会显示

2 合流操作

2.1 connect合流

//合流
val mapLowDStream: DataStream[(String, Double, String)] = lowDStream.map(x => (x.id, x.temperature, "Normal"))
val maphighLowDStream: DataStream[(String, Double, String)] = highDStream.map(x => (x.id, x.temperature, "Warning"))

val coDStream: ConnectedStreams[(String, Double, String), (String, Double, String)] = mapLowDStream.connect(maphighLowDStream)

val result: DataStream[(String, Double, String)] = coDStream.map(
  lowData => (lowData._1, lowData._2, "healthy"),
  warningData => (warningData._1, warningData._2, "warning")

)

result.print().setParallelism(1)
env.execute("transformer connect test")

2.2 union合流

val unionDStream: DataStream[(String, Double, String)] = mapLowDStream.union(maphighLowDStream)
unionDStream.print().setParallelism(1)
env.execute("transformer union test")

2.3 两者区别

1． Union之前两个流的类型必须是一样，Connect可以不一样，在之后的coMap中再去调整成为一样的。
2. Connect只能操作两个流，Union可以操作多个。