flink CEP示例

三分钟时间内,出现三次及以上的温度高于40度就算作是异常温度,进行报警输出

  • 场景介绍

    • 现在工厂当中有大量的传感设备,用于检测机器当中的各种指标数据,例如温度,湿度,气压等,并实时上报数据到数据中心,现在需要检测,某一个传感器上报的温度数据是否发生异常。
  • 异常的定义

    • 三分钟时间内,出现三次及以上的温度高于40度就算作是异常温度,进行报警输出
  • 测试数据
传感器设备mac地址,检测机器mac地址,温度,湿度,气压,数据产生时间

00-34-5E-5F-89-A4,00-01-6C-06-A6-29,38,0.52,1.1,2020-03-02 12:20:32
00-34-5E-5F-89-A4,00-01-6C-06-A6-29,47,0.48,1.1,2020-03-02 12:20:35
00-34-5E-5F-89-A4,00-01-6C-06-A6-29,50,0.48,1.1,2020-03-02 12:20:38
00-34-5E-5F-89-A4,00-01-6C-06-A6-29,31,0.48,1.1,2020-03-02 12:20:39
00-34-5E-5F-89-A4,00-01-6C-06-A6-29,52,0.48,1.1,2020-03-02 12:20:41
00-34-5E-5F-89-A4,00-01-6C-06-A6-29,53,0.48,1.1,2020-03-02 12:20:43
00-34-5E-5F-89-A4,00-01-6C-06-A6-29,55,0.48,1.1,2020-03-02 12:20:45

代码

import java.util

import org.apache.commons.lang3.time.FastDateFormat
import org.apache.flink.cep.PatternSelectFunction
import org.apache.flink.cep.scala.pattern.Pattern
import org.apache.flink.cep.scala.{CEP, PatternStream}
import org.apache.flink.streaming.api.TimeCharacteristic
import org.apache.flink.streaming.api.scala.{DataStream, KeyedStream, StreamExecutionEnvironment}
import org.apache.flink.streaming.api.windowing.time.Time

//定义温度信息pojo
case class DeviceDetail(sensorMac:String,deviceMac:String,temperature:String,dampness:String,pressure:String,date:String)

//报警的设备信息样例类
//传感器设备mac地址,检测机器mac地址,温度
case class AlarmDevice(sensorMac:String,deviceMac:String,temperature:String)

/**
  * 基于FlinkCEP的设备温度检测
  */
object checkTemperature {

  private val format: FastDateFormat = FastDateFormat.getInstance("yyyy-MM-dd HH:mm:ss")
  def main(args: Array[String]): Unit = {
    val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment
    //指定时间类型
    env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
    env.setParallelism(1)
    import org.apache.flink.api.scala._

    //接受数据
    val socDS: DataStream[String] = env.socketTextStream("node01",9999)
    val deviceDS: KeyedStream[DeviceDetail, String] = socDS.map(x => {
      val stringArr: Array[String] = x.split(",")
      DeviceDetail(stringArr(0), stringArr(1), stringArr(2), stringArr(3), stringArr(4), stringArr(5))
    })
    .assignAscendingTimestamps(x => {
       format.parse(x.date).getTime
     })
      .keyBy(x => x.sensorMac)
    //todo:定义Pattern,指定相关条件和模型序列
   val pattern: Pattern[DeviceDetail, DeviceDetail] = Pattern.begin[DeviceDetail]("start")
     .where(x => x.temperature.toInt >= 40)
     .followedByAny("second")
     .where(x => x.temperature.toInt >= 40)
     .followedByAny("third")
     .where(x => x.temperature.toInt >= 40)
     .within(Time.minutes(3))

    //todo:模式检测,将模式应用到流中
    val patternResult: PatternStream[DeviceDetail] = CEP.pattern(deviceDS,pattern)


    //todo:选取结果
    patternResult.select(new MyPatternResultFunction).print()

    //todo: 启动
    env.execute("startTempeature")

  }
}

//自定义PatternSelectFunction
class MyPatternResultFunction extends PatternSelectFunction[DeviceDetail,AlarmDevice]{
  override def select(pattern: util.Map[String, util.List[DeviceDetail]]): AlarmDevice = {
    val startDetails: util.List[DeviceDetail] = pattern.get("start")
    val startResult: DeviceDetail = startDetails.iterator().next()
    println("第一条数据: "+ startResult)
    val secondDetails: util.List[DeviceDetail] = pattern.get("start")
    val secondResult: DeviceDetail = secondDetails.iterator().next()
    println("第二条数据: "+ secondResult)
    val thirdDetails: util.List[DeviceDetail] = pattern.get("start")
    val thirdResult: DeviceDetail = thirdDetails.iterator().next()
    println("第三条数据: "+ thirdResult)
    AlarmDevice(thirdResult.sensorMac,thirdResult.deviceMac,thirdResult.temperature)
  }
}

  • 输入和输出测试-1
    左闭右开,收到第四条数据时,显示满足条件的三条数据在这里插入图片描述
  • 输入和输出测试-2
    在这里插入图片描述
    输出如图所示:共4条。
    原因是由followedByAny引起,详情如下:
    当第一个满足条件的温度52(此时有超过3条温度>=40)出现后,在接受一条数据(左闭右开)显示第一个红框内容
    当再接受一个温度53时,由于非严格紧邻定义,53在前面的3条大于40的温度中任意选择两条,组合成满足条件的:有3个大于40度的条件。此时按照组合公式:共C 3 2共3种组合
    在这里插入图片描述

创建订单之后15分钟之内一定要付款,否则就取消订单

  • 场景介绍

    • 在我们的电商系统当中,经常会发现有些订单下单之后没有支付,就会有一个倒计时的时间值,提示你在15分钟之内完成支付,如果没有完成支付,那么该订单就会被取消,主要是因为拍下订单就会减库存,但是如果一直没有支付,那么就会造成库存没有了,别人购买的时候买不到,然后别人一直不支付,就会产生有些人买不到,有些人买到了不付款,最后导致商家一件产品都卖不出去
  • 需求

    • 创建订单之后15分钟之内一定要付款,否则就取消订单
  • 订单数据格式如下类型字段说明

    • 订单编号

    • 订单状态

      • 1.创建订单,等待支付
      • 2.支付订单完成
      • 3.取消订单,申请退款
      • 4.已发货
      • 5.确认收货,已经完成
    • 订单创建时间

    • 订单金额

20160728001511050311389390,1,2016-07-28 00:15:11,295
20160801000227050311955990,1,2016-07-28 00:16:12,165
20160728001511050311389390,2,2016-07-28 00:18:11,295
20160801000227050311955990,2,2016-07-28 00:18:12,165
20160728001511050311389390,3,2016-07-29 08:06:11,295
20160801000227050311955990,4,2016-07-29 12:21:12,165
20160804114043050311618457,1,2016-07-30 00:16:15,132
20160801000227050311955990,5,2016-07-30 18:13:24,165

代码

import java.util

import org.apache.commons.lang3.time.FastDateFormat
import org.apache.flink.cep.{PatternSelectFunction, PatternTimeoutFunction}
import org.apache.flink.cep.scala.{CEP, PatternStream, pattern}
import org.apache.flink.cep.scala.pattern.Pattern
import org.apache.flink.streaming.api.TimeCharacteristic
import org.apache.flink.streaming.api.functions.timestamps.BoundedOutOfOrdernessTimestampExtractor
import org.apache.flink.streaming.api.scala.{DataStream, KeyedStream, OutputTag, StreamExecutionEnvironment}
import org.apache.flink.streaming.api.windowing.time.Time


/**
  *  订单下单未支付检测
  */

case class OrderDetail(orderId:String,status:String,orderCreateTime:String,price :Double)

object orderCheck {

  private val format: FastDateFormat = FastDateFormat.getInstance("yyy-MM-dd HH:mm:ss")

  def main(args: Array[String]): Unit = {

    val environment: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment
    environment.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
    environment.setParallelism(1)
    import org.apache.flink.api.scala._
    val sourceStream: DataStream[String] = environment.socketTextStream("node01",9999)

    val keyedStream: KeyedStream[OrderDetail, String] = sourceStream.map(x => {
      val strings: Array[String] = x.split(",")
      OrderDetail(strings(0), strings(1), strings(2), strings(3).toDouble)
    }).assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor[OrderDetail](Time.seconds(5)) {
      override def extractTimestamp(element: OrderDetail): Long = {
        format.parse(element.orderCreateTime).getTime
      }
    }).keyBy(x=>x.orderId)

    //定义pattern模式,指定条件
    val pattern: Pattern[OrderDetail, OrderDetail] = Pattern.begin[OrderDetail]("start")
      .where(order => order.status.equals("1"))
      .followedBy("second")
      .where(order => order.status.equals("2"))
      .within(Time.minutes(15))
    //调用select方法,提取事件序列,超时的事件要做报警提示
    val orderTimeOutputTag = new OutputTag[OrderDetail]("orderTimeOut")
    //将模式应用到流中
    val patternStream: PatternStream[OrderDetail] = CEP.pattern(keyedStream,pattern)

    //选择结果
    val selectedStream : DataStream[OrderDetail] = patternStream.select(orderTimeOutputTag, new OrderTimeOutPatternFunction,new OrderPatternFunction)
    //打印测输出流数据,过了15分支还没支付的数据
    selectedStream.getSideOutput(orderTimeOutputTag).print()

    environment.execute()
  }
}
//订单超时检查
class OrderTimeOutPatternFunction extends PatternTimeoutFunction[OrderDetail,OrderDetail]{
  override def timeout(pattern: util.Map[String, util.List[OrderDetail]], timeoutTimestamp: Long): OrderDetail = {
    val detail: OrderDetail = pattern.get("start").iterator().next()
    println("超时订单号为: "+ detail)
    detail
  }
}
class OrderPatternFunction extends PatternSelectFunction[OrderDetail,OrderDetail]{
  override def select(pattern: util.Map[String, util.List[OrderDetail]]): OrderDetail = {
    val detail: OrderDetail = pattern.get("second").iterator().next()
    println("支付成功的订单为: "+ detail)
    detail
  }
}

输出
在这里插入图片描述

©️2020 CSDN 皮肤主题: 大白 设计师:CSDN官方博客 返回首页