flink的侧输出流
在 flink 处理数据流时,我们经常会遇到这样的情况:在处理一个数据源时,往往需要将该源中的不同类型的数据做分割处理,如果使用 filter 算子对数据源进行筛选分割的话,势必会造成数据流的多次复制,造成不必要的性能浪费;flink 中的侧输出就是将数据流进行分割,而不对流进行复制的一种分流机制。flink 的侧输出的另一个作用就是对延时迟到的数据进行处理,这样就可以不必丢弃迟到的数据。
/**
* 把呼叫成功的Stream(主流)和不成功的Stream(侧流)分别输出。
*/
object TestSideOutputStream {
//导入隐式转换,建议写在这里,可以防止IDEA代码提示出错的问题import org.apache.flink.streaming.api.scala._
//侧输出流首先需要定义一个流的标签
var notSuccessTag= new OutputTag[StationLog]("not_success")
def main(args: Array[String]): Unit = {
//初始化Flink的Streaming(流计算)上下文执行环境val streamEnv: StreamExecutionEnvironment =
StreamExecutionEnvironment.getExecutionEnvironment
streamEnv.setParallelism(1)
//读取文件数据
val data = streamEnv.readTextFile(getClass.getResource("/station.log").getPath)
.map(line=>{
var arr =line.split(",") new
StationLog(arr(0).trim,arr(1).trim,arr(2).trim,arr(3).trim,arr(4).trim.toLong,arr(5).trim.to Long)
})
val mainStream: DataStream[StationLog] = data.process(new CreateSideOutputStream(notSuccessTag))
//得到侧流
val sideOutput: DataStream[StationLog] = mainStream.getSideOutput(notSuccessTag)
mainStream.print("main") sideOutput.print("sideoutput")
streamEnv.execute()
}
class CreateSideOutputStream(tag: OutputTag[StationLog]) extends ProcessFunction[StationLog,StationLog]{
override def processElement(value: StationLog, ctx: ProcessFunction[StationLog, StationLog]#Context, out: Collector[StationLog]): Unit = {
if(value.callType.equals("success")){//输出主流out.collect(value)
}else{//输出侧流ctx.output(tag,value)
}
}
}
}
源数据的格式
station_8,18600007699,18900003716,barring,1577080459130,0
station_0,18600003502,18900009859,fail,1577080459130,0
station_0,18600003502,18900009859,success,1577080468130,0
station_8,18600007699,18900003716,barring,1577080459130,0
station_0,18600003502,18900009859,fail,1577080459130,0
station_0,18600003502,18900009859,success,1577080468130,0
station_5,18600003713,18900000824,busy,1577080457129,0