- 需求:基于事件时间EventTime Tumbling Window窗口【5秒】,进行聚合统计:WordCount。
`准备数据`
1000,a,1
2000,a,1
5000,a,1
9999,a,1
11000,a,2
14000,b,1
14999,b,1
- 如果使用基于事件时间
EventTime
窗口统计,需要如下三个步骤:
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.timestamps.BoundedOutOfOrdernessTimestampExtractor;
import org.apache.flink.streaming.api.windowing.time.Time;
/**
* @author liu a fu
* @version 1.0
* @date 2021/3/8 0008
* @DESC [掌握]-Flink Time之事件时间案例【编程】
* 需求:==基于事件时间EventTime Tumbling Window窗口【5秒】,进行聚合统计:WordCount。== 滚动事件时间窗口
*
* 窗口统计案例演示:滚动事件时间窗口(Tumbling EventTime Window),窗口内数据进行词频统计
*/
public class StreamTumblingEventTimeWindow {
public static void main(String[] args) throws Exception {
//1-环境准备
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
// TODO: step1. 设置时间语义为事件时间EventTime
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
//2-数据源source
DataStreamSource<String> inputDataStream = env.socketTextStream("node1.itcast.cn", 9999);
// TODO: step2. 设置事件时间字段,数据类型必须为Long类型
SingleOutputStreamOperator<String> timeDataStream = inputDataStream
.filter(line -> line != null && line.trim().split(",").length == 3)
// 此时,不允许数据延迟,如果延迟,就不处理数据
.assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor<String>(Time.seconds(0)) {
@Override
public long extractTimestamp(String line) {
return Long.parseLong(line.trim().split(",")[0]);
}
});
//3-数据的transformation
/*
1000,a,1
2000,a,1
5000,a,1
9999,a,1
11000,a,2
14000,b,1
14999,b,1
*/
SingleOutputStreamOperator<Tuple2<String, Integer>> sumDataStream = timeDataStream
.map(new MapFunction<String, Tuple2<String, Integer>>() {
@Override
public Tuple2<String, Integer> map(String line) throws Exception {
String[] split = line.trim().split(",");
return Tuple2.of(split[1], Integer.parseInt(split[2]));
}
})
// 先分组
.keyBy(0) // 元组数据类型是,使用下标索引
// TODO: step3. 设置窗口: 5秒 左闭又开区间
.timeWindow(Time.seconds(5))
// 窗口内数据聚合
.sum(1);
//4-数据的sink
sumDataStream.printToErr();
//5-执行器execute
env.execute(StreamTumblingEventTimeWindow.class.getSimpleName());
}
}
1. Flink Time中apply函数
在实际项目开发中,往往对流式数据进行窗口统计时,需要指定窗口是什么(窗口开始时间和窗口结束时间),此时需要使用apply函数,进行窗口数据聚合操作,其中可以获取窗口信息
。
- 对于上面案例的改进, 显示窗口的
开始时间
和窗口结束时间
import org.apache.commons.lang.time.FastDateFormat;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.tuple.Tuple;
import org.apache.flink.api.java.tuple.Tuple1;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.timestamps.BoundedOutOfOrdernessTimestampExtractor;
import org.apache.flink.streaming.api.functions.windowing.WindowFunction;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
import org.apache.flink.util.Collector;
import static org.apache.hadoop.yarn.util.Times.format;
/**
* @author liu a fu
* @version 1.0
* @date 2021/3/8 0008
* @DESC 窗口统计案例演示:滚动事件时间窗口(Tumbling EventTime Window),窗口内数据进行词频统计
*/
public class StreamEventTimeWindowApply {
public static void main(String[] args) throws Exception {
// 1. 执行环境-env
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1) ;
// TODO: step1. 设置时间语义为事件时间EventTime
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
// 2. 数据源-source
SingleOutputStreamOperator<String> filterDataStream = env
.socketTextStream("node1.itcast.cn", 9999)
.filter(line -> null != line && line.trim().split(",").length == 3);
// TODO: step2. 设置事件时间字段,数据类型必须为Long类型
SingleOutputStreamOperator<String> timeDataStream = filterDataStream
// 此时,不允许数据延迟,如果延迟,就不处理数据
.assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor<String>(Time.seconds(0)) {
@Override
public long extractTimestamp(String line) {
return Long.parseLong(line.trim().split(",")[0]);
}
});
// 3. 数据转换-transformation
/*
1000,a,1
2000,a,1
5000,a,1
9999,a,1
11000,a,2
14000,b,1
14999,b,1
*/
SingleOutputStreamOperator<Tuple2<String, Integer>> sumDataStream = timeDataStream
.map(new MapFunction<String, Tuple2<String, Integer>>() {
private FastDateFormat format1 = FastDateFormat.getInstance("yyyy-MM-dd HH:mm:ss");
@Override
public Tuple2<String, Integer> map(String line) throws Exception {
String[] split = line.trim().split(",");
//打印每条数据 就是你输入的数据
System.out.println(format1.format(Long.parseLong(split[0])) + "," + split[1] + "," + split[2]);
//得到什么
return Tuple2.of(split[1], Integer.parseInt(split[2]));
}
})
//先分组 keyBy
.keyBy(0)
// TODO: step3. 设置窗口: 5秒
.timeWindow(Time.seconds(5))
// 窗口内数据聚合 通过apply函数来查看窗口的大小
.apply(new WindowFunction<Tuple2<String, Integer>, Tuple2<String, Integer>, Tuple, TimeWindow>() {
//TODO: 定义时间Flink中的时间格式化类 每一个算子里面定义
private FastDateFormat format2 = FastDateFormat.getInstance("yyyy-MM-dd HH:mm:ss") ;
@Override
public void apply(Tuple tuple,
TimeWindow window,
Iterable<Tuple2<String, Integer>> input,
Collector<Tuple2<String, Integer>> out) throws Exception {
//获取分组的key
String key = (String) ((Tuple1) tuple).f0;
//设置窗口开始和结束的时间
long start = window.getStart();
long end = window.getEnd();
//输出内容 就是触发窗口的结果内容
String output = "window[" + format2.format(start) + "~" + format2.format(end) + "] -> " + key ;
//对窗口中的数据进行聚合
int count = 0;
for (Tuple2<String, Integer> item : input) {
count += item.f1;
}
//最后输出 二元组方式的输出
out.collect(Tuple2.of(output, count));
}
});
// 4. 数据终端-sink
sumDataStream.printToErr();
// 5. 触发执行-execute
env.execute(StreamEventTimeWindowApply.class.getSimpleName()) ;
}
}
2. Flink Time中EventTime窗口起始时间确定?
- 基于事件时间窗口分析时,第一个窗口的起始时间是如何确定的呢??
第一条数据:1970-01-01 08:00:01,a,1
第一个窗口起始时间:1970-01-01 08:00:00
第一条数据:1970-01-01 08:18:31,a,4
第一个窗口起始时间:1970-01-01 08:18:30
假设第一条数据:1000,a,3
,那么计算第一个窗口起始时间:1970-01-01 08:00:00