Flink多并行度水印测试
数据来源为kafka,一共4个分区,此时为了直观分析,将flink的并发度设置为3;依次发送数据,分析多并行度情况下flink的水印情况;
设置flink并发度为3,flink窗口大小为3秒,水印延迟为100秒
先上代码:
//获取运行环境
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
//设置使用eventtime,默认是使用processtime
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
//设置并行度为3,默认并行度是当前机器的cpu数量
env.setParallelism(3);
String topic = "test01";
Properties prop = new Properties();
prop.setProperty("bootstrap.servers","vm01:9092");
prop.setProperty("group.id","con1");
FlinkKafkaConsumer010<String> myConsumer = new FlinkKafkaConsumer010<>(topic, new SimpleStringSchema(), prop);
myConsumer.setStartFromGroupOffsets();//默认消费策略
DataStreamSource<String> text = env.addSource(myConsumer);
//解析输入的数据
DataStream<Tuple2<String, Long>> inputMap = text.map(new MapFunction<String, Tuple2<String, Long>>() {
@Override
public Tuple2<String, Long> map(String value) throws Exception {
String[] arr = value.split(",");
return new Tuple2<>(arr[0], Long.parseLong(arr[1]));
}
});
//抽取timestamp和生成watermark
DataStream<Tuple2<String, Long>> waterMarkStream = inputMap.assignTimestampsAndWatermarks(new AssignerWithPeriodicWatermarks<Tuple2<String, Long>>() {
Long currentMaxTimestamp = 0L;
final Long maxOutOfOrderness = 100000L;// 最大允许的乱序时间是100s
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");
/**
* 定义生成watermark的逻辑
* 默认100ms被调用一次
*/
@Nullable
@Override
public Watermark getCurrentWatermark() {
return new Watermark(currentMaxTimestamp - maxOutOfOrderness);
}
//定义如何提取timestamp
@Override
public long extractTimestamp(Tuple2<String, Long> element, long previousElementTimestamp) {
long timestamp = element.f1;
currentMaxTimestamp = Math.max(timestamp, currentMaxTimestamp);
long id = Thread.currentThread().getId();
System.out.println("currentThreadId:"+id+",key:"+element.f0+",eventtime:["+element.f1+"|"+sdf.format(element.f1)+"],currentMaxTimestamp:["+currentMaxTimestamp+"|"+
sdf.format(currentMaxTimestamp)+"],watermark:["+getCurrentWatermark().getTimestamp()+"|"+sdf.format(getCurrentWatermark().getTimestamp())+"]");
return timestamp;
}
});
DataStream<String> window = waterMarkStream.keyBy(0)
.window(TumblingEventTimeWindows.of(Time.seconds(3)))//按照消息的EventTime分配窗口,和调用TimeWindow效果一样
.apply(new WindowFunction<Tuple2<String, Long>, String, Tuple, TimeWindow>() {
/**
* 对window内的数据进行排序,保证数据的顺序
* @param tuple
* @param window
* @param input
* @param out
* @throws Exception
*/
@Override
public void apply(Tuple tuple, TimeWindow window, Iterable<Tuple2<String, Long>> input, Collector<String> out) throws Exception {
String key = tuple.toString();
List<Long> arrarList = new ArrayList<Long>();
Iterator<Tuple2<String, Long>> it = input.iterator();
while (it.hasNext()) {
Tuple2<String, Long> next = it.next();
arrarList.add(next.f1);
}
Collections.sort(arrarList);
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");
String result = key + "," + arrarList.size() + "," + sdf.format(arrarList.get(0)) + "," + sdf.format(arrarList.get(arrarList.size() - 1))
+ "," + sdf.format(window.getStart()) + "," + sdf.format(window.getEnd());
out.collect(result);
}
});
//测试-把结果打印到控制台即可
window.print();
//注意:因为flink是懒加载的,所以必须调用execute方法,上面的代码才会执行
env.execute("eventtime-watermark");
此时依次传入数据,分析可以得到如下:
当第二条数据过来,其为延迟数据,此时本来44线程水印为2020/7/13 23:58:20,但水印不能倒退,所以还是使用之前的2020/7/13 23:58:50作为44线程水印。
当第八条数据达到,此时才会第一次触发001,2020/7/14 00:00:00数据的窗口计算。
结论:
1、依次通过kafka发送数据,如果kafka发送没有给定分区号,也没有给定key值,则数据使用轮循方式进入分区。
2、数据被flink线程处理,在多并行度的情况下,watermark对齐会取所有线程中最小的watermark来作为当前水印
另外如果kafka只往一个分区写入数据(比如kafka总分区数为4,数据只往分区1写),flink只会有一个线程处理该分区,其他线程不能获取数据,故其他线程无法生成或更新水印,导致无法触发窗口计算。
验证如下: