1.join
join() 算子提供的语义为"Window join",即按照指定字段和(滚动/滑动/会话)窗口进行 inner join,支持处理时间和事件时间两种时间特征,只要其中一个数据流中没有数据就不用打印出来
DataStream infoDataStream = . . .
DataStream infoDataStream1 = . . .
DataStream<String> apply = infoDataStream.join(infoDataStream1).where(info -> info.getOrder_no()).equalTo(in -> in.getRequestId())
.window(TumblingProcessingTimeWindows.of(Time.seconds(10)))
.apply(new JoinFunction<Info, UserInfo, String>() {
@Override
public String join(Info info, UserInfo userInfo) throws Exception {
return info.getOrder_no() + "==========" + userInfo.getRequestId();
}
});
apply.print();
2.coGroup
要实现 left/right outer join 可以利用 coGroup() 算子。它的调用方式类似于 join() 算子,也需要开窗,但是 CoGroupFunction 比 JoinFunction 更加灵活,可以按照用户指定的逻辑匹配左流和/或右流的数据并输出
DataStream<Tuple2<String, String>> apply = infoDataStream1.coGroup(infoDataStream).where(info -> info.getRequestId()).equalTo(in -> in.getOrder_no())
.window(TumblingProcessingTimeWindows.of(Time.seconds(10)))
.apply(new CoGroupFunction<UserInfo, Info, Tuple2<String, String>>() {
@Override
public void coGroup(Iterable<UserInfo> iterable, Iterable<Info> iterable1, Collector<Tuple2<String, String>> collector) throws Exception {
for (UserInfo in : iterable) {
boolean isMatched = false;
for (Info info : iterable1) {
// 右流中有对应的记录
collector.collect(new Tuple2<String, String>(in.getRecordCount(), info.getOrder_no()));
isMatched = true;
}
if (!isMatched) {
// 右流中没有对应的记录
collector.collect(new Tuple2<>(in.getRecordCount(), null));
}
}
}
});
apply.print();
3.intervalJoin
join() 和 coGroup() 都是基于窗口做关联的。但是在某些情况下,两条流的数据步调未必一致。例如,订单流的数据有可能在点击流的购买动作发生之后很久才被写入,如果用窗口来圈定,很容易 join 不上。所以 Flink 又提供了"Interval join"的语义,按照指定字段以及右流相对左流偏移的时间区间进行关联,即:
right.timestamp ∈ [left.timestamp + lowerBound; left.timestamp + upperBound]
在运行之前,需要分别在两个流上应用 assignTimestampsAndWatermarks() 方法获取事件时间戳和水印
SingleOutputStreamOperator<String> process = infoDataStream1.keyBy(info -> info.getRequestId()).intervalJoin(infoDataStream.keyBy(in -> in.getOrder_no()))
// 时间间隔,设定下界和上界
.between(Time.seconds(-30), Time.seconds(30))
.process(new ProcessJoinFunction<UserInfo, Info, String>() {
@Override
public void processElement(UserInfo left, Info right, Context ctx, Collector<String> out) throws Exception {
out.collect(StringUtils.join(Arrays.asList(left.getRecordCount(),
right.getOrder_no(), right.getName()),
'\t'));
}
});
process.print();