本文基于flink 1.11进行测试。
前言
这里所说的join是两个或者多个流的join,涉及流批join的内容或者批批join会另写一篇文章专门说。
Flink的join按照窗口类型分可以分为:Tumbling Window Join、Sliding Window Join和Session Window Join。
按join类型分可以分为join和intervalJoin。前者类似RDBMS中的内连接,interval join使用一个公共键连接两个流的元素(我们现在称它们为A和B),其中流B的元素的时间戳与流A中元素的时间戳之间存在相对时间间隔。
inner join示意图:
interval join示意图:
本文会用到
- Flink1.11 事件时间(event time)、watermark、watermarkstrategy使用详细案例文章的知识。
- Flink window function及常用窗口算子文章的知识。
代码
创建用于join的两个数据源
package it.kenn.source;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
import scala.Tuple3;
import java.util.Random;
public class ForJoinSource1 implements SourceFunction<Tuple3<String, Long, Double>> {
boolean flag = true;
@Override
public void run(SourceContext<Tuple3<String, Long, Double>> ctx) throws Exception {
Random random = new Random();
while (flag) {
int randInt = random.nextInt(100);
ctx.collect(new Tuple3<>("S" + randInt, System.currentTimeMillis(), random.nextDouble() * 1000));
Thread.sleep(30);
}
}
@Override
public void cancel() {
flag = false;
}
}
//-----------------------------------------------------------------------------------
package it.kenn.source;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
import scala.Tuple3;
import java.util.Random;
public class ForJoinSource2 implements SourceFunction<Tuple3<String, Long, Double>> {
boolean flag = true;
@Override
public void run(SourceContext<Tuple3<String, Long, Double>> ctx) throws Exception {
Random random = new Random();
while (flag) {
int randInt = random.nextInt(110);
ctx.collect(new Tuple3<>("S" + randInt, System.currentTimeMillis(), random.nextDouble() * 1000));
Thread.sleep(20);
}
}
@Override
public void cancel() {
flag = false;
}
}
测试主程序
package it.kenn.join;
import it.kenn.source.ForJoinSource1;
import it.kenn.source.ForJoinSource2;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.JoinFunction;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.co.ProcessJoinFunction;
import org.apache.flink.streaming.api.windowing.assigners.SlidingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.util.Collector;
import scala.Tuple3;
import java.time.Duration;
/**
* 测试join
*/
//测试主程序
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
StreamStreamJoinTest joinTest = new StreamStreamJoinTest();
//设置事件时间
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
DataStream<Tuple3<String, Long, Double>> source30 = env.addSource(new ForJoinSource1())
//指定时间戳和watermark规则,注意要指定两个:forBoundedOutOfOrderness指定watermark生成策略,withTimestampAssigner指定那个字段是事件时间
.assignTimestampsAndWatermarks(WatermarkStrategy.<Tuple3<String, Long, Double>>forBoundedOutOfOrderness(Duration.ofMillis(10)).withTimestampAssigner((e, ts) -> e._2()));
DataStream<Tuple3<String, Long, Double>> source20 = env.addSource(new ForJoinSource2())
.assignTimestampsAndWatermarks(WatermarkStrategy.<Tuple3<String, Long, Double>>forBoundedOutOfOrderness(Duration.ofMillis(10)).withTimestampAssigner((e, ts) -> e._2()));
DataStream<Tuple3<String, Long, Double>> tumbJoinedStream = joinTest.tumbJoin(source20, source30);
DataStream<Tuple3<String, Long, Double>> slidingJoinStream = joinTest.slidingJoin(source20, source30);
DataStream<Tuple3<String, Long, Double>> intervalJoinStream = joinTest.intervalJoin(source20, source30);
//对不同join进行测试
tumbJoinedStream.print();
env.execute();
}
inner Tumbling Window Join代码测试
/**
* inner Tumbling Window Join
*
* @param source20
* @param source30
* @return
*/
public DataStream<Tuple3<String, Long, Double>> tumbJoin(DataStream<Tuple3<String, Long, Double>> source20, DataStream<Tuple3<String, Long, Double>> source30) {
DataStream<Tuple3<String, Long, Double>> joinedStream = source20.join(source30)
.where(e -> e._1().split("-")[1])//左流要join的字段
.equalTo(e -> e._1().split("-")[1])//右侧流要join的字段
.window(TumblingEventTimeWindows.of(Time.milliseconds(50)))//指定窗口类型和窗口大小
//join函数,这里说是join但是跟数据库的join有一些区别,比如下面的逻辑并没有取两个流中的数据,而是比较两个流中数据的大小,只返回某个流中的数据
.apply(new JoinFunction<Tuple3<String, Long, Double>, Tuple3<String, Long, Double>, Tuple3<String, Long, Double>>() {
@Override
public Tuple3<String, Long, Double> join(Tuple3<String, Long, Double> left, Tuple3<String, Long, Double> right) throws Exception {
return left._3() > right._3() ? left : right;
}
});
return joinedStream;
}
下图是tumble window join的示意图,但是下面join结果有些歧义,像是笛卡尔积。其实只要在join的时候加上where条件就不可能会产生下面笛卡尔积的情况了。
下图还有一个信息点,在最后一个窗口的时候,只有橙色流中有数据,绿色流中并没有数据,那么这个窗口的计算不会被触发。
sliding Join 测试
/**
* sliding Join 测试
*
* @param source20
* @param source30
* @return
*/
public DataStream<Tuple3<String, Long, Double>> slidingJoin(DataStream<Tuple3<String, Long, Double>> source20, DataStream<Tuple3<String, Long, Double>> source30) {
DataStream<Tuple3<String, Long, Double>> joinedStream = source20.join(source30)
.where(e -> e._1().split("-")[1])//左流要join的字段
.equalTo(e -> e._1().split("-")[1])//右侧流要join的字段
.window(SlidingEventTimeWindows.of(Time.milliseconds(50), Time.milliseconds(30)))//指定窗口类型和窗口大小
.apply(new JoinFunction<Tuple3<String, Long, Double>, Tuple3<String, Long, Double>, Tuple3<String, Long, Double>>() {
@Override
public Tuple3<String, Long, Double> join(Tuple3<String, Long, Double> left, Tuple3<String, Long, Double> right) throws Exception {
return left._3() > right._3() ? left : right;
}
});
return joinedStream;
}
sliding join示意图:
interval join有一个需要注意的特点:有些事件可能在一个滑动窗口中没有被join但是在另外一个滑动窗口中去呗join了。比如上图橙色2号事件,在蓝色窗口中没有与绿色流join,但是在后面的绿色窗口中却与绿色3号join了。
Session Window Join测试
这种方式用到的场景好像不太多,如果哪天我用到了会在这里补上笔记的。
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.streaming.api.windowing.assigners.EventTimeSessionWindows;
import org.apache.flink.streaming.api.windowing.time.Time;
...
DataStream<Integer> orangeStream = ...
DataStream<Integer> greenStream = ...
orangeStream.join(greenStream)
.where(<KeySelector>)
.equalTo(<KeySelector>)
//指定Gap大小
.window(EventTimeSessionWindows.withGap(Time.milliseconds(1)))
.apply (new JoinFunction<Integer, Integer, String> (){
@Override
public String join(Integer first, Integer second) {
return first + "," + second;
}
});
session window join示意图:
上图是session window示意图。可以看到他的原理是通过两个流的间隔时间划分的窗口,这种窗口的数量非常不稳定。如果流中event间隔一直小于指定的GAP,那么窗口会一直不触发。换句话说,这种窗口的触发相比其他窗口而言比较被动,完全是数据驱动的触发,而不是时间驱动的触发。
interval Join 测试
/**
* interval Join 测试
*
* @param source20
* @param source30
* @return
*/
public DataStream<Tuple3<String, Long, Double>> intervalJoin(DataStream<Tuple3<String, Long, Double>> source20, DataStream<Tuple3<String, Long, Double>> source30) {
SingleOutputStreamOperator<Tuple3<String, Long, Double>> intervalJoinedStream = source20.keyBy(e -> e._1().split("-")[1])
.intervalJoin(source30.keyBy(e -> e._1().split("-")[1]))
.between(Time.milliseconds(-12), Time.milliseconds(9))
//默认情况下上面的between条件是包含边界的,如果不希望包含边界可以使用下面两个方法去除
.lowerBoundExclusive()
.upperBoundExclusive()
.process(new ProcessJoinFunction<Tuple3<String, Long, Double>, Tuple3<String, Long, Double>, Tuple3<String, Long, Double>>() {
@Override
public void processElement(Tuple3<String, Long, Double> left, Tuple3<String, Long, Double> right, Context ctx, Collector<Tuple3<String, Long, Double>> out) throws Exception {
out.collect(left._3() > right._3() ? left : right);
}
});
return intervalJoinedStream;
}
interval join示意图:
上图是interval join示意图。之前遇到过一个场景。假设绿色流和黄色流是两组人的运动轨迹。在黄色2位置,某人进了一家餐馆,求跟黄色2号事件前后5分钟同时进入这家餐馆的绿色事件,使用interval join就很合适。
还有一点需要注意,在上面注释也写明了,图中也画出来了,默认情况下between是包含边界的,如果要去掉边界,需要使用上面两个函数去除边界,当然可以根据情况只去除一个边界。