大数据之flink中join用法

原创 已于 2022-06-16 09:36:21 修改 · 4.2k 阅读 · 5 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#flink #join

于 2022-02-14 19:48:05 首次发布
本文详细介绍了Apache Flink中流处理的四种Join操作：内连接、左外连接、右外连接以及Interval Join，通过实例展示了如何设置窗口和时间区间来实现不同类型的连接。此外，还展示了如何使用Flink从MySQL数据库中查询维度数据，以增强事件处理的上下文信息。

一、join用法

1、将两个流中的数据进行join处理

package cn._51doit.flink.day05;


import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.api.java.tuple.Tuple6;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.timestamps.BoundedOutOfOrdernessTimestampExtractor;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;

public class TumblingWindowJoinDemo {

    public static void main(String[] args) throws Exception {

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);
        env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);

        //1000,A,1
        DataStreamSource<String> leftLines = env.socketTextStream("localhost", 8888);
        //2000,A,2
        DataStreamSource<String> rightLines = env.socketTextStream("localhost", 9999);

        //提取第一个流中数据的EventTime
        DataStream<String> leftWaterMarkStream = leftLines
                .assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor<String>(Time.seconds(0)) {
                    @Override
                    public long extractTimestamp(String line) {
                        return Long.parseLong(line.split(",")[0]);
                    }
                });
        //提取第二个流中数据的EventTime
        DataStream<String> rightWaterMarkStream = rightLines
                .assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor<String>(Time.seconds(0)) {
                    @Override
                    public long extractTimestamp(String line) {
                        return Long.parseLong(line.split(",")[0]);
                    }
                });
        //对第一个流整理成tuple3
        DataStream<Tuple3<Long, String, String>> leftStream = leftWaterMarkStream.map(
                new MapFunction<String, Tuple3<Long, String, String>>() {
                    @Override
                    public Tuple3<Long, String, String> map(String value) throws Exception {
                        String[] fields = value.split(",");
                        return Tuple3.of(Long.parseLong(fields[0]), fields[1], fields[2]);
                    }
                }
        );
        //对第二个流整理成tuple3
        DataStream<Tuple3<Long, String, String>> rightStream = rightWaterMarkStream.map(
                new MapFunction<String, Tuple3<Long, String, String>>() {
                    @Override
                    public Tuple3<Long, String, String> map(String value) throws Exception {
                        String[] fields = value.split(",");
                        return Tuple3.of(Long.parseLong(fields[0]), fields[1], fields[2]);
                    }
                }
        );
        //第一个流（左流）调用join方法关联第二个流（右流），并且在where方法和equalTo方法中分别指定两个流join的条件
        DataStream<Tuple6<Long, String, String, Long, String, String>> joinedStream = leftStream.join(rightStream)
                .where(new KeySelector<Tuple3<Long, String, String>, String>() {
                    @Override
                    public String getKey(Tuple3<Long, String, String> value) throws Exception {
                        return value.f1; //将左流tuple3中的f1作为join的key
                    }
                })
                .equalTo(new KeySelector<Tuple3<Long, String, String>, String>() {
                    @Override
                    public String getKey(Tuple3<Long, String, String> value) throws Exception {
                        return value.f1; //将右流tuple3中的f1作为join的key
                    }
                })
                .window(TumblingEventTimeWindows.of(Time.seconds(5))) //划分EventTime滚动窗口，窗口长度为5秒
                .apply(new MyInnerJoinFunction()); //在apply方法中传入自定义的MyInnerJoinFunction
        joinedStream.print(); //调用print sink 输出结果
        env.execute("TumblingWindowJoinDemo");
    }

}

package cn._51doit.flink.day05;

import org.apache.flink.api.common.functions.JoinFunction;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.api.java.tuple.Tuple6;

public class MyInnerJoinFunction implements JoinFunction<
        Tuple3<Long, String, String>, //第一个数据流（左流）输入的数据类型
        Tuple3<Long, String, String>, //第二个数据流（右流）输入的数据类型
        Tuple6<Long, String, String, Long, String, String>> { //join后输出的数据类型
    //第一个流和第二个流输入的数据在同一个时间窗口内并且join的key相同才会调用join方法
    @Override
    public Tuple6<Long, String, String, Long, String, String> join(
            Tuple3<Long, String, String> left, //第一个数据流（左流）输入的一条数据
            Tuple3<Long, String, String> right) //第二个数据流（右流）输入的一条数据
            throws Exception {
        //能join将两个流的数据放入tuple6中，并返回输出
        return Tuple6.of(left.f0, left.f1, left.f2, right.f0, right.f1, right.f2);
    }
}

2、左外连接

package cn._51doit.flink.day05;


import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.api.java.tuple.Tuple6;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.timestamps.BoundedOutOfOrdernessTimestampExtractor;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;

public class TumblingWindowLeftOuterJoinDemo {

    public static void main(String[] args) throws Exception {

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
        //1000,A,1
        DataStreamSource<String> leftSteam = env.socketTextStream("localhost", 8888);
        //2000,A,2
        DataStreamSource<String> rightStream = env.socketTextStream("localhost", 9999);

        //提取第一个流中数据的EventTime
        DataStream<String> leftWaterMarkStream = leftSteam
                .assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor<String>(Time.seconds(0)) {
                    @Override
                    public long extractTimestamp(String line) {
                        return Long.parseLong(line.split(",")[0]);
                    }
                });
        //提取第二个流中数据的EventTime
        DataStream<String> rightWaterMarkStream = rightStream
                .assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor<String>(Time.seconds(0)) {
                    @Override
                    public long extractTimestamp(String line) {
                        return Long.parseLong(line.split(",")[0]);
                    }
                });
        //对第一个流整理成tuple3
        DataStream<Tuple3<Long, String, String>> leftTuple = leftWaterMarkStream.map(
                new MapFunction<String, Tuple3<Long, String, String>>() {
                    @Override
                    public Tuple3<Long, String, String> map(String value) throws Exception {
                        String[] fields = value.split(",");
                        return Tuple3.of(Long.parseLong(fields[0]), fields[1], fields[2]);
                    }
                }
        );
        //对第二个流整理成tuple3
        DataStream<Tuple3<Long, String, String>> rightTuple = rightWaterMarkStream.map(
                new MapFunction<String, Tuple3<Long, String, String>>() {
                    @Override
                    public Tuple3<Long, String, String> map(String value) throws Exception {
                        String[] fields = value.split(",");
                        return Tuple3.of(Long.parseLong(fields[0]), fields[1], fields[2]);
                    }
                }
        );
        //第一个流(左流)和第二个流(右流)进行LeftOuterJoin
		//在同一个窗口，并且join的条件相等，第一个流中的数据没join上也输出
        DataStream<Tuple6<Long, String, String, Long, String, String>> joinedStream = leftTuple.coGroup(rightTuple)
                .where(new KeySelector<Tuple3<Long, String, String>, String>() {
                    @Override
                    public String getKey(Tuple3<Long, String, String> value) throws Exception {
                        return value.f1;
                    }
                })
                .equalTo(new KeySelector<Tuple3<Long, String, String>, String>() {
                    @Override
                    public String getKey(Tuple3<Long, String, String> value) throws Exception {
                        return value.f1;
                    }
                })
                .window(TumblingEventTimeWindows.of(Time.seconds(5)))
                .apply(new MyLeftOuterJoinFunction());

        joinedStream.print();

        env.execute();
    }
}

package cn._51doit.flink.day05;

import org.apache.flink.api.common.functions.CoGroupFunction;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.api.java.tuple.Tuple6;
import org.apache.flink.util.Collector;

public class MyLeftOuterJoinFunction implements CoGroupFunction<
        Tuple3<Long, String, String>, //左流输入的数据类型
        Tuple3<Long, String, String>, //右流输入的数据类型
        Tuple6<Long, String, String, Long, String, String>> { //输出的数据类型
		
    @Override
    public void coGroup(Iterable<Tuple3<Long, String, String>> first,
                        Iterable<Tuple3<Long, String, String>> second,
                        Collector<Tuple6<Long, String, String, Long, String, String>> out) throws Exception {
        //循环左流的数据，如果有数据说明触发窗口时左流中有数据
        for (Tuple3<Long, String, String> left : first) {
            boolean hasJoined = false;
            //循环右流的数据，如果有数据说明触发窗口时右流中有数据，即join上流
            for (Tuple3<Long, String, String> right : second) {
                //返回两个流join上的数据
                out.collect(Tuple6.of(left.f0, left.f1, left.f2, right.f0, right.f1, right.f2));
                hasJoined = true;
            }
            //如果没有join上，只返回左流的数据
            if (!hasJoined) {
                out.collect(Tuple6.of(left.f0, left.f1, left.f2, null, null, null));
            }
        }
    }
}

3、右外连接

package cn._51doit.flink.day05;


import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.api.java.tuple.Tuple6;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.timestamps.BoundedOutOfOrdernessTimestampExtractor;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;

public class TumblingWindowRightOuterJoinDemo {

    public static void main(String[] args) throws Exception {

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
        //1000,A,1
        DataStreamSource<String> leftSteam = env.socketTextStream("localhost", 8888);
        //2000,A,2
        DataStreamSource<String> rightStream = env.socketTextStream("localhost", 9999);

        //提取第一个流中数据的EventTime
        DataStream<String> leftWaterMarkStream = leftSteam
                .assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor<String>(Time.seconds(0)) {
                    @Override
                    public long extractTimestamp(String line) {
                        return Long.parseLong(line.split(",")[0]);
                    }
                });
        //提取第二个流中数据的EventTime
        DataStream<String> rightWaterMarkStream = rightStream
                .assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor<String>(Time.seconds(0)) {
                    @Override
                    public long extractTimestamp(String line) {
                        return Long.parseLong(line.split(",")[0]);
                    }
                });
        //对第一个流整理成tuple3
        DataStream<Tuple3<Long, String, String>> leftTuple = leftWaterMarkStream.map(
                new MapFunction<String, Tuple3<Long, String, String>>() {
                    @Override
                    public Tuple3<Long, String, String> map(String value) throws Exception {
                        String[] fields = value.split(",");
                        return Tuple3.of(Long.parseLong(fields[0]), fields[1], fields[2]);
                    }
                }
        );
        //对第二个流整理成tuple3
        DataStream<Tuple3<Long, String, String>> rightTuple = rightWaterMarkStream.map(
                new MapFunction<String, Tuple3<Long, String, String>>() {
                    @Override
                    public Tuple3<Long, String, String> map(String value) throws Exception {
                        String[] fields = value.split(",");
                        return Tuple3.of(Long.parseLong(fields[0]), fields[1], fields[2]);
                    }
                }
        );
        //调用coGroup实现left join
        DataStream<Tuple6<Long, String, String, Long, String, String>> joinedStream = leftTuple.coGroup(rightTuple)
                .where(new KeySelector<Tuple3<Long, String, String>, String>() {
                    @Override
                    public String getKey(Tuple3<Long, String, String> value) throws Exception {
                        return value.f1;
                    }
                })
                .equalTo(new KeySelector<Tuple3<Long, String, String>, String>() {
                    @Override
                    public String getKey(Tuple3<Long, String, String> value) throws Exception {
                        return value.f1;
                    }
                })
                .window(TumblingEventTimeWindows.of(Time.seconds(5)))
                .apply(new MyRightOuterJoinFunction());

        joinedStream.print();

        env.execute();
    }
}

package cn._51doit.flink.day05;

import org.apache.flink.api.common.functions.CoGroupFunction;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.api.java.tuple.Tuple6;
import org.apache.flink.util.Collector;

public class MyRightOuterJoinFunction implements CoGroupFunction<
        Tuple3<Long, String, String>, //左流输入的数据类型
        Tuple3<Long, String, String>, //右流输入的数据类型
        Tuple6<Long, String, String, Long, String, String>> { //输出的数据类型
    @Override
    public void coGroup(Iterable<Tuple3<Long, String, String>> first,
                        Iterable<Tuple3<Long, String, String>> second,
                        Collector<Tuple6<Long, String, String, Long, String, String>> out) throws Exception {
        //循环右流的数据，如果有数据说明触发窗口时右流中有数据
        for (Tuple3<Long, String, String> right : second) {
            boolean hasJoined = false;
            //循环左流的数据，如果有数据说明触发窗口时左流中有数据，即join上流
            for (Tuple3<Long, String, String> left : first) {
                //返回两个流join上的数据
                out.collect(Tuple6.of(left.f0, left.f1, left.f2, right.f0, right.f1, right.f2));
                hasJoined = true;
            }
            //如果没有join上，只返回右流的数据
            if (!hasJoined) {
                out.collect(Tuple6.of(null, null, null, right.f0, right.f1, right.f2));
            }
        }
    }
}

4、interval Join

key相等，设置数据存活的范围

package cn._51doit.flink.day05;

import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.api.java.tuple.Tuple6;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.timestamps.BoundedOutOfOrdernessTimestampExtractor;
import org.apache.flink.streaming.api.windowing.time.Time;

public class IntervalJoinDemo {

    public static void main(String[] args) throws Exception {

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
        //1000,A,1
        DataStreamSource<String> leftLines = env.socketTextStream("localhost", 8888);
        //2000,A,2
        DataStreamSource<String> rightLines = env.socketTextStream("localhost", 9999);

        //提取第一个流中数据的EventTime
        DataStream<String> leftWaterMarkStream = leftLines
                .assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor<String>(Time.seconds(0)) {
                    @Override
                    public long extractTimestamp(String line) {
                        return Long.parseLong(line.split(",")[0]);
                    }
                });
        //提取第二个流中数据的EventTime
        DataStream<String> rightWaterMarkStream = rightLines
                .assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor<String>(Time.seconds(0)) {
                    @Override
                    public long extractTimestamp(String line) {
                        return Long.parseLong(line.split(",")[0]);
                    }
                });
        //对第一个流整理成tuple3
        DataStream<Tuple3<Long, String, String>> leftStream = leftWaterMarkStream.map(
                new MapFunction<String, Tuple3<Long, String, String>>() {
                    @Override
                    public Tuple3<Long, String, String> map(String value) throws Exception {
                        String[] fields = value.split(",");
                        return Tuple3.of(Long.parseLong(fields[0]), fields[1], fields[2]);
                    }
                }
        );
        //对第二个流整理成tuple3
        DataStream<Tuple3<Long, String, String>> rightStream = rightWaterMarkStream.map(
                new MapFunction<String, Tuple3<Long, String, String>>() {
                    @Override
                    public Tuple3<Long, String, String> map(String value) throws Exception {
                        String[] fields = value.split(",");
                        return Tuple3.of(Long.parseLong(fields[0]), fields[1], fields[2]);
                    }
                }
        );
        DataStream<Tuple6<Long, String, String, Long, String, String>> joinedStream = leftStream
                .keyBy(t -> t.f1) //指定第一个流分组KeySelector
                .intervalJoin(rightStream.keyBy(t -> t.f1)) //调用intervalJoin方法并指定第二个流的分组KeySelector
                .between(Time.seconds(-1), Time.seconds(1)) //设置join的时间区间范围为当前数据时间±1秒
                .upperBoundExclusive() //默认join时间范围为前后都包括的闭区间，现在设置为前闭后开区间
                .process(new MyProcessJoinFunction()); //调用process方法中传入自定义的MyProcessJoinFunction
        joinedStream.print(); //调用print sink 输出结果
        env.execute("IntervalJoinDemo");
    }
}

二、关联mysql查询维度数据

地理位置信息：使用httpClient查询高德地图

DimDemo

package cn._51doit.flink.day05;

import cn._51doit.flink.day05.func.GeoRichMapFunction;
import com.alibaba.fastjson.JSON;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;


public class DimDemo {

    public static void main(String[] args) throws Exception{

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        DataStreamSource<String> lines = env.socketTextStream("localhost", 8888);

        SingleOutputStreamOperator<LogBean> logBeanDataStream = lines.map(new MapFunction<String, LogBean>() {
            @Override
            public LogBean map(String value) throws Exception {
                LogBean bean = null;
                try {
                    bean = JSON.parseObject(value, LogBean.class);
                } catch (Exception e) {
                    e.printStackTrace();
                }
                return bean;
            }
        });

        SingleOutputStreamOperator<LogBean> filtered = logBeanDataStream.filter(e -> e != null);

        //关联维度信息
        SingleOutputStreamOperator<LogBean> logBeanWithNameDataStream = filtered.map(new RichMapFunction<LogBean, LogBean>() {

            private transient Connection connection;
            private transient PreparedStatement prepareStatement;

            @Override
            public void open(Configuration parameters) throws Exception {
                connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/bigdata?characterEncoding=utf-8", "root", "123456");
                prepareStatement = connection.prepareStatement("select id, name from tb_category where id = ?");
            }

            @Override
            public LogBean map(LogBean value) throws Exception {
                prepareStatement.setInt(1, value.cid);
                ResultSet resultSet = prepareStatement.executeQuery();
                String name = null;
                if (resultSet.next()) {
                    name = resultSet.getString(2);
                }
                resultSet.close();
                value.name = name;
                return value;
            }

            @Override
            public void close() throws Exception {
                if (prepareStatement != null) {
                    prepareStatement.close();
                }
                if (connection != null) {
                    connection.close();
                }
            }
        });

        //查询经纬度，关联位置信息
        SingleOutputStreamOperator<LogBean> result = logBeanWithNameDataStream.map(new GeoRichMapFunction("4924f7ef5c86a278f5500851541cdcff"));

        result.print();

        env.execute();
    }
}

LogBean

package cn._51doit.flink.day05;

public class LogBean {

    public String oid;

    public Integer cid;

    public Double money;

    public Double longitude;

    public Double latitude;

    public String name;

    public String province;

    public String city;

    @Override
    public String toString() {
        return "LogBean{" +
                "oid='" + oid + '\'' +
                ", cid=" + cid +
                ", money=" + money +
                ", longitude=" + longitude +
                ", latitude=" + latitude +
                ", name='" + name + '\'' +
                ", province='" + province + '\'' +
                ", city='" + city + '\'' +
                '}';
    }
}

GeoRichMapFunction

package cn._51doit.flink.day05.func;

import cn._51doit.flink.day05.LogBean;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONObject;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.configuration.Configuration;
import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

public class GeoRichMapFunction extends RichMapFunction<LogBean, LogBean> {

    private String key;

    public GeoRichMapFunction(String key) {
        this.key = key;
    }

    private transient CloseableHttpClient httpclient;

    @Override
    public void open(Configuration parameters) throws Exception {
        httpclient = HttpClients.createDefault();
    }

    @Override
    public LogBean map(LogBean bean) throws Exception {
        double longitude = bean.longitude;
        double latitude = bean.latitude;
        HttpGet httpGet = new HttpGet("https://restapi.amap.com/v3/geocode/regeo?&location="+ longitude+"," +latitude+ "&key=" + key);
        CloseableHttpResponse response = httpclient.execute(httpGet);
        try {
            //System.out.println(response.getStatusLine)
            HttpEntity entity = response.getEntity();
            // do something useful with the response body
            // and ensure it is fully consumed
            String province = null;
            String city = null;
            if (response.getStatusLine().getStatusCode() == 200) {
                //获取请求的json字符串
                String result = EntityUtils.toString(entity);
                //转成json对象
                JSONObject jsonObj = JSON.parseObject(result);
                //获取位置信息
                JSONObject regeocode = jsonObj.getJSONObject("regeocode");
                if (regeocode != null && !regeocode.isEmpty()) {
                    JSONObject address = regeocode.getJSONObject("addressComponent");
                    //获取省市区
                    bean.province = address.getString("province");
                    bean.city = address.getString("city");
                }
            }
        } finally {
            response.close();
        }
        return bean;
    }

    @Override
    public void close() throws Exception {
        if (httpclient != null) {
            httpclient.close();
        }
    }
}
空0008
Direction_Wind的博客
04-11 3171
8。
Flink Join
独立开发，substack顶级编辑
10-30 1632
一.简介 Flink DataStream API中内置有两个可以根据实际条件对数据流进行Join算子：基于间隔的Join和基于窗口的Join。 语义注意事项 创建两个流元素的成对组合的行为类似内连接，如果来自一个流的元素与另一个流没有相对应要连接的元素，则不会发出该元素。 结合在一起的那些元素将其时间戳设置为位于各自窗口中最大时间戳。例如：以[5,10]为边界的窗口将产生连接的元素的时间戳为9。 二.窗口Join 2.1 翻滚窗口（Tumbling Window Join） 执行滚动窗口连接（Tu
Flink 中双流 Join 的深度解析与实战
天冬忘忧的博客
12-02 2344
在大数据实时处理领域，Apache Flink 凭借其强大的流处理能力备受青睐。当面临多流数据关联分析场景时，双流 Join 操作至关重要。Flink DataStream API 贴心地提供了join、coGroup、intervalJoin三个算子助力我们达成双流 Join，接下来将深入探究它们的原理、使用方式及差异。
Flink双流join
最新发布
code_fly
08-20 918
Flink双流JOIN是指将两个独立的数据流按照关联条件进行实时匹配连接的操作，其核心挑战在于处理无限数据流和乱序事件15。
Flink-join的三种方式
JinVijay的博客
02-25 5105
Join /** * * 将两个数据流,进行join * * 如果让两个流能够join上,必须满足以下两个条件 * 1.由于数据是分散在多台机器上,必须将join条件相同的数据通过网络传输到同一台机器的同一个分区中(按照条件进行KeyBy) * 2.让每个流中的数据都放慢,等等对方(划分相同类型,长度一样的窗口) * */ public class EventTumblingWindowJoin { public static void main(String[] args)
Flink 流处理API之Join
hyunbar的博客
04-04 446
1、Window Join stream.join(otherStream) .where(<KeySelector>) .equalTo(<KeySelector>) .window(<WindowAssigner>) .apply(<JoinFunction>) 1.1 Tumbling Wind...
Flink join详解（含两类API及coGroup、connect详解）
weixin_51981189的博客
02-20 4413
join、regular join、interval join、temporal join、connect、coGroup
大数据技术之Flink
weixin_42158569的博客
02-08 973
第1章 Flink概述1.1 Flink是什么1.2 Flink特点表 Flink 和 Streaming对比计算模型 流计算 微批处理时间语义 事件时间、处理时间 处理时间窗口 多、灵活 少、不灵活（窗口必须是批次的整数倍）状态 有 没有流式SQL 有 没有1.4 Flink的应用场景1.5 Flink分层API第2章 Flink快速上手2.1 创建项目在准备好所有的开发环境之后，我们就可以开始开发自己的第一个Flink程序了。
大数据之Flink（三）
wslzoooo的博客
09-08 1451
9.3、转换算子 9.3.1、基本转换算子 9.3.1.1、映射map 一一映射 package transform; import bean.WaterSensor; import org.apache.flink.streaming.api.datastream.DataStreamSource; import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator; import org.apache.flink.st
【大数据】Flink技术简介与入门实践
wnm23的专栏
03-11 1348
Flink 是一个分布式流处理和批处理计算框架，具有高性能、容错性和灵活性。JobManager：JobManager 是 Flink 集群的主节点，负责接收和处理用户提交的作业。解析和验证用户提交的作业。生成执行计划，并将作业图分发给 TaskManager。协调任务的调度和执行。管理作业的状态和元数据信息。：TaskManager 是 Flink 集群的工作节点，负责执行具体的任务。
Flink - 尚硅谷- 大数据高级 Flink 技术精讲 - 1
MiaoSO的博客
08-24 4433
Flink - 尚硅谷- 大数据高级 Flink 技术精讲 Flink - 尚硅谷- 大数据高级 Flink 技术精讲 一、Flink 简介 二、Quick Start 2.1 环境 2.1 Flink 安装包 2.2 nc 2.3 pom 配置 2.2 批处理 2.3 流处理 三、Flink 部署 3.1 Standalone 模式 3.2 Yarn 模式 3.2.1 Flink on Yarn 3.2.2 Session Cluster 3.2.3 Per Job Cluster
Flink双流Join
weixin_63297999的博客
12-01 1407
一个流中有相同 Key 并且位于同一窗口的元素都会保存在同一个迭代器(Iterable)，本示例中绿色流为 greenIterable，橘色流为 orangeIterable，如果要实现 RightJoin，实现原理跟 LeftJoin 一样，需要保证 greenIterable 中没有元素，orangeIterable 中的元素也能输出。需要注意的是，一个元素可能会落在不同的窗口中，因此会在不同窗口中发生关联，例如，绿色流中的0元素。其中a和b分别是上图中绿色流和橘色流中的元素，并且有相同的 key。
Flink Sql：四种Join方式详解（基于flink1.15官方文档）
weixin_52134189的博客
06-07 6596
Regular Joins（常规连接 ），Interval Joins（间隔连接），Temporal Joins（时态连接）， lookup join（查找连接）
Flink维表Join
GavinKai
11-05 4037
衡量指标 总体来讲，关联维表有三个基础的方式：实时数据库查找关联（Per-Record Reference Data Lookup）、预加载维表关联（Pre-Loading of Reference Data）和维表变更日志关联（Reference Data Change Stream），而根据实现上的优化可以衍生出多种关联方式，且这些优化还可以灵活组合产生不同效果（不过为了简单性这里不讨论同时应用多种优化的实现方式）。对于不同的关联方式，我们可以从以下 7 个关键指标来衡量（每个指标的得分将以 1-..
谈谈Flink DataStream API中的三种双流join操作
LittleMagic's Blog
06-15 7118
前言 本文是基础中的基础，看官可以放心食用。 在数据库中的静态表上做OLAP分析时，两表join是非常常见的操作。同理，在流式处理作业中，有时也需要在两条流上做join以获得更丰富的信息。Flink DataStream API为用户提供了3个算子来实现双流join，分别是： join() coGroup() intervalJoin() 本文举例说明它们的使用方法，顺便聊聊比较特殊的inte...
flink DataStream API（六）算子-joining
李玉志的博客
11-17 979
joining 窗口连接将共享一个公共键并位于同一窗口中的两个流的元素连接起来。这些窗口可以通过使用窗口分配器来定义，并在来自两个流的元素上进行计算。 然后将两侧的元素传递给用户定义的 JoinFunction 或 FlatJoinFunction，用户可以在其中输出满足连接条件的结果。 一般用法可以总结如下： stream.join(otherStream) .where(<KeySelector>) .equalTo(<KeySelector>) .win
【Flink知识汇总】【Join专题】
weixin_43589563的博客
03-30 2217
Flink Join 专题Join 的应用场景批Join和流Join的区别Flink 双流Join1. Window Join认识Window JoinFlink APISQL API解决方案的特点解决方案的适用场景2. Interval Join2.1 认识Interval Join2.2 Flink API2.3 SQL API2.4 解决方案的特点2.5 解决方案的适用场景3. Regular Join3.1 认识Regular Join3.2 SQL API3.3 解决方案的特点3.4 解决方案的适
【调优】大数据常见 Join 的使用场景
每天更新大数据面经和技术
11-26 1307
join
flink的regular join/window join/interval join/temporal join/lookup join
oTianShangDiXia的博客
03-18 2020
flink的regular join/window join/interval join/temporal join/lookup join对比
尚大数据技术flink优化
02-28
当存在一对多关系的数据集间需频繁交互时，将较小规模的一方作为广播状态保存下来供另一方查询匹配使用不失为明智之举。此方式特别适用于维表Join操作中，其中一方变动相对较少但又必须保持最新记录的情况[^3]。 ``...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

大数据同盟会

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值