Flink 算子汇总

最新推荐文章于 2024-04-12 23:46:39 发布
最新推荐文章于 2024-04-12 23:46:39 发布
阅读量372 收藏 1
点赞数 1
分类专栏： flink 文章标签： flink java 大数据
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_31021213/article/details/125768696
版权

文章目录

Flink 算子汇总

API文档

Map

DataStream → DataStream ：接受一个元素并产生一个元素，一对一关系。

DataStream<Integer> dataStream = //...
dataStream.map(new MapFunction<Integer, Integer>() {
    @Override
    public Integer map(Integer value) throws Exception {
        return 2 * value;
    }
});

FlatMap

DataStream → DataStream ：接受一个元素并产生零个、一个、多个元素

dataStream.flatMap(new FlatMapFunction<String, String>() {
    @Override
    public void flatMap(String value, Collector<String> out)
        throws Exception {
        for(String word: value.split(" ")){
            out.collect(word);
        }
    }
});

Filter

DataStream → DataStream ：评估每个元素的布尔函数并保留函数返回 true 的那些

dataStream.filter(new FilterFunction<Integer>() {
    @Override
    public boolean filter(Integer value) throws Exception {
        return value != 0;
    }
});

KeyBy

DataStream → KeyedStream ：在逻辑上将流划分为不相交的分区。具有相同键的所有记录都分配到同一个分区。在内部，keyBy()是通过散列分区实现的

dataStream.keyBy(value -> value.getSomeKey());
dataStream.keyBy(value -> value.f0);

Reduce

KeyedStream → DataStream ：对同一key的数据进行reduce聚合操作

keyedStream.reduce(new ReduceFunction<Integer>() {
    @Override
    public Integer reduce(Integer value1, Integer value2)
    throws Exception {
        return value1 + value2;
    }
});

Union

DataStream* → DataStream ：两个或多个数据流的联合创建一个包含所有流中的所有元素的新流。注意：如果将数据流与其自身合并，您将在结果流中获得每个元素两次。

dataStream.union(otherStream1, otherStream2, ...);

Split，Select（废弃）->测输出流实现分流

该算子已被废弃，可以通过process接口，打上相应标记来实现分流

Connect，CoMap，CoFlatMap

DataStream,DataStream → ConnectedStream ：“连接”两个保留其类型的数据流。连接允许两个流之间的共享状态，两个流的类型可以不一致。搭配CoMap，CoFlatMap来处理ConnectedStreams(多个流的connect)。

DataStream<Integer> someStream = //...
DataStream<String> otherStream = //...

ConnectedStreams<Integer, String> connectedStreams = someStream.connect(otherStream);

案例Demo

  • source数据流：
package com.ali.flink.demo.utils;

import com.ali.flink.demo.bean.TemperatureBean;
import org.apache.commons.math3.random.RandomDataGenerator;
import org.apache.flink.api.common.functions.RuntimeContext;
import org.apache.flink.runtime.state.FunctionInitializationContext;
import org.apache.flink.streaming.api.functions.source.datagen.DataGenerator;

import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Random;

public class DataGeneratorImpl004 implements DataGenerator<TemperatureBean> {


    RandomDataGenerator generator;

    @Override
    public void open(String s, FunctionInitializationContext functionInitializationContext, RuntimeContext runtimeContext) throws Exception {
        generator = new RandomDataGenerator();
    }

    @Override
    public boolean hasNext() {
        return true;
    }

    @Override
    public TemperatureBean next() {
        Random random = new Random();
        int sleep_cnt = random.nextInt(10);
        try {
            Thread.sleep(1000 * sleep_cnt);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        String[] ids = new String[]{"T1","T2","T3","T4"};
        SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        Date date = new Date();

        TemperatureBean temperatureBean = new TemperatureBean(ids[random.nextInt(4)], random.nextInt(10), simpleDateFormat.format(date), date.getTime());

        return temperatureBean;
    }
}
  • Map
package com.ali.flink.demo.driver;

import com.ali.flink.demo.bean.TemperatureBean;
import com.ali.flink.demo.utils.DataGeneratorImpl004;
import com.ali.flink.demo.utils.FlinkEnv;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource;

/**
 * flink 算子操作
 */
public class FlinkTransformerFunctionDemo01 {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = FlinkEnv.FlinkDataStreamRunEnv();

        env.setParallelism(1);

        DataGeneratorSource<TemperatureBean> dataGeneratorSource = new DataGeneratorSource<>(new DataGeneratorImpl004());

        DataStream<TemperatureBean> dataGeneratorStream = env.addSource(dataGeneratorSource).returns(TemperatureBean.class);
        dataGeneratorStream.print("source");

        // 对TemperatureBean对象数据的temperatureNum+10
        DataStream<TemperatureBean> mapStream = dataGeneratorStream.map(new MapFunction<TemperatureBean, TemperatureBean>() {
            @Override
            public TemperatureBean map(TemperatureBean temperatureBean) throws Exception {
                temperatureBean.setTemperatureNum(temperatureBean.getTemperatureNum() + 10);
                return temperatureBean;
            }
        });

        mapStream.print("mapStream");
        env.execute("tumble window test");
    }
}

-----------------------------结果----------------------------------
source> TemperatureBean{id='T4', temperatureNum=9, eventTime='2022-07-13 18:16:34', wartermark=1657707394228}
mapStream> TemperatureBean{id='T4', temperatureNum=19, eventTime='2022-07-13 18:16:34', wartermark=1657707394228}
  • FlatMap
package com.ali.flink.demo.driver;

import com.ali.flink.demo.bean.TemperatureBean;
import com.ali.flink.demo.utils.DataGeneratorImpl004;
import com.ali.flink.demo.utils.FlinkEnv;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource;
import org.apache.flink.util.Collector;

/**
 * flink 算子操作
 */
public class FlinkTransformerFunctionDemo02 {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = FlinkEnv.FlinkDataStreamRunEnv();

        env.setParallelism(1);

        DataGeneratorSource<TemperatureBean> dataGeneratorSource = new DataGeneratorSource<>(new DataGeneratorImpl004());

        DataStream<TemperatureBean> dataGeneratorStream = env.addSource(dataGeneratorSource).returns(TemperatureBean.class);
        dataGeneratorStream.print("source");

        // 使用map算子构造按逗号分割的字符串
        DataStream<String> mapStream = dataGeneratorStream.map(new MapFunction<TemperatureBean, String>() {
            @Override
            public String map(TemperatureBean temperatureBean) throws Exception {
                return temperatureBean.getId() + "," + temperatureBean.getTemperatureNum();
            }
        });

        // flatmap算子，对map流的数据按逗号分割
        DataStream<String> flapMapStream = mapStream.flatMap(new FlatMapFunction<String, String>() {
            @Override
            public void flatMap(String s, Collector<String> collector) throws Exception {
                String[] split = s.split(",");
                for (String value : split) {
                    collector.collect(value);
                }
            }
        });

        mapStream.print("mapStream");
        flapMapStream.print("flapMapStream");

        env.execute("tumble window test");
    }
}

-----------------------------结果----------------------------------
source> TemperatureBean{id='T1', temperatureNum=3, eventTime='2022-07-13 18:15:10', wartermark=1657707310157}
mapStream> T1,3
flapMapStream> T1
flapMapStream> 3
  • Filter
package com.ali.flink.demo.driver;

import com.ali.flink.demo.bean.TemperatureBean;
import com.ali.flink.demo.utils.DataGeneratorImpl004;
import com.ali.flink.demo.utils.FlinkEnv;
import org.apache.flink.api.common.functions.FilterFunction;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource;

/**
 * flink filter算子操作
 */
public class FlinkTransformerFunctionDemo03 {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = FlinkEnv.FlinkDataStreamRunEnv();

        env.setParallelism(1);

        DataGeneratorSource<TemperatureBean> dataGeneratorSource = new DataGeneratorSource<>(new DataGeneratorImpl004());

        DataStream<TemperatureBean> dataGeneratorStream = env.addSource(dataGeneratorSource).returns(TemperatureBean.class);
        dataGeneratorStream.print("source");

        // 过滤temperatureNum < 5的数据
        SingleOutputStreamOperator<TemperatureBean> filterStream = dataGeneratorStream.filter(new FilterFunction<TemperatureBean>() {
            @Override
            public boolean filter(TemperatureBean temperatureBean) throws Exception {
                return temperatureBean.getTemperatureNum() >= 5;
            }
        });

        filterStream.print("filterStream");


        env.execute("tumble window test");
    }
}

-----------------------------结果----------------------------------
source> TemperatureBean{id='T3', temperatureNum=3, eventTime='2022-07-13 18:21:50', wartermark=1657707710937}
source> TemperatureBean{id='T3', temperatureNum=0, eventTime='2022-07-13 18:21:58', wartermark=1657707718951}
source> TemperatureBean{id='T4', temperatureNum=5, eventTime='2022-07-13 18:21:58', wartermark=1657707718951}
filterStream> TemperatureBean{id='T4', temperatureNum=5, eventTime='2022-07-13 18:21:58', wartermark=1657707718951}
source> TemperatureBean{id='T2', temperatureNum=2, eventTime='2022-07-13 18:22:00', wartermark=1657707720957}
source> TemperatureBean{id='T1', temperatureNum=6, eventTime='2022-07-13 18:22:00', wartermark=1657707720957}
filterStream> TemperatureBean{id='T1', temperatureNum=6, eventTime='2022-07-13 18:22:00', wartermark=1657707720957}
  • KeyBy
package com.ali.flink.demo.driver;

import com.ali.flink.demo.bean.TemperatureBean;
import com.ali.flink.demo.utils.DataGeneratorImpl004;
import com.ali.flink.demo.utils.FlinkEnv;
import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource;
import org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;

import java.time.Duration;

/**
 * flink KeyBy算子操作
 */
public class FlinkTransformerFunctionDemo04 {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = FlinkEnv.FlinkDataStreamRunEnv();

        env.setParallelism(1);

        DataGeneratorSource<TemperatureBean> dataGeneratorSource = new DataGeneratorSource<>(new DataGeneratorImpl004());

        DataStream<TemperatureBean> dataGeneratorStream = env.addSource(dataGeneratorSource).returns(TemperatureBean.class);
        dataGeneratorStream.print("source");

        DataStream<TemperatureBean> maxStream = dataGeneratorStream.assignTimestampsAndWatermarks(WatermarkStrategy.<TemperatureBean>forBoundedOutOfOrderness(Duration.ofSeconds(2))
                .withTimestampAssigner(new SerializableTimestampAssigner<TemperatureBean>() {
                    @Override
                    public long extractTimestamp(TemperatureBean temperatureBean, long l) {
                        return temperatureBean.getWartermark();
                    }
                }))
                // 按 id进行分组
                .keyBy(new KeySelector<TemperatureBean, String>() {
                    @Override
                    public String getKey(TemperatureBean temperatureBean) throws Exception {
                        return temperatureBean.getId();
                    }
                })
                // 设置滚动窗口
                .window(TumblingProcessingTimeWindows.of(Time.seconds(20)))
                // 取分组内的最大温度
                .max("temperatureNum");

        maxStream.print("max temperature");

        env.execute("tumble window test");
    }
}

-----------------------------结果----------------------------------
source> TemperatureBean{id='T2', temperatureNum=3, eventTime='2022-07-13 18:33:43', wartermark=1657708423591}
source> TemperatureBean{id='T2', temperatureNum=1, eventTime='2022-07-13 18:33:49', wartermark=1657708429601}
source> TemperatureBean{id='T4', temperatureNum=2, eventTime='2022-07-13 18:33:55', wartermark=1657708435610}
source> TemperatureBean{id='T3', temperatureNum=6, eventTime='2022-07-13 18:33:55', wartermark=1657708435610}
max temperature> TemperatureBean{id='T2', temperatureNum=3, eventTime='2022-07-13 18:33:43', wartermark=1657708423591}
max temperature> TemperatureBean{id='T3', temperatureNum=6, eventTime='2022-07-13 18:33:55', wartermark=1657708435610}
max temperature> TemperatureBean{id='T4', temperatureNum=2, eventTime='2022-07-13 18:33:55', wartermark=1657708435610}
  • Reduce
package com.ali.flink.demo.driver;

import com.ali.flink.demo.bean.TemperatureBean;
import com.ali.flink.demo.utils.DataGeneratorImpl004;
import com.ali.flink.demo.utils.FlinkEnv;
import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.ReduceFunction;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource;
import org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;

import java.time.Duration;

/**
 * flink Reduce算子操作
 */
public class FlinkTransformerFunctionDemo05 {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = FlinkEnv.FlinkDataStreamRunEnv();

        env.setParallelism(1);

        DataGeneratorSource<TemperatureBean> dataGeneratorSource = new DataGeneratorSource<>(new DataGeneratorImpl004());

        DataStream<TemperatureBean> dataGeneratorStream = env.addSource(dataGeneratorSource).returns(TemperatureBean.class);
        dataGeneratorStream.print("source");

        DataStream<TemperatureBean> reduceStream = dataGeneratorStream.assignTimestampsAndWatermarks(WatermarkStrategy.<TemperatureBean>forBoundedOutOfOrderness(Duration.ofSeconds(2))
                .withTimestampAssigner(new SerializableTimestampAssigner<TemperatureBean>() {
                    @Override
                    public long extractTimestamp(TemperatureBean temperatureBean, long l) {
                        return temperatureBean.getWartermark();
                    }
                }))
                // 按 id进行分组
                .keyBy(new KeySelector<TemperatureBean, String>() {
                    @Override
                    public String getKey(TemperatureBean temperatureBean) throws Exception {
                        return temperatureBean.getId();
                    }
                })
                // 设置滚动窗口
                .window(TumblingProcessingTimeWindows.of(Time.seconds(20)))
                // 取分组内的最大温度
                .reduce(new ReduceFunction<TemperatureBean>() {
                    @Override
                    public TemperatureBean reduce(TemperatureBean temperatureBean, TemperatureBean t1) throws Exception {
                        return temperatureBean.getTemperatureNum() > t1.getTemperatureNum() ? temperatureBean : t1;
                    }
                });

        reduceStream.print("reduceStream");

        env.execute("tumble window test");
    }
}

-----------------------------结果----------------------------------
source> TemperatureBean{id='T1', temperatureNum=2, eventTime='2022-07-13 18:38:29', wartermark=1657708709906}
source> TemperatureBean{id='T3', temperatureNum=7, eventTime='2022-07-13 18:38:32', wartermark=1657708712916}
source> TemperatureBean{id='T3', temperatureNum=2, eventTime='2022-07-13 18:38:39', wartermark=1657708719921}
reduceStream> TemperatureBean{id='T1', temperatureNum=2, eventTime='2022-07-13 18:38:29', wartermark=1657708709906}
reduceStream> TemperatureBean{id='T3', temperatureNum=7, eventTime='2022-07-13 18:38:32', wartermark=1657708712916}
  • Union
package com.ali.flink.demo.driver;

import com.ali.flink.demo.bean.TemperatureBean;
import com.ali.flink.demo.utils.DataGeneratorImpl004;
import com.ali.flink.demo.utils.FlinkEnv;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource;

/**
 * flink union算子操作
 */
public class FlinkTransformerFunctionDemo06 {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = FlinkEnv.FlinkDataStreamRunEnv();

        env.setParallelism(1);

        DataGeneratorSource<TemperatureBean> dataGeneratorSource1 = new DataGeneratorSource<>(new DataGeneratorImpl004());
        DataStream<TemperatureBean> dataGeneratorStream1 = env.addSource(dataGeneratorSource1).returns(TemperatureBean.class);
        dataGeneratorStream1.print("temperature1 source");

        DataGeneratorSource<TemperatureBean> dataGeneratorSource2 = new DataGeneratorSource<>(new DataGeneratorImpl004());
        DataStream<TemperatureBean> dataGeneratorStream2 = env.addSource(dataGeneratorSource2).returns(TemperatureBean.class);
        dataGeneratorStream2.print("temperature2 source");

        DataStream<TemperatureBean> unionStream = dataGeneratorStream1.union(dataGeneratorStream2);

        unionStream.print("union stream");


        env.execute("tumble window test");
    }
}

-----------------------------结果----------------------------------
temperature2 source> TemperatureBean{id='T1', temperatureNum=4, eventTime='2022-07-13 18:45:19', wartermark=1657709119888}
union stream> TemperatureBean{id='T1', temperatureNum=4, eventTime='2022-07-13 18:45:19', wartermark=1657709119888}
union stream> TemperatureBean{id='T4', temperatureNum=0, eventTime='2022-07-13 18:45:20', wartermark=1657709120888}
temperature1 source> TemperatureBean{id='T4', temperatureNum=0, eventTime='2022-07-13 18:45:20', wartermark=1657709120888}
union stream> TemperatureBean{id='T4', temperatureNum=5, eventTime='2022-07-13 18:45:27', wartermark=1657709127900}
temperature2 source> TemperatureBean{id='T1', temperatureNum=8, eventTime='2022-07-13 18:45:27', wartermark=1657709127900}
temperature1 source> TemperatureBean{id='T4', temperatureNum=5, eventTime='2022-07-13 18:45:27', wartermark=1657709127900}
union stream> TemperatureBean{id='T1', temperatureNum=8, eventTime='2022-07-13 18:45:27', wartermark=1657709127900}
  • split, select
package com.ali.flink.demo.driver;

import com.ali.flink.demo.bean.TemperatureBean;
import com.ali.flink.demo.utils.DataGeneratorImpl004;
import com.ali.flink.demo.utils.FlinkEnv;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.ProcessFunction;
import org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource;
import org.apache.flink.util.Collector;
import org.apache.flink.util.OutputTag;

/**
 * flink split,select算子已经废弃，可以通过测输出流的方式来进行分流操作
 */
public class FlinkTransformerFunctionDemo07 {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = FlinkEnv.FlinkDataStreamRunEnv();

        env.setParallelism(1);

        DataGeneratorSource<TemperatureBean> dataGeneratorSource = new DataGeneratorSource<>(new DataGeneratorImpl004());
        DataStream<TemperatureBean> dataGeneratorStream = env.addSource(dataGeneratorSource).returns(TemperatureBean.class);
        dataGeneratorStream.print("temperature source");

        // 创建低温、高温两个数据流
        final OutputTag<TemperatureBean> lowOutputTag = new OutputTag<TemperatureBean>("low_temperature"){};
        final OutputTag<TemperatureBean> highOutputTag = new OutputTag<TemperatureBean>("high_temperature"){};


        SingleOutputStreamOperator<TemperatureBean> processStream = dataGeneratorStream.process(new ProcessFunction<TemperatureBean, TemperatureBean>() {
            @Override
            public void processElement(TemperatureBean temperatureBean, Context context, Collector<TemperatureBean> collector) throws Exception {
                if (temperatureBean.getTemperatureNum() > 5) {
                    context.output(highOutputTag, temperatureBean);
                } else {
                    context.output(lowOutputTag, temperatureBean);
                }
            }
        });

        DataStream<TemperatureBean> highStream = processStream.getSideOutput(highOutputTag);
        DataStream<TemperatureBean> lowStream = processStream.getSideOutput(lowOutputTag);

        highStream.print("high temperature");
        lowStream.print("low temperature");

        env.execute("tumble window test");
    }
}

-----------------------------结果----------------------------------
temperature source> TemperatureBean{id='T4', temperatureNum=9, eventTime='2022-07-13 19:08:59', wartermark=1657710539773}
high temperature> TemperatureBean{id='T4', temperatureNum=9, eventTime='2022-07-13 19:08:59', wartermark=1657710539773}
temperature source> TemperatureBean{id='T2', temperatureNum=2, eventTime='2022-07-13 19:09:08', wartermark=1657710548774}
low temperature> TemperatureBean{id='T2', temperatureNum=2, eventTime='2022-07-13 19:09:08', wartermark=1657710548774}
  • Connect，CoMap，CoFlatMap
package com.ali.flink.demo.driver;

import com.ali.flink.demo.bean.TemperatureBean;
import com.ali.flink.demo.utils.DataGeneratorImpl004;
import com.ali.flink.demo.utils.FlinkEnv;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.api.java.tuple.Tuple4;
import org.apache.flink.streaming.api.datastream.ConnectedStreams;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.ProcessFunction;
import org.apache.flink.streaming.api.functions.co.CoMapFunction;
import org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource;
import org.apache.flink.util.Collector;
import org.apache.flink.util.OutputTag;

/**
 * flink connect, CoMap算子
 */
public class FlinkTransformerFunctionDemo08 {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = FlinkEnv.FlinkDataStreamRunEnv();

        env.setParallelism(1);

        DataGeneratorSource<TemperatureBean> dataGeneratorSource = new DataGeneratorSource<>(new DataGeneratorImpl004());
        DataStream<TemperatureBean> dataGeneratorStream = env.addSource(dataGeneratorSource).returns(TemperatureBean.class);
        dataGeneratorStream.print("temperature source");

        // 创建低温、高温两个数据流
        final OutputTag<TemperatureBean> lowOutputTag = new OutputTag<TemperatureBean>("low_temperature"){};
        final OutputTag<TemperatureBean> highOutputTag = new OutputTag<TemperatureBean>("high_temperature"){};


        SingleOutputStreamOperator<TemperatureBean> processStream = dataGeneratorStream.process(new ProcessFunction<TemperatureBean, TemperatureBean>() {
            @Override
            public void processElement(TemperatureBean temperatureBean, Context context, Collector<TemperatureBean> collector) throws Exception {
                if (temperatureBean.getTemperatureNum() > 5) {
                    context.output(highOutputTag, temperatureBean);
                } else {
                    context.output(lowOutputTag, temperatureBean);
                }
            }
        });

        DataStream<TemperatureBean> highStream = processStream.getSideOutput(highOutputTag);
        DataStream<TemperatureBean> lowStream = processStream.getSideOutput(lowOutputTag);

        highStream.print("high temperature");
        lowStream.print("low temperature");

        DataStream<Tuple3<String, Integer, String>> mapStream = highStream.map(new MapFunction<TemperatureBean, Tuple3<String, Integer, String>>() {
            @Override
            public Tuple3<String, Integer, String> map(TemperatureBean temperatureBean) throws Exception {
                return Tuple3.of(temperatureBean.getId(), temperatureBean.getTemperatureNum(), temperatureBean.getEventTime());
            }
        });

        mapStream.print("mapStream");

        ConnectedStreams<Tuple3<String, Integer, String>, TemperatureBean> connectStream = mapStream.connect(lowStream);

        SingleOutputStreamOperator<Object> resultStream = connectStream.map(new CoMapFunction<Tuple3<String, Integer, String>, TemperatureBean, Object>() {
            @Override
            public Object map1(Tuple3<String, Integer, String> t3) throws Exception {
                return Tuple4.of(t3.f0, t3.f1, t3.f2, "high");
            }

            @Override
            public Object map2(TemperatureBean temperatureBean) throws Exception {
                return Tuple3.of(temperatureBean.getId(), temperatureBean.getTemperatureNum(), temperatureBean.getEventTime());
            }
        });

        resultStream.print("connectStream");

        env.execute("tumble window test");
    }
}

-----------------------------结果----------------------------------
temperature source> TemperatureBean{id='T2', temperatureNum=8, eventTime='2022-07-13 19:19:32', wartermark=1657711172552}
high temperature> TemperatureBean{id='T2', temperatureNum=8, eventTime='2022-07-13 19:19:32', wartermark=1657711172552}
mapStream> (T2,8,2022-07-13 19:19:32)
connectStream> (T2,8,2022-07-13 19:19:32,high)
temperature source> TemperatureBean{id='T2', temperatureNum=3, eventTime='2022-07-13 19:19:35', wartermark=1657711175562}
low temperature> TemperatureBean{id='T2', temperatureNum=3, eventTime='2022-07-13 19:19:35', wartermark=1657711175562}
connectStream> (T2,3,2022-07-13 19:19:35)
coding or coded
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
数据转换必须熟悉的 Operator(算子)
http://www.54tianzhisheng.cn/
09-22 629
在 Flink 应用程序中，无论你的应用程序是批程序，还是流程序，都是上图这种模型，有数据源（source），有数据下游（sink），我们写的应用程序多是对数据源过来的数据做一系列操作，总结如下。 Source: 数据源，Flink 在流处理和批处理上的 source 大概有 4 类：基于本地集合的 source、基于文件的 source、基于网络套接字的 source、自定义的 source...
硬核！一文学完Flink流计算常用算子（Flink算子大全）
wangxi06的专栏
03-12 3116
Flink和Spark类似，也是一种一站式处理的框架；既可以进行批处理（DataSet），也可以进行实时处理（DataStream）。 所以下面将Flink的算子分为两大类：一类是DataSet，一类是DataStream。 DataSet 一、Source算子 1. fromCollection fromCollection：从本地集合读取数据 例： valenv=ExecutionEnvironment.getExecutionEnvironment valtextDataSet...
Flink基础概念及算子
m0_57697768的博客
06-08 2059
无界数据流例如从Kafka这样的消息组件中读取的数据一般，没有数据流结束的定义，即使没有数据也在进行消费。有界数据流有界数据流能够等到所有数据都提取之后再进行处理。有状态流处理将数据的中间状态进行存储，能够重复使用该状态进行处理。Flink的特点Flink计算模型流计算微批处理时间语义事件时间、处理时间处理时间窗口多、灵活少、不灵活（窗口必须是批次的整数倍）状态有没有流式SQL有没有Flink分层API//todo Watersensor Pojo对象。
Flink基础概念及算子_flink 算子
最新发布
2401_84164672的博客
04-12 757
当Flink提供的所有分区策略都不能满足用户的需求时，我们可以通过使用partitionCustom()方法来自定义分区策略。
最强Flink算子大全手册，面试拿捏了~
03-01 3755
Flink算子大全手册，面试好帮手！
Flink:算子
难得糊涂
10-25 1355
整理一下flink的任务吧.我们在搞定了source和sink后.那算子就很重要了.为啥子呢?因为算子就是我们处理source的过程,最后需要sink到指定的存储空间里[email protected]本文基于官网v1.13.2的版本整理。
Flink 自定义算子 读取MySQL (scala源代码)
06-29
Flink 自定义算子 读取MySQL (scala源代码)
flink真实案例-读取kafka下沉redis-中间逻辑处理带原始数据.zip
12-06
java语言实现的flink真实开发案例，从kafka获取数据，中间有逻辑处理，最终下沉到redis中，封装有内部方法，方便根据自己的情况使用。 -- 要求：jdk1.8 ,kafka ,redis 资源珍贵，喜欢的话可以下载。
flink-sql集成rabbitmq
06-29
flink-sql集成rabbitmq
Flink-1.16.2 CDH6.3.2
09-26
CDH6.3.2集成安装Flink-1.16.2，所需的包都在文件里面了，不用再自己打包了 参考部署教程： https://blog.csdn.net/qq_31454379/article/details/110440037 https://www.cnblogs.com/lshan/p/16469294.html
Flink流计算常用算子大全
qq_63483349的博客
01-03 961
Flink 已经提供了若干实现好了的 source functions，当然我们也可以通过实现 SourceFunction 来自定义非并行的source或者实现 ParallelSourceFunction 接口或者扩展 RichParallelSourceFunction 来自定义并行的 source。这个时候本来总体数据量只需要10分钟解决的问题，出现了数据倾斜，机器1上的任务需要4个小时才能完成，那么其他3台机器执行完毕也要等待机器1执行完毕后才算整体将任务完成；逻辑上将流分区为不相交的分区。
大数据之flink常用算子
大数据同盟会的博客
12-13 7506
一、source分类 单并行的Source直接实现了SourceFunction接口 1、socketTextStream("localhost", 8888) 2、fromCollection(Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)) 3、fromElements(1,2,3,4,5,6,7,8,9,10,11,12) 多并行的Source实现ParallelSourceFunction接口 4、generateSequence(1, 100)
Flink的常用算子以及实例（详解）
weixin_51194902的博客
04-23 3748
1. map 特性：接收一个数据，经过处理之后，就返回一个数据 1.1. 源码分析 我们来看看map的源码 map需要接收一个MapFunction<T,R>的对象，其中泛型T表示传入的数据类型，R表示经过处理之后输出的数据类型 我们继续往下点，看看MapFunction<T,R>的源码 这是一个接口，那么在代码中，我们就需要实现这个接口 1.2. 案例 那么我们现在要实现一个功能，就是从给一个文件中读取数据，返回每一行的字符串长度。 我们要读取的文件内容如下 代码贴在这
Flink算子学习笔记-算子详细介绍及示例
qq_29137837的博客
07-27 889
本篇文章深入介绍了 Apache Flink 中常用的算子及其在实际应用中的场景和示例，涵盖了流处理和批处理任务的关键算子。适用于对 Flink 感兴趣的开发者。读者通过本文能够深入了解不同算子的底层逻辑和适用场景，掌握如何利用 Flink 解决实际数据处理问题。文章通过丰富的代码示例和详细解释，帮助读者更好地理解算子的使用方法和效果。无论是处理大规模数据、实时流计算还是批处理任务，本文都为读者提供了有用的指导，使其能够更好地应用 Flink 构建高效、稳定的数据处理应用。
flink常用算子介绍
L13763338360的博客
06-01 3322
Flink中的算子是将一个或多个DataStream转换为新的DataStream，可以将多个转换组合成复杂的数据流拓扑。 在Flink中，有多种不同的DataStream类型，他们之间是使用各种算子进行的。如下图所示： flink中常用的算子 map DataStream --> DataStream：输入一个参数产生一个参数，map的功能是对输入的参数进行转换操作。 flatMap DataStream --> DataStream：输入一个参数，产生0、1或.
Flink学习——基本转换算子
Helen_1997_1997的博客
05-17 531
Flink学习——基本转换算子 filter、map、min、minBy、max、maxBy、sum、reduce
【flink番外篇】1、flink的23种常用算子介绍及详细示例（4）- union、window join、connect、outputtag、cache、iterator、project
热门推荐
alanchanchn的专栏
12-06 1万+
根据 key 相等并且满足指定的时间范围内（e1.timestamp + lowerBound
flink 算子如何结束
07-11
Flink 算子的结束可以通过以下几种方式来实现： 1. 数据源结束：当数据源没有更多的数据可供处理时，Flink 算子会自动结束。这通常发生在流式处理任务中，当数据源没有新的事件到达时。 2. 窗口操作：使用窗口操作...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值