Flink-入门编程及算子介绍

最新推荐文章于 2024-04-25 15:07:47 发布
最新推荐文章于 2024-04-25 15:07:47 发布
阅读量374 收藏 1
点赞数 1
分类专栏： flink 文章标签： flink
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_30868737/article/details/113339221
版权

编程模型

在这里插入图片描述
Flink提供了不同级别的编程抽象，通过调用抽象的数据集调用算子构建DataFlow就可以实现对分布式的数据进行流式计算和离线计算，DataSet是批处理的抽象数据集，DataStream是流式计算的抽象数据集，他们的方法都分别为Source、Transformation、Sink

  • Source主要负责数据的读取
  • Transformation主要负责对数据的转换操作
  • Sink负责最终计算好的结果数据输出。

DataStream实时wordcount

package com.wedoctor.flink
import org.apache.flink.streaming.api.scala._

object WordCountDemo {

  def main(args: Array[String]): Unit = {

    val env = StreamExecutionEnvironment.getExecutionEnvironment
    val lines: DataStream[String] = env.socketTextStream("192.168.xx.xx",9999)
    val words: DataStream[String] = lines.flatMap(_.split(" "))
    val wordWithOne: DataStream[(String, Int)] = words.map((_,1))
    val keyedData: KeyedStream[(String, Int), String] = wordWithOne.keyBy(_._1)
    val sumData: DataStream[(String, Int)] = keyedData.sum(1)
    sumData.print()
    env.execute("Flink WordCount")
  }
}

DataSet 离线wordcount

package com.wedoctor.flink
import org.apache.flink.api.scala._
object WordCountDemo2 {
  def main(args: Array[String]): Unit = {
    val env = ExecutionEnvironment.getExecutionEnvironment
    val words: DataSet[Int] = env.fromElements(1,2,3)
    val tt: DataSet[Int] = words.map(t=>t*2)
    tt.print()
  }
}

Flink常见算子

1.map

package com.wedoctor.flink;

import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class MapTest {
    public static void main(String[] args) throws Exception {

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> lines = env.socketTextStream("192.168.xx.xx", 9999);
        SingleOutputStreamOperator<String> words = lines.map(new MapFunction<String, String>() {
            @Override
            public String map(String value) throws Exception {
                return value.toUpperCase();
            }
        });
        words.print();
        env.execute();
    }
}

2.RichMapFunction

package com.wedoctor.flink;

import org.apache.flink.api.common.functions.FilterFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class RichMapTest {

    //RichMapFunction
    //1.可以获取运行时上下文，可以得到很多的信息，subTaskIndex、状态数据等
    //2.还可以使用两个生命周期方法、open和close
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> lines = env.socketTextStream("192.168.XX.XX", 9999);
        SingleOutputStreamOperator<String> map = lines.map(new RichMapFunction<String, String>() {
            //构造对象完成后，map方法执行之前，执行一次
            @Override
            public void open(Configuration parameters) throws Exception {
                super.open(parameters);
                //此处可以建立连接
            }
            @Override
            public String map(String value) throws Exception {
                //处理数据
                return value + "222222222";
            }
            //subtask在停止之前，执行一次
            @Override
            public void close() throws Exception {
                super.close();
                //关闭连接
            }
        });
        map.print();
        env.execute();
    }
}

3.flatMap

package com.wedoctor.flink;

import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

public class FlatMapTest {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> lines = env.socketTextStream("192.168.xx.xx", 9999);
        SingleOutputStreamOperator<String> flatMap = lines.flatMap(new FlatMapFunction<String, String>() {
            @Override
            public void flatMap(String value, Collector<String> collector) throws Exception {
                String[] words = value.split(" ");
                for (String word : words) {
                    collector.collect(word);
                }
            }
        });
        flatMap.print();
        env.execute();
    }
}

4.filter

package com.wedoctor.flink;

import org.apache.flink.api.common.functions.FilterFunction;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class RichMapTest {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> lines = env.socketTextStream("192.168.xx.xx", 9999);
        SingleOutputStreamOperator<String> filter = lines.filter(new FilterFunction<String>() {
            @Override
            public boolean filter(String value) throws Exception {
                return value.length() == 2;
            }
        });
        filter.print();
        env.execute();
    }
}

5.keyBy

5.1.单个字段keyby

package com.wedoctor.flink;

import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.java.tuple.Tuple;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

public class KeyByDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> lines = env.socketTextStream("192.168.xx.xx", 9999);
        SingleOutputStreamOperator<Tuple2<String, Integer>> flatMap = lines.flatMap(new FlatMapFunction<String, Tuple2<String, Integer>>() {
            @Override
            public void flatMap(String s, Collector<Tuple2<String, Integer>> collector) throws Exception {
                String[] words = s.split(" ");
                for (String word : words) {
                    collector.collect(Tuple2.of(word, 1));
                }
            }
        });
        //按照单个字段分组 keyby
        KeyedStream<Tuple2<String, Integer>, Tuple> keyBy = flatMap.keyBy(0);
        KeyedStream<Tuple2<String, Integer>, String> keyBy1 = flatMap.keyBy(t -> t.f0);
        keyBy.print();
        keyBy1.print();
        env.execute();
    }
}

5.2 多个字段keyBy(过时API)

package com.wedoctor.flink;

import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.tuple.Tuple;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class KeyByDemo {
    public static void main(String[] args) throws Exception {
       // jack 01 1232
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> lines = env.socketTextStream("192.168.xx.xx", 9999);
        SingleOutputStreamOperator<Tuple3<String, String, Integer>> map = lines.map(new MapFunction<String, Tuple3<String, String, Integer>>() {
            @Override
            public Tuple3<String, String, Integer> map(String s) throws Exception {
                String[] words = s.split(" ");
                String userId = words[0];
                String monthId = words[1];
                Integer orderCnt = Integer.parseInt(words[2]);
                return Tuple3.of(userId, monthId, orderCnt);
            }
        });
        KeyedStream<Tuple3<String, String, Integer>, Tuple> key = map.keyBy(0, 1);
        SingleOutputStreamOperator<Tuple3<String, String, Integer>> summed = key.sum(2);
        summed.print();
        env.execute();
    }
}

5.3.多个字段KeyBy(新API，Tuple封装)

package com.wedoctor.flink;

import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.api.java.tuple.Tuple;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class KeyByDemo {
    public static void main(String[] args) throws Exception {
       // jack 01 1232
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> lines = env.socketTextStream("192.168.xx.xx", 9999);
        SingleOutputStreamOperator<Tuple3<String, String, Integer>> map = lines.map(new MapFunction<String, Tuple3<String, String, Integer>>() {
            @Override
            public Tuple3<String, String, Integer> map(String s) throws Exception {
                String[] words = s.split(" ");
                String userId = words[0];
                String monthId = words[1];
                Integer orderCnt = Integer.parseInt(words[2]);
                return Tuple3.of(userId, monthId, orderCnt);
            }
        });
        KeyedStream<Tuple3<String, String, Integer>, String> keyBy = map.keyBy(t -> t.f0 + t.f1);
        SingleOutputStreamOperator<Tuple3<String, String, Integer>> summed = keyBy.sum(2);
        summed.print();
        env.execute();
    }
}

5.4 多个字段KeyBy(POJO封装，终极)

package com.wedoctor.flink;

public class WordCount {
    public String word;
    public Integer count;
    public WordCount(String word, Integer count) {
        this.word = word;
        this.count = count;
    }
    public WordCount() {
    }
    public static WordCount of(String word,Integer count){
        return new WordCount(word,count);
    }
    @Override
    public String toString() {
        return "WordCount{" +
                "word='" + word + '\'' +
                ", count=" + count +
                '}';
    }
}

package com.wedoctor.flink;

public class WordCount {
    public String word;
    public Integer count;
    public WordCount(String word, Integer count) {
        this.word = word;
        this.count = count;
    }
    public WordCount() {
    }
    public static WordCount of(String word,Integer count){
        return new WordCount(word,count);
    }
    @Override
    public String toString() {
        return "WordCount{" +
                "word='" + word + '\'' +
                ", count=" + count +
                '}';
    }
}

6.reduce

package com.wedoctor.flink;

import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.ReduceFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

public class ReduceDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> lines = env.socketTextStream("192.168.xx.xx", 9999);
        SingleOutputStreamOperator<Tuple2<String, Integer>> flatMap = lines.flatMap(new FlatMapFunction<String, Tuple2<String, Integer>>() {
            @Override
            public void flatMap(String s, Collector<Tuple2<String, Integer>> collector) throws Exception {
                String[] words = s.split(" ");
                for (String word : words) {
                    collector.collect(Tuple2.of(word, 1));
                }
            }
        });
        SingleOutputStreamOperator<Tuple2<String, Integer>> reduce = flatMap.keyBy(t -> t.f0).reduce(new ReduceFunction<Tuple2<String, Integer>>() {
            @Override
            public Tuple2<String, Integer> reduce(Tuple2<String, Integer> t1, Tuple2<String, Integer> t2) throws Exception {
                return Tuple2.of(t1.f0, t1.f1 + t2.f1);
            }
        });
        reduce.print();
        env.execute();
    }
}

7.Aggregations

7.1 sum

package com.wedoctor.flink;

import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.api.java.tuple.Tuple;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class KeyByDemo {
    public static void main(String[] args) throws Exception {
       // jack 01 1232
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> lines = env.socketTextStream("192.168.xx.xx", 9999);
        SingleOutputStreamOperator<Tuple3<String, String, Integer>> map = lines.map(new MapFunction<String, Tuple3<String, String, Integer>>() {
            @Override
            public Tuple3<String, String, Integer> map(String s) throws Exception {
                String[] words = s.split(" ");
                String userId = words[0];
                String monthId = words[1];
                Integer orderCnt = Integer.parseInt(words[2]);
                return Tuple3.of(userId, monthId, orderCnt);
            }
        });
        KeyedStream<Tuple3<String, String, Integer>, String> keyBy = map.keyBy(t -> t.f0 + t.f1);
        SingleOutputStreamOperator<Tuple3<String, String, Integer>> summed = keyBy.sum(2);
        summed.print();
        env.execute();
    }
}

7.2 min

package com.wedoctor.flink;

import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class AggDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> lines = env.socketTextStream("192.168.x.xx", 9999);
        SingleOutputStreamOperator<Tuple2<String, Integer>> wordAndCnt = lines.map(new MapFunction<String, Tuple2<String, Integer>>() {
            @Override
            public Tuple2<String, Integer> map(String s) throws Exception {
                String[] fileds = s.split(" ");
                String word = fileds[0];
                int cnt = Integer.parseInt(fileds[1]);
                return Tuple2.of(word, cnt);
            }
        });
        //按照单个字段分组 keyby
        KeyedStream<Tuple2<String, Integer>, String> keyBy1 = wordAndCnt.keyBy(t -> t.f0);
        keyBy1.min(1).print();
        env.execute();
    }
}

7.3 max

package com.wedoctor.flink;

import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class AggDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> lines = env.socketTextStream("192.168.xx.xx", 9999);
        SingleOutputStreamOperator<Tuple2<String, Integer>> wordAndCnt = lines.map(new MapFunction<String, Tuple2<String, Integer>>() {
            @Override
            public Tuple2<String, Integer> map(String s) throws Exception {
                String[] fileds = s.split(" ");
                String word = fileds[0];
                int cnt = Integer.parseInt(fileds[1]);
                return Tuple2.of(word, cnt);
            }
        });
        //按照单个字段分组 keyby
        KeyedStream<Tuple2<String, Integer>, String> keyBy1 = wordAndCnt.keyBy(t -> t.f0);
        keyBy1.max(1).print();
        env.execute();
    }
}

7.4 minBy

package com.wedoctor.flink;

import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class AggDemo {
    //lucy 2020-05 15
    //jack 2020-02 25
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> lines = env.socketTextStream("192.168.x.xx", 9999);
        SingleOutputStreamOperator<Tuple3<String,String, Integer>> map = lines.map(new MapFunction<String, Tuple3<String,String, Integer>>() {
            @Override
            public Tuple3<String, String,Integer> map(String s) throws Exception {
                String[] fileds = s.split(" ");
                String userId = fileds[0];
                String monthId = fileds[1];
                int orderCnt = Integer.parseInt(fileds[2]);
                return Tuple3.of(userId,monthId,orderCnt);
            }
        });
        KeyedStream<Tuple3<String, String, Integer>, String> keyBy = map.keyBy(t -> t.f0);
        keyBy.minBy(2,false).print();
        env.execute();
    }
}

7.5 maxBy

package com.wedoctor.flink;

import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class AggDemo {
    //lucy 2020-05 15
    //jack 2020-02 25
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> lines = env.socketTextStream("192.168.xx.xx", 9999);
        SingleOutputStreamOperator<Tuple3<String,String, Integer>> map = lines.map(new MapFunction<String, Tuple3<String,String, Integer>>() {
            @Override
            public Tuple3<String, String,Integer> map(String s) throws Exception {
                String[] fileds = s.split(" ");
                String userId = fileds[0];
                String monthId = fileds[1];
                int orderCnt = Integer.parseInt(fileds[2]);
                return Tuple3.of(userId,monthId,orderCnt);
            }
        });
        KeyedStream<Tuple3<String, String, Integer>, String> keyBy = map.keyBy(t -> t.f0);
        keyBy.maxBy(2,false).print();
        env.execute();
    }
}

8 union

package com.wedoctor.flink;

import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
public class UnionDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        //调用Source创建DataStream
        DataStreamSource<Integer> s1 = env.fromElements(1, 2, 3, 4, 5);
        DataStreamSource<Integer> s2 = env.fromElements(5, 7, 8, 9, 10);
        DataStream<Integer> unioned = s1.union(s2);
        unioned.print();
        env.execute();
    }
}

DataSet API
在这里插入图片描述
DataStream API
和DataSet API的区别为DataStream输入为一个无限的流
在这里插入图片描述
DataStream独有的API
在这里插入图片描述

参考
https://mp.weixin.qq.com/s?__biz=MzIxMjI3NTI5OQ==&mid=2650461709&idx=1&sn=b6f027e02ae9632a38766b5243c4ed32&chksm=8f46ef01b831661733e1c2f78e7f50fd2d806b9032a1b110b60692dd242700f41e90b7d6f474&scene=21#wechat_redirect
Flink DataStream常用算子
Flink的DataSet基本算子总结
Flink系列：常用算子一览表
官网地址

安然烟火
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
48、Flink DataStream API 编程指南（1）- DataStream 入门示例
alanchanchn的专栏
11-29 3万+
通常，只需要使用 getExecutionEnvironment() 即可，因为该方法会根据上下文做正确的处理：如果你在 IDE 中执行你的程序或将其作为一般的 Java 程序执行，那么它将创建一个本地环境，该环境将在你的本地机器上执行你的程序。如果你基于程序创建了一个 JAR 文件，并通过命令行运行它，Flink 集群管理器将执行程序的 main 方法，同时 getExecutionEnvironment() 方法会返回一个执行环境以在集群上执行你的程序。
最全面的flink入门编程案例
09-18
【标题】：“最全面的Flink入门编程案例” 在大数据处理领域，Apache Flink是一个流行的开源流处理框架，它提供了一种低延迟、高吞吐量的数据处理能力，支持批处理和流处理两种模式。本篇文章将深入浅出地引导初学...
Flink系列文档-(YY02)-Flink编程基础-入门示例
HANG.NIAN
11-07 998
Flink系列文档-(YY02)-Flink编程基础-入门示例
10分钟入门Flink--了解Flink
最新发布
baily_ycl的博客
04-25 573
Flink是一个框架和分布式处理引擎，用于对无边界和有边界的数据流进行有状态的计算。Flink是原生的流处理系统，但也提供了批处理API，用于基于流式计算引擎处理批量数据的计算能力，真正实现了批流统一。Flink支持有状态的计算。在流式计算过程中将算子的中间结果保存在内存或者文件系统中，等下一个事件进入算子后可以从之前的状态中获取中间结果，以便计算当前的结果，从而无需每次都基于全部的原始数据来统计结果，极大地提升了系统性能。
Flink 学习二 Flink 编程基础API
weixin_44244088的博客
06-20 2275
flink 的入门代码;介绍source ,sink transformation 等算子的基础概念和用法
flink入门介绍
weixin_45063957的博客
10-15 1614
所以 flatMap 方法也可以实现 map 方法和 filter 方法的功能，当返回结果是 0 个的时候，就相当于对数据进行了过滤，当返回结果是 1 个的时候，相当于对数据进行了简单的转换操作。基于不同的 key，流中的数据将被分配到不同的分区中去，如图 5-8 所示；KeyedStream 可以认为是“分区流”或者“键控流”，它是对 DataStream 按照key 的一个逻辑分区，所以泛型有两个类型：除去当前流中的元素类型外(event)，还需要指定 key 的类型(string)。
Flink1.12 - 概述、安装部署及快速入门
erainm
03-05 2887
1. Flink概述 1.1 Flink官方介绍 flink官网地址 1.2 Flink组件栈 一个计算框架要有长远的发展，必须打造一个完整的 Stack。只有上层有了具体的应用，并能很好的发挥计算框架本身的优势，那么这个计算框架才能吸引更多的资源，才会更快的进步。所以 Flink 也在努力构建自己的 Stack。 Flink分层的组件栈如下图所示：每一层所包含的组件都提供了特定的抽象，用来服务于上层组件。 各层详细介绍： 物理部署层：Flink 支持本地运行、能在独立集群或者在被 YARN 管理
flink-sql所有语法详解-1.13
第一片心意的博客
08-02 6288
本文为 flink 1.13官网中 flink sql 所有语法部分内容的翻译整理。
01-Flink安装部署及入门案例（仅供学习）
weixin_44204449的博客
02-27 635
Apache Flink是一个框架和分布式处理引擎，用于对无界和有界数据流进行有状态计算。Flink设计为在所有常见的集群环境中运行，以内存速度和任何规模执行计算。
Flink从入门到放弃—Stream API—常用算子(reduce）
京河小蚁的博客
11-15 794
reduce对键控数据流的“滚动”统计或者计算。将当前元素与最后减少的值合并并发出新值。普通reduce窗口reduce1、keyBy() -> reduce 算子组合，会计算历史以来所有数据的和，每过来一条数据，就输出一次结果；2、window()->reduce 算子组合，计算的是window窗口内的数据和（也就是一定时间内的数据统计），每次窗口触发的时候，才会输出一次结果。
flink-1.7-中文文档.pdf
10-30
- **数据流编程模型**：介绍Flink的数据流模型及其核心概念，包括流式处理、批处理等。 - **分布式运行时环境**：阐述Flink如何利用分布式系统特性来实现高效的数据处理。 #### 二、教程 - **API教程**： - **...
史上最全最详细的flink 中文教程(一千多页pdf).pdf
11-05
最全最详细的flink 中文教程,详细介绍各个接口,并附带demo(一千多页pdf) 最全最详细的flink 中文教程,详细介绍各个接口,并附带demo(一千多页pdf)
Flink超神文档.pdf
01-29
#### 五、Flink API详解及实操 **5.1 Flink API介绍** - **DataStream API**: 主要用于流式处理。 - **Table API**: 用于 SQL 样式的声明式编程。 - **DataFlow API**: 包括一系列的转换操作，如 Map、Filter 等。...
flink_2.11.tgz.zip
07-21
1. WordCount示例：作为Flink入门的经典例子，WordCount程序读取输入文本，统计每个单词出现的次数，展示了Flink的基本数据处理流程。 2. 时间窗口应用：在实时分析场景中，例如实时电商销售数据分析，可以通过时间...
Flink-安装部署及部署模式介绍
qq_30868737的博客
01-27 5593
Flink支持三大部署模式： 1. Local 本地部署 Flink 可以运行在 Linux、Mac OS X 和 Windows 上。本地模式的安装唯一需要的只是Java 1.7.x或更高版本，本地运行会启动Single JVM，主要用于测试调试代码。 2. Standalone Cluster集群部署 Flink自带了集群模式Standalone，这个模式对软件有些要求： 1.安装Java1.8或者更高版本 2.集群各个节点需要ssh免密登录 3. Flink ON YARN Flink ON YAR
flink-addSource和addSink分别是kafka、自定义数据、mysql、hbase的java实现
qq_30868737的博客
05-25 4361
flink主程序 public class FinkTest { public static void main(String[] args) throws Exception{ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.setStreamTimeCharacteristic(TimeCharacteristic.Ingesti
Flink-flink原理解读
qq_30868737的博客
01-28 1240
1 Task和subtask 1.1 概念 Task(任务):Task 是一个阶段多个功能相同 subTask 的集合，类似于 Spark 中的 TaskSet。 subTask(子任务)：subTask 是 Flink 中任务最小执行单元，是一个 Java 类的实例，这个 Java 类中有属性和方法，完成具体的计算逻辑。 Operator Chains(算子链)：没有 shuffle 的多个算子合并在一个 subTask 中，就形成了 Operator Chains，类似于 Spark 中的
Flink-windows10测试环境配置
qq_30868737的博客
02-07 1073
1.JDK安装 安装(不会请百度)—版本在1.8及之上(如果有多版本的话,在path中将JAVA_HOME放在最前面) 2.Scala安装 下载地址:https://www.scala-lang.org/download/ 环境变量的配置 新增环境变量: SCALA_HOME 3.Flink的下载(JDK1.8及之上) 下载地址: https://flink.apache.org/downloads.html 此处使用版本的是1.9.1 下载后直接解压可使用; 进入解压目录的bin目录，运行start-cl
flink-sql-connector 和flink-connector 区别
05-17
Flink-SQL-Connector 和 Flink-Connector 都是 Apache Flink 的连接器，但它们的主要区别在于用途和功能。 Flink-SQL-Connector 是为了支持 Flink SQL 而开发的一组连接器，它们包括 JDBC、Elasticsearch、Kafka、...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值