Flink重温笔记(一)
前言:最近发现 Flink 技术在实时数仓开发上占比越来越明显,希望通过抓紧复习,加强对 Flink 的理解和应用,提升自己实时数仓的开发能力,以下是今日整理的学习笔记,各种图像均按照自己理解重画,有疏漏的地方还请各位大佬指出,希望能和大家一起共同学习,互相进步。
Tips:这是我的第一篇 CSDN 博客,后续会持续分享每日学习总结和心得,2024年大家一起加油!
一、Flink基础入门
1. Flink架构体系
-
1- JobManager处理器:
- 别名:Master,如果是HA模式,可能有多个 Master(一个 leader,多个 standby)
- 作用:协调分布式执行,调度 task,协调检查点,数据恢复
-
2- TaskManager处理器:
- 别名:Worker
- 作用:执行一个 dataflow 的 task(特殊的 subtask),数据缓存,data stream交换
-
3- Slot任务执行槽位:
- 概念:物理概念,一个 TaskManager划分多个 slot
- 作用:
- 1个 slot最多运行1个 task( subtask),或者由其组成的任务链
- 1个 slot均分该 taskmanager的资源
-
4- 任务划分:
- job - > 多个 task (有多少个 worker 节点)-> 多个 subtask(每个 worker 节点的并行度)
-
5- 并行度优先级:
- 算子No1:
- - 针对每个算子进行单独设置:sum(1).setParallelism(3)
- 代码全局No2:
- -在代码中设置全局并行度:env.setParallelism(3)
- 命令行参数No3:
- -启动 Flink 任务,动态提交参数:比如:bin/flink run -p 3
- 配置文件No4:
- -配置文件默认并行度:conf/flink-conf.yaml 的 parallelism.default
- 算子No1:
2. Flink on Yarn
(1) 运行机制
-
1- client 将 flink 应用 jar 包和配置文件上传至 HDFS
-
2- client 向 ResourceManager 注册 Resources 和申请 Container,用于启动 Application Master 和 Job Manager
-
3- yarn 分配资源,第一个 container 上 APP Master 和 Job Manager 共处一起(APP Master 知道 Job Manager 地址),就为 Task Manager 生成新的 flink 配置文件(使得 Task Manager 可以连接到 Job Manager),并将配置文件上传到 HDFS, APP Master 提供临时端口,允许用户并行执行多个 Flink
-
4- APP Master 为 Flink 的 Task Manager 分配新的容器,NodeManager 加载 Flink 的 jar 包和配置环境,启动 Task Manager,Task Manager 内部由于进程设置划分 slot,会自动从 HDFS 上下载 jar 包和修改后的配置文件,运行相应 slot 上面的 task,Job Manager 和 Task Manager 也会交互,维持心跳等。
(2) 三种运行方式
- 1- Session 模式
- 特点:需要事先申请资源,使用 flink 中的 yarn-session( yarn 客户端),启动 Job Manager 和 Task Manager
- 优点:会话申请资源后,不需要每次递交作业申请资源,资源充分共享,提升资源利用率
- 缺点:资源隔离性差,任务挂了需要重启所有任务,作业完成后不会释放资源
- 应用场景:适合作业频繁递交,小作业比较多的场景
# 测试:yarn-session.sh(开辟资源)+flink run(提交任务)
bin/yarn-session.sh -tm 1024 -s 4 -d
# 上面的命令的意思是:
每个 TaskManager 拥有4个 Task Slot(-s 4)
被创建的每个 TaskManager 所在的 YARN Container 申请 1024M 的内存
同时额外申请一个 Container 用以运行 ApplicationMaster 以及 Job Manager。
# TaskManager数量取决于并行度数量
bin/flink run -p 8 examples/batch/WordCount.jar
# 8 / 4 = 2, 需要启动 2 个 TaskManager
-
实验现象1:(客户端递交作业后,随机端口打印:JobManager Web Interface: http://node2:8081)
-
实验现象2:( yarn 端口:8088,发现 jobmanager 没有释放资源,因此需要手动释放)
-
手动释放前
-
手动释放后
yarn application -kill application_1707650674062_0001
-
- 2- Per-Job 模式
- 特点:每次递交作业申请一次资源
- 优点:作业完成释放资源,资源隔离效果好
- 缺点:每次提交 job 申请资源,影响执行效率(time)
- 应用场景:适合作业少,大作业场景
# 使用 flink 直接提交任务,指定模式 -m
bin/flink run -m yarn-cluster ./examples/batch/WordCount.jar
# 结果:不需要手动释放资源,yarn 端口:8088,发现 jobmanager 已释放资源
- 3- Application 模式
- 特点:在 per-job 模式基础上,将 main 函数在集群中,而不是客户端
- 优点:每个作业单独启动集群,资源隔离性好,JM 负载均衡,节省 CPU 周期,和节省下载依赖项所需要带宽
# hdfs 上事先上传好 jar 包,包括用户 jar 包,指定结果输出路径
bin/flink run-application -t yarn-application \
-Djobmanager.memory.process.size=1024m \
-Dtaskmanager.memory.process.size=1024m \
-Dtaskmanager.numberOfTaskSlots=2 \
-Dparallelism.default=2 \
-Dyarn.provided.lib.dirs="hdfs://node1:8020/flink/lib;hdfs://node1:8020/flink/plugin" \
-Dyarn.application.name="batchWordCount" \
hdfs://node1:8020/flink/user-libs/WordCount.jar --output hdfs://node1:8020/wordcount/output_54
# 结果:hdfs 对应目录下出现新的小文件
- 注意:
如果使用的是 flink on yarn 方式,想切换回 standalone 模式
(需要单独启动 bin/cluster.sh)的话,这时候 Flink 的 8081 Web UI 可以登录,需要删除隐藏文件:【/tmp/.yarn-properties-root】
--------------------- node1 ----------------
86583 Jps
85963 StandaloneSessionClusterEntrypoint
86446 TaskManagerRunner
--------------------- node2 ----------------
44099 Jps
43819 TaskManagerRunner
--------------------- node3 ----------------
29461 TaskManagerRunner
29678 Jps
原因:因为默认查找当前 yarn 集群中已有的 yarn-session 信息中的jobmanager
例子1:如果是分离模式运行的YARN JOB后,其运行完成会自动删除这个文件;
例子2:会话模式的话,如果是kill掉任务,其不会执行自动删除这个文件的步骤,所以需要我们手动删除这个文件。
3. Flink 入门案例
(1) java的pom文件
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>cn.itcast</groupId>
<artifactId>flinkbase_pro</artifactId>
<version>1.0-SNAPSHOT</version>
<!-- 指定仓库位置,依次为aliyun、apache和cloudera仓库 -->
<repositories>
<!-- <repository>-->
<!-- <id>aliyun</id>-->
<!-- <url>http://maven.aliyun.com/nexus/content/groups/public/</url>-->
<!-- </repository>-->
<repository>
<id>apache</id>
<url>https://repository.apache.org/content/repositories/snapshots/</url>
</repository>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<properties>
<encoding>UTF-8</encoding>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<java.version>1.8</java.version>
<scala.version>2.11</scala.version>
<flink.version>1.13.1</flink.version>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
</properties>
<dependencies>
<!--依赖Scala语言-->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.12</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-scala_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<!-- web ui的依赖 -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-runtime-web_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<!-- Apache Flink 的依赖 -->
<!-- 这些依赖项,不应该打包到JAR文件中. -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<!-- 用于通过自定义功能,格式等扩展表生态系统的通用模块-->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-api-scala-bridge_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-api-java-bridge_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<!-- blink执行计划,1.11+默认的-->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-planner-blink_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-common</artifactId>
<version>${flink.version}</version>
</dependency>
<!--<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-cep_2.11</artifactId>
<version>${flink.version}</version>
</dependency>-->
<!-- flink连接器-->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-sql-connector-kafka_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-jdbc_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-csv</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-json</artifactId>
<version>${flink.version}</version>
</dependency>
<!-- <dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-filesystem_2.11</artifactId>
<version>${flink.version}</version>
</dependency>-->
<!--<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-jdbc_2.11</artifactId>
<version>${flink.version}</version>
</dependency>-->
<!--<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-parquet_2.11</artifactId>
<version>${flink.version}</version>
</dependency>-->
<!--<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.9.2</version>
</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-avro</artifactId>
<version>1.10.0</version>
</dependency>-->
<dependency>
<groupId>org.apache.bahir</groupId>
<artifactId>flink-connector-redis_2.11</artifactId>
<version>1.0</version>
<exclusions>
<exclusion>
<artifactId>flink-streaming-java_2.11</artifactId>
<groupId>org.apache.flink</groupId>
</exclusion>
<exclusion>
<artifactId>flink-runtime_2.11</artifactId>
<groupId>org.apache.flink</groupId>
</exclusion>
<exclusion>
<artifactId>flink-core</artifactId>
<groupId>org.apache.flink</groupId>
</exclusion>
<exclusion>
<artifactId>flink-java</artifactId>
<groupId>org.apache.flink</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-hive_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-metastore</artifactId>
<version>2.1.0</version>
<exclusions>
<exclusion>
<artifactId>hadoop-hdfs</artifactId>
<groupId>org.apache.hadoop</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>2.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2-uber</artifactId>
<version>2.7.5-10.0</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>2.1.0</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.38</version>
<!--<version>8.0.20</version>-->
</dependency>
<!-- 高性能异步组件:Vertx-->
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-core</artifactId>
<version>3.9.0</version>
</dependency>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-jdbc-client</artifactId>
<version>3.9.0</version>
</dependency>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-redis-client</artifactId>
<version>3.9.0</version>
</dependency>
<!-- 日志 -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.7</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.44</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.2</version>
<scope>provided</scope>
</dependency>
<!-- 参考:https://blog.csdn.net/f641385712/article/details/84109098-->
<!--<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-collections4</artifactId>
<version>4.4</version>
</dependency>-->
<!--<dependency>
<groupId>org.apache.thrift</groupId>
<artifactId>libfb303</artifactId>
<version>0.9.3</version>
<type>pom</type>
<scope>provided</scope>
</dependency>-->
<!--<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>28.2-jre</version>
</dependency>-->
</dependencies>
<build>
<sourceDirectory>src/main/java</sourceDirectory>
<plugins>
<!-- 编译插件 -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<!--<encoding>${project.build.sourceEncoding}</encoding>-->
</configuration>
</plugin>
<!-- 打包插件(会包含所有依赖) -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<!--
zip -d learn_spark.jar META-INF/*.RSA META-INF/*.DSA META-INF/*.SF -->
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<!-- 设置jar包的入口类(可选) -->
<mainClass></mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
(2) 批处理词频统计
- 批环境的获取:ExecutionEnvironment
- 文本数据获取:readTextFile()
- 空格拆分(flatMap)— 单词计数(map)— 单词分组(groupby)—词频统计(sum)
package cn.itcast.day01;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.api.java.operators.*;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.util.Collector;
/**
* @author lql
* @time 2024-02-11 21:31:51
* @description TODO:批处理词频统计
*/
public class BatchWordCount {
public static void main(String[] args) throws Exception {
// 获取批处理运行环境
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
// 指定读取文件路径,获取数据
DataSource<String> inputDataSet = env.readTextFile("D:\\IDEA_Project\\BigData_Java\\flinkbase_pro\\data\\input\\wordcount.txt");
// 对获取到的数据进行空格拆分
FlatMapOperator<String, String> words = inputDataSet.flatMap(new FlatMapFunction<String, String>() {
@Override
public void flatMap(String line, Collector<String> out) throws Exception {
// 切分
String[] words = line.split(" ");
for (String word : words) {
out.collect(word);
}
}
});
// 对拆分后的单词,每个单词记一次数
MapOperator<String, Tuple2<String, Integer>> wordAndOne = words.map(new MapFunction<String, Tuple2<String, Integer>>() {
@Override
public Tuple2<String, Integer> map(String word) throws Exception {
return Tuple2.of(word, 1);
}
});
// 对拆分后的单词进行分组
UnsortedGrouping<Tuple2<String, Integer>> groupedStream = wordAndOne.groupBy(0);
// 根据单词的次数进行聚合
AggregateOperator<Tuple2<String, Integer>> summed = groupedStream.sum(1);
// 打印输出
summed.print();
// 启动执行
}
}
(3) 流处理词频统计
- 流环境的获取:StreamExecutionEnvironment
- socket流数据:socketTextStream
- 空格拆分(flatMap)— 单词计数(map)— 单词分组(keyby)—词频统计(sum)
package cn.itcast.day01;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;
/**
* @author lql
* @time 2024-02-11 22:10:36
* @description TODO:编写Flink程序,接收socket的单词数据,并以空格进行单词拆分打印
*/
public class StreamWordCount {
public static void main(String[] args) throws Exception {
//获取流处理运行环境
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
//构建socket流数据源,并指定IP地址和端口号
DataStreamSource<String> lines = env.socketTextStream("node1", 9999);
//对接收到的数据进行空格拆分
SingleOutputStreamOperator<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
@Override
public void flatMap(String line, Collector<String> out) throws Exception {
String[] words = line.split(" ");
for (String word : words) {
out.collect(word);
}
}
});
//对拆分后的单词,每个单词记一次数
SingleOutputStreamOperator<Tuple2<String, Integer>> wordAndOne = words.map(new MapFunction<String, Tuple2<String, Integer>>() {
@Override
public Tuple2<String, Integer> map(String word) throws Exception {
return Tuple2.of(word, 1);
}
});
//对拆分后的单词进行分组
KeyedStream<Tuple2<String, Integer>, String> grouped = wordAndOne.keyBy(t -> t.f0);
//根据单词的次数进行聚合
SingleOutputStreamOperator<Tuple2<String, Integer>> summed = grouped.sum(1);
//打印输出
summed.print();
//启动执行
env.execute();
//在Linux中,使用nc -lk 端口号监听端口,并发送单词
//安装nc: yum install -y nc
}
}
4. Flink 提交部署
-
IDEA 中 maven 的程序 jar 包打包:clean -> test -> clean -> package
-
以 UI 方式提交:需要切换为
standlone
模式才能出现 web UI 提交页面(如果报错需要删除上述的隐藏文件)
# 启动 jobmanager 和 taskmanager
bin/start-cluster.sh
- 以命令方式提交:web 端口随机,转瞬即逝
# 示例:per- job 模式命令提交,其他模式也可以命令提交
./bin/flink run \
-Dexecution.runtime-mode=BATCH \
-m yarn-cluster \
-yjm 1024 \
-ytm 1024 \
-c cn.itcast.day01.BatchWordCountToYarn ./original-flinkbase_pro-1.0-SNAPSHOT.jar \
--output hdfs://node1:8020/wordcount/output_50
5. Flink 任务调度原理
执行图原理:
-
Flink 执行 Executor 会根据代码生成 DAG 数据流图
-
Flink 执行图分为四层:
StreamGraph(代码结构) —> JobGraph(优化合并相关任务) —> ExecutionGraph(并行度拆分) —> 物理执行图