Flink概念
连接:https://ashiamd.github.io/docsify-notes/#/study/BigData/Flink/尚硅谷Flink入门到实战-学习笔记
Flink是为分布式,高性能,随时可用以及准确的流处理应用程序打造的开源处理框架
特点:
- 事件驱动
- 流处理
Flink快速上手
pom依赖:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>UserBehaviorAnalysis</artifactId>
<packaging>pom</packaging>
<version>1.0-SNAPSHOT</version>
<modules>
<module>HotItemsAnalysis</module>
</modules>
<properties>
<flink.version>1.10.1</flink.version>
<scala.binary.version>2.12</scala.binary.version>
<kafka.version>2.2.0</kafka.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
</plugins>
</build>
</project>
代码实现:
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.java.DataSet;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.util.Collector;
/**
* @author : Ashiamd email: ashiamd@foxmail.com
* @date : 2021/1/29 10:46 PM
* 批处理 wordcount
*/
public class WordCount {
public static void main(String[] args) throws Exception {
// 创建执行环境
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
// 从文件中读取数据
String inputPath = "/tmp/Flink_Tutorial/src/main/resources/hello.txt";
DataSet<String> inputDataSet = env.readTextFile(inputPath);
// 对数据集进行处理,按空格分词展开,转换成(word, 1)二元组进行统计
// 按照第一个位置的word分组
// 按照第二个位置上的数据求和
DataSet<Tuple2<String, Integer>> resultSet = inputDataSet.flatMap(new MyFlatMapper())
.groupBy(0)
.sum(1);
resultSet.print();
}
// 自定义类,实现FlatMapFunction接口
public static class MyFlatMapper implements FlatMapFunction<String, Tuple2<String, Integer>> {
@Override
public void flatMap(String s, Collector<Tuple2<String, Integer>> out) throws Exception {
// 按空格分词
String[] words = s.split(" ");
// 遍历所有word,包成二元组输出
for (String str : words) {
out.collect(new Tuple2<>(str, 1));
}
}
}
}
输出:
(scala,1)
(flink,1)
(world,1)
(hello,4)
(and,1)
(fine,1)
(how,1)
(spark,1)
(you,3)
(are,1)
(thank,1)
流处理:
package wc;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.client.program.StreamContextEnvironment;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
/**
* @author : Ashiamd email: ashiamd@foxmail.com
* @date : 2021/1/29 11:13 PM
*/
public class StreamWordCount {
public static void main(String[] args) throws Exception {
// 创建流处理执行环境
StreamExecutionEnvironment env = StreamContextEnvironment.getExecutionEnvironment();
// 设置并行度,默认值 = 当前计算机的CPU逻辑核数(设置成1即单线程处理)
// env.setMaxParallelism(32);
// 从文件中读取数据
String inputPath = "/tmp/Flink_Tutorial/src/main/resources/hello.txt";
DataStream<String> inputDataStream = env.readTextFile(inputPath);
// 基于数据流进行转换计算
DataStream<Tuple2<String,Integer>> resultStream = inputDataStream.flatMap(new WordCount.MyFlatMapper())
.keyBy(item->item.f0)
.sum(1);
resultStream.print();
// 执行任务
env.execute();
}
}
也可以用socket代替文件输入
在控制台中使用nc -lk 7777来开启该端口
Flink执行流程
- environment
ExecutionEnvironment env = ExecutionEnvironment.*getExecutionEnvironment*();
StreamExecutionEnvironment env = StreamExecutionEnvironment.*getExecutionEnvironment*();
- source
2.1 从集合读取数据
public class SourceTest1_Collection {
public static void main(String[] args) throws Exception{
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
// 1.Source:从集合读取数据
DataStream<SensorReading> sensorDataStream = env.fromCollection(
Arrays.asList(
new SensorReading("sensor_1", 1547718199L, 35.8),
new SensorReading("sensor_6", 1547718201L, 15.4),
new SensorReading("sensor_7", 1547718202L, 6.7),
new SensorReading("sensor_10", 1547718205L, 38.1)
)
);
// 2.打印
sensorDataStream.print();
// 3.执行
env.execute();
}
}
2.2 从文件读取数据
DataStream<String> dataStream = env.readTextFile("YOUR_FILE_PATH ");
- Transform
3.1 map
DataStream<Integer> mapStram = dataStream.map(new MapFunction<String, Integer>() {
public Integer map(String value) throws Exception {
return value.length();
}
});
3.2 flatmap
DataStream<String> flatMapStream = dataStream.flatMap(new FlatMapFunction<String,
String>() {
public void flatMap(String value, Collector<String> out) throws Exception {
String[] fields = value.split(",");
for( String field: fields )
out.collect(field);
}
});
3.3 filter
DataStream<Interger> filterStream = dataStream.filter(new FilterFunction<String>()
{
public boolean filter(String value) throws Exception {
return value == 1;
}
});
3.4 KeyBy
可以将流进行分组
3.5 滚动聚合算子
sum, min, max, minby, maxby
3.6 聚合reduce
DataStream<String> inputStream = env.readTextFile("sensor.txt");
// 转换成 SensorReading 类型
DataStream<SensorReading> dataStream = inputStream.map(new
MapFunction<String, SensorReading>() {
// 分组
public SensorReading map(String value) throws Exception {
String[] fileds = value.split(",");
return new SensorReading(fileds[0], new Long(fileds[1]), new
Double(fileds[2]));
});
}
KeyedStream<SensorReading, Tuple> keyedStream = dataStream.keyBy("id");
// reduce 聚合,取最小的温度值,并输出当前的时间戳
DataStream<SensorReading> reduceStream = keyedStream.reduce(new
ReduceFunction<SensorReading>() {
@Override
public SensorReading reduce(SensorReading value1, SensorReading value2)
throws Exception {
return new SensorReading(
value1.getId(),
value2.getTimestamp(),
Math.min(value1.getTemperature(), value2.getTemperature()));
}
});
3.7 split拆分
将一个流拆分成两个流
SplitStream<SensorReading> splitStream = dataStream.split(new
OutputSelector<SensorReading>() {
@Override
public Iterable<String> select(SensorReading value) {
return (value.getTemperature() > 30) ? Collections.singletonList("high") :
Collections.singletonList("low");
}
});
DataStream<SensorReading> highTempStream = splitStream.select("high");
DataStream<SensorReading> lowTempStream = splitStream.select("low");
DataStream<SensorReading> allTempStream = splitStream.select("high", "low");
3.8 Connect合并
3.9 Union合并
窗口
- 滚动窗口
- 滑动窗口
- 窗口函数
时间语义和watermark
三种时间语义:
- Event Time:事件的创建时间
- Ingest Time:数据进入Flink时间
- Process Time:执行时的本地系统时间
一般常用的为Processing Time
watermark:
用来处理乱序事件,能够让事件延时;