搭建maven 工程FlinkTutorial
1、准备工作
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.atguigu</groupId>
<artifactId>FlinkTutorial</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-scala_2.12</artifactId>
<version>1.10.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-scala -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala_2.12</artifactId>
<version>1.10.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<!-- 该插件用于将Scala代码编译成class文件 -->
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>4.4.0</version>
<executions>
<execution>
<!-- 声明绑定到maven的compile阶段 -->
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.3.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
2、批处理 word count
import org.apache.flink.api.scala.ExecutionEnvironment
import org.apache.flink.api.scala._
// 批处理的word count
object WordCount {
def main(args: Array[String]): Unit = {
// 创建一个批处理的执行环境
val env: ExecutionEnvironment = ExecutionEnvironment.getExecutionEnvironment
// 从文件中读取数据
val inputPath: String = "D:\\Flink基础教程\\理论阶段代码\\src\\main\\resources\\hello.txt"
val inputDataSet: DataSet[String] = env.readTextFile(inputPath)
// 对数据进行转换处理统计,先分词,再按照word进行分组,最后进行聚合统计
val resultDataSet: DataSet[(String, Int)] = inputDataSet
.flatMap(_.split(" "))
.map((_, 1))
.groupBy(0) // 以第一个元素作为key,进行分组
.sum(1) // 对所有数据的第二个元素求和
// 打印输出
resultDataSet.print()
}
}
3、流处理 word count
import org.apache.flink.api.java.utils.ParameterTool
import org.apache.flink.streaming.api.scala._
// 流处理word count
object StreamWordCount {
def main(args: Array[String]): Unit = {
// 创建流处理的执行环境
val env = StreamExecutionEnvironment.getExecutionEnvironment
// env.setParallelism(8)
// 从外部命令中提取参数,作为socket主机名和端口号
val paramTool: ParameterTool = ParameterTool.fromArgs(args)
val host: String = paramTool.get("host")
val port: Int = paramTool.getInt("port")
// 接收一个socket文本流
val inputDataStream: DataStream[String] = env.socketTextStream(host, port)
// 进行转化处理统计
val resultDataStream: DataStream[(String, Int)] = inputDataStream
.flatMap(_.split(" "))
.filter(_.nonEmpty)
.map((_, 1))
.keyBy(0)
.sum(1)
resultDataStream.print()
// .setParallelism(1)
// 启动任务执行
env.execute("stream word count")
}
}
先启动nc,再启动程序,最后输入内容
关于并行度
默认并行度=本机核数,也可通过env.setParallelism(8)设置全局并行度。 还可以设置每一步骤的并行度:
map((_, 1)).setParallelism(8)
keyBy(0).setParallelism(8)
sum(1).setParallelism(8)
resultDataStream.print().setParallelism(1)
当最后结果要输入到一个文件中时通常将写入步骤的并行度设置为1