Docker搭建Flink1.17.0 使用Scala编写词频统计Demo

山塘小鱼儿

已于 2023-04-07 14:35:32 修改

阅读量851

点赞数 2

文章标签： scala docker java

于 2023-04-07 14:29:30 首次发布

本文链接：https://blog.csdn.net/airyearth/article/details/130011204

版权

文章详细介绍了如何在Docker环境下安装ApacheFlink1.17.0，包括拉取镜像、创建网络、启动JobManager和TaskManager。接着，作者展示了如何设置和复制配置文件，以及使用Scala开发Flink流处理项目，包含项目的Maven配置和WordCountStream示例代码。最后，文章提到了通过nc命令发送数据到Flink进行实时处理。

摘要由CSDN通过智能技术生成

docker安装Flink1.17.0

拉取镜像

docker pull flink: 1.17.0-scala_2.12-java8

# 创建 docker 网络，方便 JobManager 和 TaskManager 内部访问
 docker network create flink-network

# 创建 JobManager 
 docker run \
  -itd \
  --name=jobmanager \
  --publish 8081:8081 \
  --network flink-network \
  --env FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager" \
  flink:1.17.0-scala_2.12-java8 jobmanager 
  
# 创建 TaskManager 
 docker run \
  -itd \
  --name=taskmanager \
  --network flink-network \
  --env FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager" \
  flink:1.17.0-scala_2.12-java8 taskmanager

启动成功

web页面

copy 配置文件

# jobmanager 容器
 docker cp jobmanager:/opt/flink/conf ./JobManager/
# taskmanager 容器
docker cp taskmanager:/opt/flink/conf ./TaskManager/

启动容器挂载配置文件

# 启动 jobmanager   
docker run -itd  -v /home/uu/JobManager/conf/:/opt/flink/conf/ --name=jobmanager --publish 8081:8081 --env FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager" --network flink-network flink:1.17.0-scala_2.12-java8 jobmanager
# 启动 taskmanager   
docker run -itd  -v /home/uu/TaskManager/conf/:/opt/flink/conf/ --name=taskmanager --network flink-network --env FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"  flink:1.17.0-scala_2.12-java8 taskmanager

编写scala项目代码

IDEA社区版搭建scala环境请见：IDEA社区版安装scala，查看twitter推荐算法the-algorithm_山塘小鱼儿的博客-CSDN博客

项目目录结构如下

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>myscala</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-scala -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-scala_2.12</artifactId>
            <version>1.17.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-scala -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-scala_2.12</artifactId>
            <version>1.17.0</version>
            <scope>provided</scope>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-clients -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients</artifactId>
            <version>1.17.0</version>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <!-- 该插件用于将 Scala 代码编译成 class 文件 -->
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.4.6</version>
                <executions>
                    <execution>
                        <!-- 声明绑定到 maven 的 compile 阶段 -->
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

WordCountStream

package com.my.scala
import org.apache.flink.streaming.api.scala._
import org.apache.flink.api.scala.ExecutionEnvironment
object WordCountStream {
  def main(args: Array[String]): Unit = {
    //流处理的执行环境
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    //监听端口,socket文本流
    val inputDataStream = env.socketTextStream("ip", 7196)
    //数据转换处理
    val resultDataStream = inputDataStream.flatMap(_.split(" "))
      .filter(_.nonEmpty)
      .map((_, 1))
      .keyBy(0)
      .sum(1)
    resultDataStream.print()
    //启动程序
    env.execute("wordcount")
  }
}

服务开启发送字符服务