FLink学习笔记：02-Flink简单应用WordCount

已于 2022-08-03 09:27:58 修改
阅读量3.8k 收藏 2
点赞数 3
分类专栏： 大数据 FLink 文章标签： flink
于 2022-04-24 18:10:26 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wangzhongyudie/article/details/124388816
版权

文章目录

idea新建maven项目，配置scala环境

File–>Project Structure -->Modules，添加scala依赖库
在这里插入图片描述

配置pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.yxh.hjt.hw</groupId>
    <artifactId>FlinkDemo</artifactId>
    <version>1.0-SNAPSHOT</version>
    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-scala -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-scala_2.12</artifactId>
            <version>1.14.4</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-scala -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-scala_2.12</artifactId>
            <version>1.14.4</version>
        </dependency>
        
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-clients -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_2.12</artifactId>
            <version>1.14.4</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-simple -->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.7.25</version>
            <!--<scope>test</scope>-->
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.4.6</version>
                <executions>
                    <execution> <!-- 声明绑定到 maven 的 compile 阶段 -->
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>3.0.0</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Flink 批处理

import org.apache.flink.api.scala._

object WordCount {

  def main(args: Array[String]): Unit = {

    //创建环境
    val env = ExecutionEnvironment.getExecutionEnvironment

    //input
    val inputPath:String = "D:\\LearnWorkSpace\\FlinkDemo\\src\\main\\resources\\Data\\hello.txt"
    val inputDs:DataSet[String] = env.readTextFile(inputPath)

    //Transmation
    val wordCountDs: AggregateDataSet[(String,Int)]=inputDs.flatMap((_: String).split(" ")).map((_,1)).groupBy(0).sum(1)

    //oupput sink
    wordCountDs.print()
    wordCountDs.writeAsText("flink-ret")
    env.execute("text WordCount")
  }

}

运行时如果出现的报错:

[main] INFO org.apache.flink.api.java.utils.PlanGenerator - The job has 0 registered types and 0 default Kryo serializers
Exception in thread "main" java.lang.IllegalStateException: No ExecutorFactory found to execute the application.
	at org.apache.flink.core.execution.DefaultExecutorServiceLoader.getExecutorFactory(DefaultExecutorServiceLoader.java:88)
	at org.apache.flink.api.java.ExecutionEnvironment.executeAsync(ExecutionEnvironment.java:1043)
	at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:958)
	at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:942)
	at org.apache.flink.api.java.DataSet.collect(DataSet.java:417)
	at org.apache.flink.api.java.DataSet.print(DataSet.java:1748)
	at org.apache.flink.api.scala.DataSet.print(DataSet.scala:1864)
	at WordCount$.main(WordCount.scala:20)
	at WordCount.main(WordCount.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at com.intellij.rt.execution.application.AppMainV2.main(AppMainV2.java:131)

则需要在pom.xml中添加依赖：

 <!--https://mvnrepository.com/artifact/org.apache.flink/flink-clients -->
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-clients_2.12</artifactId>
    <version>1.14.4</version>
</dependency>

如果报错

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

则需要在pom.xml中添加slf4j-simple依赖

<!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-simple -->
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-simple</artifactId>
    <version>1.7.25</version>
    <!--<scope>test</scope>-->
</dependency>

Flink流处理

scala代码

import org.apache.flink.streaming.api.scala._

object WordCountFlow {

  def main(args: Array[String]): Unit = {
    // 创建流处理的执行环境
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    //    env.setParallelism(8)
    //    env.disableOperatorChaining()

    // 从外部命令中提取参数，作为socket主机名和端口号
    val host: String = "192.168.0.47"
    val port: Int = 8888

    // 接收一个socket文本流
    val inputDataStream: DataStream[String] = env.socketTextStream(host, port)
    // 进行转化处理统计
    val resultDataStream: DataStream[(String, Int)] = inputDataStream
      .flatMap(_.split(" "))
      .filter(_.nonEmpty)
      .map((_, 1))
      .keyBy(0)
      .sum(1)

    resultDataStream.print().setParallelism(1)

    // 启动任务执行
    env.execute("stream word count")
  }
}

在Linux上可以使用nc -lk 8888模拟一个Socket服务。

可能遇到的问题

  • module java.base does not “opens java.lang” to unnamed module @482f8f11
Exception in thread "main" java.lang.reflect.InaccessibleObjectException: Unable to make field private final byte[] java.lang.String.value accessible: module java.base does not "opens java.lang" to unnamed module @482f8f11
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:357)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:177)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:171)
	at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:104)
	at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:126)
	at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:71)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:1821)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.java:1584)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.java:1529)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.socketTextStream(StreamExecutionEnvironment.java:1333)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.socketTextStream(StreamExecutionEnvironment.java:1373)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.socketTextStream(StreamExecutionEnvironment.java:1390)
	at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.socketTextStream(StreamExecutionEnvironment.scala:608)
	at WordCountFlow$.main(WordCountFlow.scala:16)
	at WordCountFlow.main(WordCountFlow.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at com.intellij.rt.execution.application.AppMainV2.main(AppMainV2.java:131)

解决方法:VM options中添加:–add-opens java.base/java.lang=ALL-UNNAMED

  • 问题二:cannot access class sun.net.util.IPAddressUtil (in module java.base) because module java.base does not export sun.net.util to unnamed module @0x4501b7af
Exception in thread "main" java.lang.IllegalAccessError: class org.apache.flink.util.NetUtils (in unnamed module @0x4501b7af) cannot access class sun.net.util.IPAddressUtil (in module java.base) because module java.base does not export sun.net.util to unnamed module @0x4501b7af
	at org.apache.flink.util.NetUtils.unresolvedHostToNormalizedString(NetUtils.java:167)
	at org.apache.flink.util.NetUtils.unresolvedHostAndPortToNormalizedString(NetUtils.java:195)
	at org.apache.flink.runtime.clusterframework.BootstrapTools.startActorSystem(BootstrapTools.java:254)
	at org.apache.flink.runtime.clusterframework.BootstrapTools.startActorSystem(BootstrapTools.java:164)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils.createRpcService(AkkaRpcServiceUtils.java:126)
	at org.apache.flink.runtime.metrics.util.MetricUtils.startMetricsRpcService(MetricUtils.java:139)
	at org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:286)
	at org.apache.flink.client.program.PerJobMiniClusterFactory.submitJob(PerJobMiniClusterFactory.java:79)
	at org.apache.flink.client.deployment.executors.LocalExecutor.execute(LocalExecutor.java:81)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1733)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1634)
	at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:74)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1620)
	at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:678)
	at WordCountFlow$.main(WordCountFlow.scala:28)
	at WordCountFlow.main(WordCountFlow.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at com.intellij.rt.execution.application.AppMainV2.main(AppMainV2.java:131)

解决方法：VM options中添加参数:
–add-opens java.base/sun.net.util=ALL-UNNAMED

提交作业

打成jar包后，将jar上传到flink,可以通过命令行或者web页面的方式提交作业。

命令行模式提交作业：

flink -c 入口类 -p 并行度 jar包的路径 程序中的外部参数（eg. --host=localhost）

[root@k8s-node3 myLearning]# flink run -c com.hjt.yxh.hw.WordCountFlow -p 2 /home/software/flink-1.14.4/examples/myLearning/FlinkDemo-1.0-SNAPSHOT.jar
  • 取消作业：flink cancel jobid
  • flink list -a 查看所有的作业

WEB端提交作业

作业提交

wangzhongyudie
关注
  • 3
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Flink与JAVA不匹配导致无法启动集群
qq_42247247的博客
04-06 3万+
flink版本：1.8.1 java版本：java se 16 日志文档中错误： org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Could not start cluster entrypoint StandaloneSessionClusterEntrypoint. 原因： Caused by: java.lang.IllegalAccessError: class org.apache.flink.util.NetUtils (in un
cannot access class jdk.internal.misc.Unsafe（java9的模块概念）
thewindkee的博客
03-20 5104
现象 java11的本地环境，服务中使用到netty。启动后，服务无法访问，查看日志，有提示： cannot access class jdk.internal.misc.Unsafe ... because the module java.base not export jdk.internal.misc to unnamed module 解决办法 stackoverflow给了原因以及解决办法。 --add-opens java.base/jdk.internal.misc=ALL-UNNAMED
Flink 1.19.1 standalone 集群模式部署及配置
最新发布
SunTecTec
06-20 673
config.yaml: jobmanager address 统一使用 dev001，bind-port 统一改成 0.0.0.0，taskmanager address 分别更改为dev所在host。flink 1.19起 conf/flink-conf.yaml 更改为新的 conf/config.yaml。standalone集群: dev001、dev002、dev003。
运行flink-runtime_2.11_1.11.2报错：java.base does not export sun.net.util to unnamed module @0x1a407d53
u013771019的专栏
05-10 333
【代码】运行flink-runtime_2.11_1.11.2报错：java.base does not export sun.net.util to unnamed module @0x1a407d53。
java.lang.IllegalAccessError
qq_42807952的博客
07-23 2946
在一次回顾以前写的东西时候，运行之后报了这么一次错误。3.不在报错，成功解决!2.换成JDK8版本的。
JDK17遇到报错 module java.base does not “opens java.util“ to unnamed module 问题解决
热门推荐
cs779193543的博客
11-23 11万+
在Java 9及以上版本运行应用程序时，在各种情况下都会发生此异常。 详细可以参考module java.base does not "opens java.lang" to unnamed module - 滔天蟹 - 博客园 又由于是框架的问题，所以我们能采取的办法是启动JVM加入如下参数: # --add-opens has the following syntax: {A}/{package}={B} --add-opens java.base/java.util=ALL-UNNAMED
flink-table-runtime-blink_2.11-1.10.0-API文档-中文版.zip
04-23
赠送jar包：flink-table-runtime-blink_2.11-1.10.0.jar； 赠送原API文档：flink-table-runtime-blink_2.11-1.10.0-javadoc.jar； 赠送源代码：flink-table-runtime-blink_2.11-1.10.0-sources.jar； 赠送Maven依赖...
flink-connector-kafka-0.10-2.11-1.10.0-API文档-中文版.zip
04-23
赠送jar包：flink-connector-kafka-0.10_2.11-1.10.0.jar； 赠送原API文档：flink-connector-kafka-0.10_2.11-1.10.0-javadoc.jar； 赠送源代码：flink-connector-kafka-0.10_2.11-1.10.0-sources.jar； 赠送Maven...
flink-table-planner-2.12-1.14.3-API文档-中文版.zip
04-07
赠送jar包：flink-table-planner_2.12-1.14.3.jar 赠送原API文档：flink-table-planner_2.12-1.14.3-javadoc.jar 赠送源代码：flink-table-planner_2.12-1.14.3-sources.jar 包含翻译后的API文档：flink-table-...
flink-table-planner-blink_2.11-1.13.2-API文档-中文版.zip
06-05
赠送jar包：flink-table-planner-blink_2.11-1.13.2.jar； 赠送原API文档：flink-table-planner-blink_2.11-1.13.2-javadoc.jar； 赠送源代码：flink-table-planner-blink_2.11-1.13.2-sources.jar； 赠送Maven依赖...
flink-table-planner-blink_2.11-1.13.2-API文档-中英对照版.zip
06-05
赠送jar包：flink-table-planner-blink_2.11-1.13.2.jar； 赠送原API文档：flink-table-planner-blink_2.11-1.13.2-javadoc.jar； 赠送源代码：flink-table-planner-blink_2.11-1.13.2-sources.jar； 赠送Maven依赖...
hadoop源码导入到eclipse后程序报错，sun.net.util.IPAddressUtil这个类找不到与之应的jar包
panguoyuan的专栏
04-25 3798
hadoop源码导入到eclipse后程序报错，sun.net.util.IPAddressUtil这个类找不到与之应的jar包 在eclipse>>window-preference->java->complier->errors/warning->deprecated and restricted API把 Forbidden reference 的Error改成warning
flink 运维中遇到的问题
h952520296的博客
03-03 8445
1. Java.lang.IllegalStateException: No Executor found. Please make sure to export the HADOOP_CLASSPATH environment variable or have hadoop in your classpath. For more information refer to the "Deployment" section of the official Apache Flink documentation.
Flink学习之旅：（二）构建Flink demo工程并提交到集群执行
小健的博客
10-18 905
在idea中创建一个 名为 MyFlinkFirst 工程。
解决Flink报错Exception in thread “main“ org.apache.flink.api.common.functions.InvalidTypesException
云来山更佳，云去山如画
03-16 6246
2种方式解决Flink报错Exception in thread "main" org.apache.flink.api.common.functions.InvalidTypesException
flink启动报错：java.lang.NoClassDefFoundError: scala/collection/immutable/StringOps 解决方法
mask哥
10-05 4519
Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/immutable/StringOps at org.apache.flink.runtime.akka.AkkaUtils$.testDispatcherConfig(AkkaUtils.scala:374) at org.apache.flink.runtime.akka.AkkaUtils.testDispatcherConfig(Ak...
Flink多流共用一个方法，状态编程报错
Johnson8702的博客
08-03 1078
Flink多流共用一个方法，状态编程报错
Flink : Could not resolve substitution to a value: ${akka.stream.materializer}
九师兄
07-04 2581
flink代码从1.9升级到1.10 但是报错， 找不到类 然后我加入依赖 然后运行错误如下 然后采用网上方案无法解决参考：报错：Flink Could not resolve substitution to a value: ${akka.stream.materializer}参考：添加链接描述 后面经过验证，是flink 1.9 的时候，加不加这段xml 都不会报错，但是一旦修改版本就会报错，猜测和flink的版本有关。那么要看看flink 1.10版本，这个为什么会出现这个问题了。然后我就去看看fl
flink入门1-Flink使用socketTextStream对接netcat完成入门demo
summon的博客
06-07 2891
pom.xml配置文件如下： 2、编写测试类，该类的功能是将所有的字母变成大写 3、执行netcat命令打开端口 4、运行结果如下
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值