目的:
-
学习Flink的基本使用方法
-
掌握在一般使用中需要注意的事项
手把手的过程中会讲解各种问题的定位方法,相对啰嗦,内容类似结对编程。
大家遇到什么问题可以在评论中说一下,我来完善文档
Flink专辑的各篇文章链接:
步骤:
准备工作
1. 本地环境
-
操作系统: macOS 10.15.2
-
JDK: 1.8.0_191
-
Docker: 2.2.0.0
-
IntelliJ IDEA: 2019.3
-
git version: 2.20.1
2. 安装本地docker环境
➜ operations-playground git:(release-1.9) ✗ docker-compose up -d
Creating network "operations-playground_default" with the default driver
Creating volume "operations-playground_flink-checkpoints-directory" with default driver
Creating operations-playground_zookeeper_1 ... done
Creating operations-playground_kafka_1 ... done
Creating operations-playground_jobmanager_1 ... done
Creating operations-playground_clickevent-generator_1 ... done
Creating operations-playground_client_1 ... done
Creating operations-playground_taskmanager_1 ... done
➜ operations-playground git:(release-1.9) ✗
启动正常
3. 创建本地项目
我选择的是Gradle Java,使用如下命令完成,全部默认选项
bash -c "$(curl https://flink.apache.org/q/gradle-quickstart.sh)" -- 1.9.0 2.11
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 8375 100 8375 0 0 5836 0 0:00:01 0:00:01 --:--:-- 5832
This script creates a Flink project using Java and Gradle.
Project name (quickstart):
Organization (org.myorg.quickstart):
Version (0.1-SNAPSHOT):
Flink version (1.9.0):
Scala version (2.11):
-----------------------------------------------
Project Name: quickstart
Organization: org.myorg.quickstart
Version: 0.1-SNAPSHOT
Scala binary version: 2.11
Flink version: 1.9.0
-----------------------------------------------
Create Project? (Y/n): Y
Creating Flink project under quickstart
Downloading QuickStart files from https://repository.apache.org/content/repositories/releases/org/apache/flink/flink-quickstart-java/1.9.0/flink-quickstart-java-1.9.0.jar
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 12566 100 12566 0 0 3929 0 0:00:03 0:00:03 --:--:-- 3929
Extracting QuickStart files
Done.
当前项目中内容是盘中文件word-count-project.zip
链接:https://pan.baidu.com/s/19PQzxWQsQsf8E7v-a-sRUA 密码:9uuq
快乐的时间总是过得特别快,下面开始解决各种问题
运行无参数WordCount
本地运行
用IDE打开刚才创建的quickstart项目,编译报错
-
Cannot add task 'wrapper' as a task with that name already exists.
解决办法,删除build.gradle中
task wrapper(type: Wrapper) {
gradleVersion = '3.1'
}
默认提供了两个Job:BatchJob、StreamingJob,但是这两个Job都是空的,不能执行,执行会报错
Execution failed for task ':BatchJob.main()'.
> Process 'command '/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/bin/java'' finished with non-zero exit value 1
我们来添加一点逻辑,我选择了最经典了WordCount程序,参见代码: https://github.com/apache/flink/blob/master//flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/wordcount/WordCount.java
添加各种引用,发现没有org.apache.flink.api.java.utils.MultipleParameterTool。原来MultipleParameterTool是新增加的工具,在flink 1.9.0中还没有,我们升级到1.9.2,问题解决。build.gradle中修改
flinkVersion = '1.9.2'
也可以在创建项目,执行gradle-quickstart.sh的时候把参数写成“1.9.0 2.11”。
在IDE中直接执行WordCount的main方法得到结果
(we,4)
(weary,1)
(when,2)
(whips,1)
BUILD SUCCESSFUL in 14s
3 actionable tasks: 2 executed, 1 up-to-date
4:02:23 PM: Task execution finished 'WordCount.main()'.
部署到docker的flink集群
构建部署的jar包
➜ quickstart ./gradlew clean shadowJar
> Task :shadowJar FAILED
FAILURE: Build failed with an exception.
* What went wrong:
A problem was found with the configuration of task ':shadowJar'.
> No value has been specified for property 'mainClassName'.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 1s
4 actionable tasks: 4 executed
➜ quickstart
从https://mvnrepository.com/可以查到最新的shadow版本。 修改build.gradle,更新shadow版本到5.0.0
buildscript {
repositories {
jcenter() // this applies only to the Gradle 'Shadow' plugin
}
dependencies {
classpath 'com.github.jengelman.gradle.plugins:shadow:5.0.0'
}
}
plugins {
id 'java'
id 'application'
// shadow plugin to produce fat JARs
id 'com.github.johnrengelman.shadow' version '5.0.0'
}
重新执行./gradlew clean shadowJar
➜ quickstart ./gradlew clean shadowJar
BUILD SUCCESSFUL in 2s
4 actionable tasks: 4 executed
➜ quickstart
大功告成。
部署jar包到Flink平台
选择项目的build/libs/quickstart-0.1-SNAPSHOT-all.jar
在Entry Class处填写org.myorg.quickstart.WordCount
出现一个错误
通过命令docker logs operations-playground_jobmanager_1 查看jobmanager的日志,日志中有如下内容:
2020-02-06 08:38:05,589 WARN org.apache.flink.runtime.webmonitor.handlers.JarRunHandler - Configuring the job submission via query parameters is deprecated. Please migrate to submitting a JSON request instead.
2020-02-06 08:38:05,602 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler - Unhandled exception.
java.lang.RuntimeException: Could not look up the main(String[]) method from the class org.myorg.quickstart.WordCount: org/apache/flink/api/java/utils/MultipleParameterTool
at org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:546)
at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:234)
at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:161)
at org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:125)
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$6(JarRunHandler.java:142)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/api/java/utils/MultipleParameterTool
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:540)
... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.java.utils.MultipleParameterTool
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 14 more
原来找不到MultipleParameterTool。我们的程序是使用flink 1.9.2构建的,但是我们的playground版本是1.9.1。
我们看一下如何升级本地运行环境,首先看看启动环境使用的flink image名字。 在步骤“安装本地docker环境“中,我们下载了一个文件/flink-playgrounds/operations-playground/docker-compose.yaml
jobmanager:
image: flink:1.9-scala_2.11
command: "jobmanager.sh start-foreground"
...
taskmanager:
image: flink:1.9-scala_2.11
depends_on:
既然使用了image flink,我们到https://hub.docker.com/上找一下最新的flink版本。
https://hub.docker.com/_/flink中内容
我们选择1.9.2-scala_2.11,这个版本和我们的程序版本完全匹配。修改后如下
jobmanager:
image: flink:1.9.2-scala_2.11
command: "jobmanager.sh start-foreground"
...
taskmanager:
image: flink:1.9.2-scala_2.11
depends_on:
重启flink-playground。执行如下命令
docker-compose down -v
docker-compose up -d
这个命令需要在目录/flink-playgrounds/operations-playground/下执行。
重启后可以在地址http://localhost:8082/#/overview看到版本号已经变成了1.9.2了。重新上传包,并执行看效果。这次可以看到已经启动了。
好开心,好开心。静等结果……
可是都已经跑了3m了,怎么还没有结束?哪里不太对呢?
终于还是出错了。
通过这里能看到出错原因
原来是资源不够,没有分配host。
我们查看一下当前都有哪些资源和Job
从这里可以看到这个play-ground总共有2个Slot,当前Free slot是0。
从这里可以看到当前正在运行着一个Job Click Event Count。这个Job是fink play-ground用来测试使用的job,但它已经把所有资源都用了,我们可以先把它停掉。按照如下流程操作
这个操作仅仅一次性的取消掉了这个Job,重启play-ground还会再次起来,需要手工再次停止它,或者到docker-compose.yaml中将相关配置去掉,将这些注释掉的内容删除
version: "2.1"
services:
client:
build: ../docker/ops-playground-image
image: apache/flink-ops-playground:2-FLINK-1.9-scala_2.11
# command: "flink run -d -p 2 /opt/ClickCountJob.jar --bootstrap.servers kafka:9092 --checkpointing --event-time"
depends_on:
- jobmanager
- kafka
volumes:
- ./conf:/opt/flink/conf
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
# clickevent-generator:
# image: apache/flink-ops-playground:2-FLINK-1.9-scala_2.11
# command: "java -classpath /opt/ClickCountJob.jar:/opt/flink/lib/* org.apache.flink.playgrounds.ops.clickcount.ClickEventGenerator --bootstrap.servers kafka:9092 --topic input"
# depends_on:
# - kafka
jobmanager:
如果你采用注释的方式处理,一定要注意缩进,yaml文件对缩进要求严格,如果不生效或者出错,请调整缩进。
下面重新执行一次看效果。不需要重新上传jar包,直接进入Submit New Job,选择之前上传的jar,submit即可。
这次一秒完成。
当前项目中内容是盘中文件word-count-can-run-on-flink.zip
链接:https://pan.baidu.com/s/19PQzxWQsQsf8E7v-a-sRUA 密码:9uuq
通过这个路径可以看到结果,显示内容很不友好,之所以结果显示在这里,是因为我们代码中最后的输出是
counts.print();
正常的项目不会将结果直接print,而是将结果输出到数据库或者文件。
后面在其他章节中会讲解带参数运行WordCount和如何编写复杂的处理逻辑。
大家遇到什么问题可以在评论中说一下,我来完善文档
大家既然看到了这里,那就顺手给个赞吧👍