flink入坑记

最新推荐文章于 2024-07-01 02:25:02 发布

端口初始化

最新推荐文章于 2024-07-01 02:25:02 发布

阅读量1.8k

点赞数

文章标签： flink

本文链接：https://blog.csdn.net/u013591757/article/details/123956932

版权

flink是个好东西，但是入门很痛苦。因为没有step by step，没有helloworld。

软件上的任何新东西，要入门首先需要有个能跑起来的程序，跑起来了再去改就容易多了。

官网上的FraudDetector，洗钱检测是个很好的例子，但是没有完整的跑起来的步骤。

基于 DataStream API 实现欺诈检测 | Apache Flink

step 1 安装必要的开发工具，jdk、maven和IDE工具

安装java和maven，最好把maven默认源改成国内镜像。

IDE工具选择IDEA社区版

step 2 下载源码

运行maven命令下载源码

mvn archetype:generate \
    -DarchetypeGroupId=org.apache.flink \
    -DarchetypeArtifactId=flink-walkthrough-datastream-java \
    -DarchetypeVersion=1.14.4 \
    -DgroupId=frauddetection \
    -DartifactId=frauddetection \
    -Dversion=0.1 \
    -Dpackage=spendreport \
    -DinteractiveMode=false

如果顺利运行不报错，那么恭喜你得到一个可以运行的源码

step 3 配置运行参数

此时如果你直接在idea中点run去运行，一定是失败，一般你会看到下面的信息

[INFO] ------------------------------------------------------------------------
[ERROR] No goals have been specified for this build. You must specify a valid lifecycle phase or a goal in the format <plugin-prefix>:<goal> or <plugin-group-id>:<plugin-artifact-id>[:<plugin-version>]:<goal>. Available lifecycle phases are: validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy, pre-clean, clean, post-clean, pre-site, site, post-site, site-deploy. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/NoGoalSpecifiedException

Process finished with exit code 1

原因是你没有配置运行参数。
在edit configure里配置命令行参数：

exec:java -Dexec.mainClass=spendreport.FraudDetectionJob

如下图所示：

step 4 修改依赖配置

当你满心欢喜的以为everything ok，再去run时，会崩溃的又遇到一个错误

[INFO] 
[INFO] --- exec-maven-plugin:3.0.0:java (default-cli) @ frauddetection ---
[WARNING] 
java.lang.Exception: The specified mainClass doesn't contain a main method with appropriate signature.
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:258)
    at java.lang.Thread.run (Thread.java:834)
Caused by: java.lang.IllegalAccessException: no such method: spendreport.FraudDetectionJob.main(String[])void/invokeStatic
    at java.lang.invoke.MemberName.makeAccessException (MemberName.java:959)
    at java.lang.invoke.MemberName$Factory.resolveOrFail (MemberName.java:1101)
    at java.lang.invoke.MethodHandles$Lookup.resolveOrFail (MethodHandles.java:2030)
    at java.lang.invoke.MethodHandles$Lookup.findStatic (MethodHandles.java:1102)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:251)
    at java.lang.Thread.run (Thread.java:834)
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/streaming/api/functions/source/SourceFunction
    at java.lang.invoke.MethodHandleNatives.resolve (Native Method)
    at java.lang.invoke.MemberName$Factory.resolve (MemberName.java:1070)
    at java.lang.invoke.MemberName$Factory.resolveOrFail (MemberName.java:1098)
    at java.lang.invoke.MethodHandles$Lookup.resolveOrFail (MethodHandles.java:2030)
    at java.lang.invoke.MethodHandles$Lookup.findStatic (MethodHandles.java:1102)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:251)
    at java.lang.Thread.run (Thread.java:834)
Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.api.functions.source.SourceFunction
    at java.net.URLClassLoader.findClass (URLClassLoader.java:471)
    at java.lang.ClassLoader.loadClass (ClassLoader.java:588)
    at java.lang.ClassLoader.loadClass (ClassLoader.java:521)
    at java.lang.invoke.MethodHandleNatives.resolve (Native Method)
    at java.lang.invoke.MemberName$Factory.resolve (MemberName.java:1070)
    at java.lang.invoke.MemberName$Factory.resolveOrFail (MemberName.java:1098)
    at java.lang.invoke.MethodHandles$Lookup.resolveOrFail (MethodHandles.java:2030)
    at java.lang.invoke.MethodHandles$Lookup.findStatic (MethodHandles.java:1102)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:251)
    at java.lang.Thread.run (Thread.java:834)

什么？说我找不到main入口，明明代码里是有main函数，并且包名和类名全部对的啊。

原来是依赖项搞鬼

把如图所示的依赖项目都由provided改为runtime

step 5 enjoy it！

这下真的可以运行了，这时点运行，你才会看到和官网一致的输出结果：

2019-08-19 14:22:06,220 INFO  org.apache.flink.walkthrough.common.sink.AlertSink            - Alert{id=3}
2019-08-19 14:22:11,383 INFO  org.apache.flink.walkthrough.common.sink.AlertSink            - Alert{id=3}
2019-08-19 14:22:16,551 INFO  org.apache.flink.walkthrough.common.sink.AlertSink            - Alert{id=3}
2019-08-19 14:22:21,723 INFO  org.apache.flink.walkthrough.common.sink.AlertSink            - Alert{id=3}
2019-08-19 14:22:26,896 INFO  org.apache.flink.walkthrough.common.sink.AlertSink            - Alert{id=3}

最后，你就可以按照官网的说明增加例子的功能，好好玩耍了