提交第一个Flink流式作业

提交第一个Flink流式作业

1. 使用如下mvn命令编译项目

yitian@flink:~/flink-idea/quickstart$ mvn clean package
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Flink Quickstart Job 0.1
[INFO] ------------------------------------------------------------------------
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 46.834 s
[INFO] Finished at: 2019-03-17T06:11:13-07:00
[INFO] Final Memory: 51M/311M
[INFO] ------------------------------------------------------------------------

2. 提交并运行quickstart项目中的WikepediaAnalysis job，命令如下：

yitian@flink:~/flink/flink-1.7.2$ bin/flink run -c my.example.WikipediaAnalysis /home/yitian/flink-idea/quickstart/target/quickstart-0.1.jar 
Starting execution of program

3. 启动完成后，在webui中可以查看Flink job的各种状态：

 

4. 如果停止项目，比较简单的可以直接在上面图中右上方点击cancel按钮，即可停止任务。

Local模式下的Flink日志

Local（Standalone）模式下的flink日志，在如下目录中：

yitian@flink:~/flink/flink-1.7.2/log$ ls
flink-yitian-client-flink.log
flink-yitian-standalonesession-0-flink.log
flink-yitian-standalonesession-0-flink.log.1
flink-yitian-standalonesession-0-flink.log.2
flink-yitian-standalonesession-0-flink.out
flink-yitian-standalonesession-0-flink.out.1
flink-yitian-standalonesession-0-flink.out.2
flink-yitian-taskexecutor-0-flink.log
flink-yitian-taskexecutor-0-flink.log.1
flink-yitian-taskexecutor-0-flink.log.2
flink-yitian-taskexecutor-0-flink.out
flink-yitian-taskexecutor-0-flink.out.1
flink-yitian-taskexecutor-0-flink.out.2

打开其中的文件flink-yitian-standalonesession-0-flink.log，内容如下：

2019-03-17 06:12:44,331 INFO  org.apache.flink.client.cli.CliFrontend                       - --------------------------------------------------------------------------------
2019-03-17 06:12:44,335 INFO  org.apache.flink.client.cli.CliFrontend                       -  Starting Command Line Client (Version: 1.7.2, Rev:ceba8af, Date:11.02.2019 @ 14:17:09 UTC)
2019-03-17 06:12:44,335 INFO  org.apache.flink.client.cli.CliFrontend                       -  OS current user: yitian
2019-03-17 06:12:45,866 INFO  org.apache.flink.client.cli.CliFrontend                       -  Current Hadoop/Kerberos user: yitian
2019-03-17 06:12:45,866 INFO  org.apache.flink.client.cli.CliFrontend                       -  JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.151-b12
2019-03-17 06:12:45,867 INFO  org.apache.flink.client.cli.CliFrontend                       -  Maximum heap size: 875 MiBytes
2019-03-17 06:12:45,867 INFO  org.apache.flink.client.cli.CliFrontend                       -  JAVA_HOME: /usr/java/jdk1.8.0_151
2019-03-17 06:12:45,871 INFO  org.apache.flink.client.cli.CliFrontend                       -  Hadoop version: 2.7.5
2019-03-17 06:12:45,872 INFO  org.apache.flink.client.cli.CliFrontend                       -  JVM Options:
2019-03-17 06:12:45,872 INFO  org.apache.flink.client.cli.CliFrontend                       -     -Dlog.file=/home/yitian/flink/flink-1.7.2/log/flink-yitian-client-flink.log
2019-03-17 06:12:45,872 INFO  org.apache.flink.client.cli.CliFrontend                       -     -Dlog4j.configuration=file:/home/yitian/flink/flink-1.7.2/conf/log4j-cli.properties
2019-03-17 06:12:45,872 INFO  org.apache.flink.client.cli.CliFrontend                       -     -Dlogback.configurationFile=file:/home/yitian/flink/flink-1.7.2/conf/logback.xml
2019-03-17 06:12:45,872 INFO  org.apache.flink.client.cli.CliFrontend                       -  Program Arguments:
2019-03-17 06:12:45,872 INFO  org.apache.flink.client.cli.CliFrontend                       -     run
2019-03-17 06:12:45,872 INFO  org.apache.flink.client.cli.CliFrontend                       -     -c
2019-03-17 06:12:45,873 INFO  org.apache.flink.client.cli.CliFrontend                       -     my.example.WikipediaAnalysis
2019-03-17 06:12:45,873 INFO  org.apache.flink.client.cli.CliFrontend                       -     /home/yitian/flink-idea/quickstart/target/quickstart-0.1.jar
2019-03-17 06:12:45,873 INFO  org.apache.flink.client.cli.CliFrontend                       -  Classpath: /home/yitian/flink/flink-1.7.2/lib/flink-python_2.11-1.7.2.jar:/home/yitian/flink/flink-1.7.2/lib/flink-shaded-hadoop2-uber-1.7.2.jar:/home/yitian/flink/flink-1.7.2/lib/log4j-1.2.17.jar:/home/yitian/flink/flink-1.7.2/lib/slf4j-log4j12-1.7.15.jar:/home/yitian/flink/flink-1.7.2/lib/flink-dist_2.11-1.7.2.jar:::
2019-03-17 06:12:45,873 INFO  org.apache.flink.client.cli.CliFrontend                       - --------------------------------------------------------------------------------
2019-03-17 06:12:45,880 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, localhost
2019-03-17 06:12:45,881 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2019-03-17 06:12:45,882 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.size, 1024m
2019-03-17 06:12:45,882 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.size, 1024m
2019-03-17 06:12:45,882 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2019-03-17 06:12:45,882 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2019-03-17 06:12:45,883 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: rest.port, 8081
2019-03-17 06:12:46,731 INFO  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop user set to yitian (auth:SIMPLE)
2019-03-17 06:12:46,834 INFO  org.apache.flink.client.cli.CliFrontend                       - Running 'run' command.
2019-03-17 06:12:46,840 INFO  org.apache.flink.client.cli.CliFrontend                       - Building program from JAR file
2019-03-17 06:12:47,136 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address'
2019-03-17 06:12:47,765 INFO  org.apache.flink.runtime.rest.RestClient                      - Rest client endpoint started.
2019-03-17 06:12:47,770 INFO  org.apache.flink.client.cli.CliFrontend                       - Starting execution of program
2019-03-17 06:12:47,771 INFO  org.apache.flink.client.program.rest.RestClusterClient        - Starting program in interactive mode (detached: false)
2019-03-17 06:12:47,933 INFO  org.apache.flink.api.java.typeutils.TypeExtractor             - class org.apache.flink.streaming.connectors.wikiedits.WikipediaEditEvent does not contain a setter for field timestamp
2019-03-17 06:12:47,933 INFO  org.apache.flink.api.java.typeutils.TypeExtractor             - Class class org.apache.flink.streaming.connectors.wikiedits.WikipediaEditEvent cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.
2019-03-17 06:12:48,393 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, localhost
2019-03-17 06:12:48,394 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2019-03-17 06:12:48,394 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.size, 1024m
2019-03-17 06:12:48,394 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.size, 1024m
2019-03-17 06:12:48,395 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2019-03-17 06:12:48,395 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2019-03-17 06:12:48,396 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: rest.port, 8081
2019-03-17 06:12:48,431 INFO  org.apache.flink.client.program.rest.RestClusterClient        - Submitting job d8db8545fab8ed5dbaa71c4857e69177 (detached: false).
2019-03-17 06:26:30,645 INFO  org.apache.flink.runtime.rest.RestClient                      - Shutting down rest endpoint.
2019-03-17 06:26:30,652 INFO  org.apache.flink.runtime.rest.RestClient                      - Rest endpoint shutdown complete.
2019-03-17 06:26:30,654 ERROR org.apache.flink.client.cli.CliFrontend                       - Error while running the command.