flink扫盲-实验环境搭建与入门

写作缘由

从flink的Github 介绍我们可以得到如下信息：

Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities.

可以看出，它主推的两个功能stream和batch-processing,我们可以拿这两个功能和业界一些其他产品进行对比。对于stream也就是数据流的处理，有storm,Heron等等。对于batch-processing也就是批处理，那就更多了，同时拥有stream和batch-processing处理能力，大家比较熟悉的有spark。所以之前经常会把spark和flink作为对比。从两者发展的速度来讲，因为spark太过光芒，导致一时间大家都毫不犹豫的选择了spark而摒弃flink.最近因为国内某大厂大肆宣扬flink。所以flink热点一些高涨起来。那么下面一系列的文章我们就来扫扫flink的盲

flink 入门实验

下面来实验一些flink 中的hello-world

直接下载地址进行安装,如下为实验机器的flink 解压之后的环境

➜  flink pwd
/usr/local/flink
➜  flink ls
LICENSE    README.txt conf       lib        log
NOTICE     bin        examples   licenses   opt

step1:启动本地集群

$ ./bin/start-cluster.sh  # Start Flink

之后访问http://localhost:8081/可以看到如下界面

同时可以查看一下启动日志

➜  log tail flink-xx-standalonesession-1-xx.log
2018-12-29 10:18:43,481 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Rest endpoint listening at localhost:8081
2018-12-29 10:18:43,482 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - http://localhost:8081 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000
2018-12-29 10:18:43,482 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Web frontend listening at http://localhost:8081.
2018-12-29 10:18:43,807 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager .
2018-12-29 10:18:43,861 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
2018-12-29 10:18:43,904 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - ResourceManager akka.tcp://flink@localhost:6123/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000
2018-12-29 10:18:43,905 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Starting the SlotManager.
2018-12-29 10:18:43,944 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Dispatcher akka.tcp://flink@localhost:6123/user/dispatcher was granted leadership with fencing token 00000000-0000-0000-0000-000000000000
2018-12-29 10:18:43,949 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Recovering all persisted jobs.
2018-12-29 10:18:46,369 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - Registering TaskManager with ResourceID 61207bdc3873f4dbc3837f9abb60a79c (akka.tcp://flink@legotimedemacbook-air.local:50280/user/taskmanager_0) at ResourceManager

step2:启动本地的输入

➜  ~ nc -l 9000

step3:启动flink计算程序

➜  flink ./bin/flink run examples/streaming/SocketWindowWordCount.jar --port 9000

step4:实验

nc端输入一些字符，其他不懂

➜  ~ nc -l 9000
hello,world
hello,legotime
this is flink

此刻在log目录下面的flink-xx-taskexecutor-xx.out会有一些日志文件，如下

➜  log cat flink-legotime-taskexecutor-2-legotimedeMacBook-Air.local.out
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/flink/lib/slf4j-log4j12-1.7.15.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/legotime/App/hive/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
hello,world : 1
hello,legotime : 1
this : 1
flink : 1
is : 1

发现有

hello,world : 1
hello,legotime : 1
this : 1
flink : 1
is : 1

上面的就是计算结果，恭喜你，调通一个flink hello-world的程序

更多配置信息

配置信息请访问官方链接