windows快速搭建Flink官方练习环境

最新推荐文章于 2024-08-19 10:36:31 发布

vkingnew

最新推荐文章于 2024-08-19 10:36:31 发布

阅读量1.9k

点赞数

分类专栏： Flink

本文链接：https://blog.csdn.net/vkingnew/article/details/108580444

版权

Flink 专栏收录该内容

3 篇文章 0 订阅

订阅专栏

运行环境：
windows 10
git 2.2x 版本
docker-desktop
java 8

IDEA 
注意：需要开启windows系统的hyper-v

网站：

https://github.com/apache/flink-playgrounds

目前支持Flink 1.8 ，1.9，1.10,1.11版本。

git windows 下载：

https://git-scm.com/

或者

https://git-scm.com/download/win

https://www.docker.com/products/docker-desktop

docker下载：

docker desktop 比较大有374M之大，最好借助下载工具下载。

这里自动安装Hyper-V 需要重启windows 操作系统。

安装过程慢慢等待约需要3-5分钟。

>docker version
Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:43:18 2020
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:49:27 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

C:\>cd C:\dev

C:\dev>git clone --branch release-1.11 https://github.com/apache/flink-playgrounds.git
Cloning into 'flink-playgrounds'...
remote: Enumerating objects: 114, done.
remote: Counting objects: 100% (114/114), done.
remote: Compressing objects: 100% (53/53), done.
remote: Total 278 (delta 38), reused 99 (delta 30), pack-reused 164
Receiving objects: 100% (278/278), 61.00 KiB | 41.00 KiB/s, done.
Resolving deltas: 100% (113/113), done.


C:\>cd C:\dev\flink-playgrounds\operations-playground
C:\dev\flink-playgrounds\operations-playground>docker-compose build

此步骤消耗时间甚多，需要有稳定的网络连接https://repo.maven.apache.org 下载jar包。

我这里编译消耗了36分
INFO] Installing /opt/flink-playground-clickcountjob/dependency-reduced-pom.xml to /root/.m2/repository/org/apache/flink/flink-playground-clickcountjob/1-FLINK-1.11_2.11/flink-playground-clickcountjob-1-FLINK-1.11_2.11.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  36:36 min
[INFO] Finished at: 2020-09-14T09:09:29Z
[INFO] ------------------------------------------------------------------------
Removing intermediate container c02151277b1d
 ---> 33ec534d2037
Step 5/7 : FROM flink:1.11.0-scala_2.11
1.11.0-scala_2.11: Pulling from library/flink
                                                                                                                                                   Digest: sha256:11ec0bcfdab8c3c0a33f847f3cedded3f3e71c30fff9a08f7a2279e910386032
Status: Downloaded newer image for flink:1.11.0-scala_2.11
 ---> 8bd2014daa3f
Step 6/7 : WORKDIR /opt/flink/bin
 ---> Running in 1931166fd820
Removing intermediate container 1931166fd820
 ---> ab6a88a88e1a
Step 7/7 : COPY --from=builder /opt/flink-playground-clickcountjob/target/flink-playground-clickcountjob-*.jar /opt/ClickCountJob.jar
 ---> 95f66a0032a6
Successfully built 95f66a0032a6
Successfully tagged apache/flink-ops-playground:1-FLINK-1.11-scala_2.11

启动docker中的flink运行环境：

C:\dev\flink-playgrounds\operations-playground>docker-compose up -d
operations-playground_kafka_1 is up-to-date
operations-playground_jobmanager_1 is up-to-date
operations-playground_zookeeper_1 is up-to-date
operations-playground_clickevent-generator_1 is up-to-date
operations-playground_taskmanager_1 is up-to-date
operations-playground_client_1 is up-to-date


查看 正在运行的docker容器：

C:\dev\flink-playgrounds\operations-playground>docker-compose ps
                    Name                                  Command               State                   Ports
-----------------------------------------------------------------------------------------------------------------------------
operations-playground_clickevent-generator_1   /docker-entrypoint.sh java ...   Up       6123/tcp, 8081/tcp
operations-playground_client_1                 /docker-entrypoint.sh flin ...   Exit 0
operations-playground_jobmanager_1             /docker-entrypoint.sh jobm ...   Up       6123/tcp, 0.0.0.0:8081->8081/tcp
operations-playground_kafka_1                  start-kafka.sh                   Up       0.0.0.0:9094->9094/tcp
operations-playground_taskmanager_1            /docker-entrypoint.sh task ...   Up       6123/tcp, 8081/tcp
operations-playground_zookeeper_1              /bin/sh -c /usr/sbin/sshd  ...   Up       2181/tcp, 22/tcp, 2888/tcp, 3888/tcp


在docker中默认配置了8081端口访问：

下次开机之后只需要做如下操作就可以使用了：

cd C:\dev\flink-playgrounds\operations-playground
docker-compose up -d
docker-compose ps

列举常见的一些简单操作：
若需要停止开发环境：
docker-compose down -v

查看日志：
JobManager日志可以通过docker-compose查看：
docker-compose logs -f jobmanager

TaskManager 日志：
docker-compose logs -f taskmanager

Flink CLI：
docker-compose run --no-deps client flink --help

Flink RestAPI：
curl localhost:8081/jobs

Kafka 操作命令：
//input
docker-compose exec kafka kafka-console-consumer.sh  --bootstrap-server localhost:9092 --topic input

//output
docker-compose exec kafka kafka-console-consumer.sh  --bootstrap-server localhost:9092 --topic output


列出正在运行的作业：
docker-compose run --no-deps client flink list

故障观察和恢复：
1.输出的信息：
docker-compose exec kafka kafka-console-consumer.sh  --bootstrap-server localhost:9092 --topic output

2.故障模拟：
可以模拟taskmanager进程的丢失：
docker-compose kill taskmanager

几秒钟之后JobManager 将意识到TaskManager信息数据的丢失，取消受影响的job然后重新提交进行恢复。

3.恢复taskmanager
docker-compose up -d taskmanager

查看运行的job: