基于flink1.15.0
Session Mode on yarn
开启flink yarn-session集群
./bin/yarn-session.sh -d \
-Dtaskmanager.numberOfTaskSlots=5 \
-Dflink.cluster.name=flink-yarn-session \
-Dyarn.application.name="flink-yarn-session" \
-Dresourcemanager.taskmanager-timeout=30000 \
-Drest.flamegraph.enabled=true
提交任务
- 方式一,不指定 yarn app-id提交
./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
或者
- 方式二,手动指定flink集群,application_1652928425471_0282是flink集群yarn app-id
./bin/flink run -t yarn-session -d
-Dyarn.application.id=application_1652928425471_0285
./examples/streaming/TopSpeedWindowing.jar
- 方式三,在flink web界面上传jar包提交
进入客户端交互界面
./bin/yarn-session.sh -id application_1652928425471_0252
优雅地停止任务
echo “stop” | ./bin/yarn-session.sh -id application_1652928425471_0245
强制杀掉任务
yarn application -kill application_1652928425471_0266
NOTE: 启动集群只有jobmanager,提交job才去yarn申请资源启动taskmanager,任务完成自动释放taskmanager
Per-Job Mode on yarn
启动perjob集群
./bin/flink run -t yarn-per-job --detached \
-Dyarn.application.name="flink-yarn-perjob" \
-Dflink.cluster.name=flink-yarn-perjob \
-Dtaskmanager.numberOfTaskSlots=5 \
./examples/streaming/TopSpeedWindowing.jar
查看集群上运行的job
./bin/flink list -t yarn-per-job -Dyarn.application.id=application_1652928425471_0252
取消某个job
./bin/flink cancel -t yarn-per-job -Dyarn.application.id=application_1652928425471_0252
./bin/flink cancel -t yarn-per-job -Dyarn.application.id=application_1652928425471_0252 ac02048e90c0ad7ec370542b412e498c
**NOTE:**在perjob模式,取消某个job相当于停止集群
Application Mode on yarn
flink1.11之后引入
启动application集群
./bin/flink run-application \
-t yarn-application \
-Dyarn.application.name="flink-yarn-application" \
-Dflink.cluster.name=flink-yarn-perjob \
-Dtaskmanager.numberOfTaskSlots=5 \
-Djobmanager.memory.process.size=1024m \
-Dtaskmanager.memory.process.size=1024m \
-Drest.flamegraph.enabled=true \
./examples/streaming/TopSpeedWindowing.jar
查看集群上运行的job
./bin/flink list -t yarn-application -Dyarn.application.id=application_1652928425471_0268
取消某个job
./bin/flink cancel -t yarn-application -Dyarn.application.id=application_1652928425471_0268
./bin/flink cancel -t yarn-application -Dyarn.application.id=application_1652928425471_0254 17e0ba867df35d98c2d70be8103fff2e
**NOTE:**在perjob模式,取消某个job相当于停止集群
另外可以结合yarn依赖预上传设置,来优化job的提交效果