论一次flink checkpoint失败的排查过程

Container id: container_e910_1637406318221_42547_01_000003
Exit code: 1
Shell output: main : command provided 1
main : run as user is
main : requested yarn user is
Getting exit code file…
Creating script paths…
Writing pid file…
Writing to tmp file 10_1637406318221_42547_01_000003.pid.tmp
Writing to cgroup task files…
Creating local dirs…
Launching container…

[2021-12-23 17:10:06.266]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :

[2021-12-23 17:10:06.266]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :

报错信息如上, 程序总是在一段时间内就会失败, 报错信息中只能找到以上信息, 找了很长时间, 发现重启后checkpoint失败后

如图, 变成了 n/a, 往下深究, 发现在查询mysql的维表时, 出现了查询超时, 最终才导致的任务失败, 反查逻辑, 重新梳理后问题解决