Flink

目录

问题1：

问题2：

问题3：

问题4：

---

问题1：使用CDC执行flink任务时报错

flink Can''t call rollback when autocommit=true

解决：可能是kafka的topic没有数据，判空就可以了。

---

问题2：集群上执行flink任务

per-job模式运行报异常，global limit =2; local limit is =1

解决：代码引用的依赖和集群上的hadoop环境冲突。

---

问题3：数据量大的时候执行per-job模式报错

Caused by: java.lang.Exception: [2021-08-17 18:42:50.353]Exception from container-launch.
Container id: container_e11_1627612938926_0154_01_000004
Exit code: 239

[2021-08-17 18:42:50.354]Container exited with a non-zero exit code 239. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :

[2021-08-17 18:42:50.355]Container exited with a non-zero exit code 239. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :

 

解决：上面的报错一般都是其他报错导致的Container异常（下面的报错只是其中一种情况），目前的解决方案是数据量太大，处理不过来。任务挂掉之后从最新位置开始读取数据就可以了。会导致丢失任务失败的间隔段。

可以调大参数

flink run -d  -m yarn-cluster -ynm retrans -yjm 1024 -ytm 3072 -yD taskmanager.memory.jvm-overhead.max=1G -yD taskmanager.memory.jvm-overhead.min=1G -c xxx xxx.jar    

---

问题4：执行per-job模式报错

Caused by: java.lang.RuntimeException: org.apache.flink.runtime.client.JobInitializationException: Could not instantiate JobManager.
	at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:309)
	at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:76)
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
	at java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:443)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: org.apache.flink.runtime.client.JobInitializationException: Could not instantiate JobManager.
	at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:463)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.util.FlinkRuntimeException: Failed to create checkpoint storage at checkpoint coordinator side.

解决：hdfs的地址写错了，要用active的。

---

问题5：执行任务控制台出现很多异常

一般都是jar包日志冲突，下载个maven-helper插件，把冲突的都去掉，只留一个就OK了。

---