具体报错如下:连接远程kafka服务失败。
ckpoints to JobManager) (checkpoints: 'hdfs://craw-node212:9000/flink-checkpoints', savepoints: 'null', asynchronous: TRUE, maxStateSize: 5242880)
2022-03-13 22:48:00.641 [Process (3/4)] INFO org.apache.flink.runtime.taskmanager.Task - Loading JAR files for task Process (3/4) (8ca755fdd1f0138278dc1a2ad31b3a7d) [DEPLOYING].
2022-03-13 22:48:00.642 [Process (3/4)] INFO org.apache.flink.runtime.taskmanager.Task - Registering task at network: Process (3/4) (8ca755fdd1f0138278dc1a2ad31b3a7d) [DEPLOYING].
2022-03-13 22:48:00.642 [Process (4/4)] INFO org.apache.flink.runtime.taskmanager.Task - Process (4/4) (e5a99f4f92428d9c1d1f33bf54a454aa) switched from DEPLOYING to RUNNING.
2022-03-13 22:48:00.644 [Process (3/4)] INFO o.apache.flink.streaming.runtime.tasks.StreamTask - No state backend has been configured, using default (Memory / JobManager) MemoryStateBackend (data in heap memory / checkpoints to JobManager) (checkpoints: 'hdfs://craw-node212:9000/flink-checkpoints', savepoints: 'null', asynchronous: TRUE, maxStateSize: 5242880)
2022-03-13 22:48:00.644 [Process (3/4)] INFO org.apache.flink.runtime.taskmanager.Task - Process (3/4) (8ca755fdd1f0138278dc1a2ad31b3a7d) switched from DEPLOYING to RUNNING.
2022-03-13 22:48:00.647 [Source: Custom Source (2/2)] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : 2.0.1
2022-03-13 22:48:00.647 [Source: Custom Source (2/2)] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId : fa14705e51bd2ce5
2022-03-13 22:48:00.653 [Source: Custom Source (1/2)] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : 2.0.1
2022-03-13 22:48:00.654 [Source: Custom Source (1/2)] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId : fa14705e51bd2ce5
2022-03-13 22:49:00.650 [Source: Custom Source (2/2)] WARN org.apache.flink.runtime.taskmanager.Task - Source: Custom Source (2/2) (3214cdc7c1e6343f65d3ceb83c74c479) switched from RUNNING to FAILED.
org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2022-03-13 22:49:00.650 [Source: Custom Source (2/2)] INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Source: Custom Source (2/2) (3214cdc7c1e6343f65d3ceb83c74c479).
2022-03-13 22:49:00.652 [flink-akka.actor.default-dispatcher-2] INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and sending final execution state FAILED to JobManager for task Source: Custom Source (2/2) 3214cdc7c1e6343f65d3ceb83c74c479.
2022-03-13 22:49:00.658 [Source: Custom Source (1/2)] WARN org.apache.flink.runtime.taskmanager.Task - Source: Custom Source (1/2) (cb24bf621522953767b2bcd32c5bb474) switched from RUNNING to FAILED.
org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2022-03-13 22:49:00.658 [Source: Custom Source (1/2)] INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Source: Custom Source (1/2) (cb24bf621522953767b2bcd32c5bb474).
查看页面,消费不到数据:
程序端:查看代码里的配置文件:(错误,这样配置单独flink集群能用,放入yarn上就不行)
修改为远端kafka服务器的hostname:(修改后如果确保修改的正确,启动时还会消费不到数据的话,需要稍等半分钟,等待kafka自动均衡完毕再启动)
1、远程kafka服务端配置情况:远程服务端是可以连接外网
2、集群中可能有台机器连接不上kafka 9092