flink
消八哥
这个作者很懒,什么都没留下…
展开
-
flink on yarn ——报错ResourceLocalizationService: Could not carry out resource dir checks
报错:java.io.FileNotFoundException: File /home/data/software/hadoop-3.2.2/data/tmp/nm-local-dir/filecache does not exist2022-03-29 15:58:07,592 INFO org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection: Directory /home/data/software/hadoop-3.2.原创 2022-03-29 17:43:38 · 2282 阅读 · 0 评论 -
flink 1.10以上执行程序报错NoClassDefFounderror
是flink的lib下少了包:log4j-slf4j-impl-2.12.1.jarlib依赖包下载地址:链接:https://pan.baidu.com/s/11g531yXyyjl73LRC9ZMhpg提取码:9vvf--来自百度网盘超级会员V7的分享原创 2022-03-29 17:25:36 · 1665 阅读 · 0 评论 -
flink——监控启动脚本报错:/bin/flink: No such file or directory
场景:crontab定时执行脚本内容:#!/bin/bashsource /etc/profile #添加有解决/bin/flink: No such file or directory报错path=/home/jkop/software/flink-1.11.0/publish/arry_list1=("Monitorzip_news_first" "MonitorZip_news_second" "IntranetNewsES_first" "IntranetAttNewsES_原创 2022-03-19 17:06:09 · 2354 阅读 · 0 评论 -
flink on yarn ——报错:Deployment took more than 60 seconds. Please check if the requested resources are
报错描述,看着似乎是资源不够使用。Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster 解决方案:配置yarn-site.xml <property> <name>yarn.scheduler.minimum-allocation-mb</name>...原创 2022-03-15 16:45:23 · 10801 阅读 · 0 评论 -
flink——配置理解
flink-conf.yaml# Slot的个数taskmanager.numberOfTaskSlots: 2此配置分两种情况使用:1、在flink单独集群的时候可以配置多些,供整个flink集群使用。2、在flink on yarn 中使用是需要配置不超过服务器的vCores,供每次启动一个job任务时使用。...原创 2022-03-14 00:02:13 · 940 阅读 · 1 评论 -
报错处理——org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
具体报错如下:ckpoints to JobManager) (checkpoints: 'hdfs://craw-node212:9000/flink-checkpoints', savepoints: 'null', asynchronous: TRUE, maxStateSize: 5242880)2022-03-13 22:48:00.641 [Process (3/4)] INFO org.apache.flink.runtime.taskmanager.Task - Loading J原创 2022-03-13 23:10:16 · 7656 阅读 · 0 评论 -
flink on yarn -启动任务报错——FileNotFoundException: File file:/root/.flink/application_1647864661406_0365/
Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment. Diagnostics from YARN: Application application_1647077633391_0005 failed 1 times (global limit =.原创 2022-03-12 18:37:06 · 3430 阅读 · 0 评论 -
flink- 错误:启动多个任务,只有个一任务执行
环境:flink on yarn问题描述:执行多个任务,只有个一任务执行,查看界面发现:显示没错点开flink页面发现两个任务是同一个,而且只有这一个任务在执行。问题解决:因为配置了flink的高可用导致的,把flink-conf.yaml中高可用的配置注掉后能正常使用了...原创 2022-03-01 15:55:01 · 1740 阅读 · 0 评论 -
flink- state 状态
一、flink- state是什么?flink的状态一般指一个具体的task/ operate某时刻在内存的状态(某属性值)。例如定义变量:ValueState<String> name;有flink管理在内存中,可以恢复。二、flink- state的作用· 增量计算聚合操作机器学习训练模式等· 容错job故障重启升级...原创 2022-02-25 16:13:50 · 1339 阅读 · 0 评论 -
flink- AsyncDataStream 异步处理
1、AsyncDataStream有2个方法:unorderedWait表示数据不需要关注顺序,处理完立即发送,orderedWait表示数据需要关注顺序,为了实现该目标,操作算子会在该结果记录之前的记录为发送之前缓存该记录。这往往会引入额外的延迟和一些Checkpoint负载,因为相比于无序模式结果记录会保存在Checkpoint状态内部较长的时间。...原创 2022-02-25 14:41:34 · 2690 阅读 · 0 评论