Flink on yarn Yarn-Per-Job 作业日志 查询 方式记录

最新推荐文章于 2024-04-15 03:37:45 发布
最新推荐文章于 2024-04-15 03:37:45 发布
阅读量3.7k 收藏
点赞数
分类专栏： flink 文章标签： flink 大数据 yarn
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/htmljsp/article/details/123636481
版权

Flink 日志查询命令

获取application的contain信息

yarn logs -show_application_log_info --applicationId application_1639366579669_896581

WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
Application State: Completed.
Container: container_e50_1639366579669_896581_01_000005 on 192-168-16-13.yyipa.com_8041
Container: container_e50_1639366579669_896581_01_000006 on 192-168-16-13.yyipa.com_8041
Container: container_e50_1639366579669_896581_01_000007 on 192-168-16-13.yyipa.com_8041
Container: container_e50_1639366579669_896581_01_000002 on 192-168-16-13.yyipa.com_8041
Container: container_e50_1639366579669_896581_01_000003 on 192-168-16-13.yyipa.com_8041
Container: container_e50_1639366579669_896581_01_000004 on 192-168-16-13.yyipa.com_8041
Container: container_e50_1639366579669_896581_01_000001 on 192-168-16-43.yyipa.com_8041

获取每个container上的日志信息

yarn logs -containerId container_e50_1639366579669_896581_01_000005 --applicationId application_1639366579669_896581

WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.

End of LogType:prelaunch.err
******************************************************************************

Container: container_e50_1639366579669_896581_01_000007 on 192-168-16-13.yyipa.com_8041
LogAggregationType: AGGREGATED
=======================================================================================
LogType:prelaunch.out
LogLastModifiedTime:Sun Mar 20 03:08:56 +0800 2022
LogLength:70
LogContents:
Setting up env variables
Setting up job resources
Launching container

End of LogType:prelaunch.out
******************************************************************************

Container: container_e50_1639366579669_896581_01_000007 on 192-168-16-13.yyipa.com_8041
LogAggregationType: AGGREGATED
=======================================================================================
LogType:taskmanager.err
LogLastModifiedTime:Sun Mar 20 03:08:56 +0800 2022
LogLength:557
LogContents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/yarn/nm/usercache/bduser/appcache/application_1639366579669_896581/filecache/11/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

End of LogType:taskmanager.err
********************************************************************************

Container: container_e50_1639366579669_896581_01_000007 on 192-168-16-13.yyipa.com_8041
LogAggregationType: AGGREGATED
=======================================================================================
LogType:taskmanager.out
LogLastModifiedTime:Sun Mar 20 03:08:56 +0800 2022
LogLength:12237
LogContents:
12:58:24.502 [main-EventThread] ERROR org.apache.flink.shaded.curator4.org.apache.curator.ConnectionState - Authentication failed
03:08:39.648 [Legacy Source Thread - Source: Custom Source -> Flat Map -> Timestamps/Watermarks -> Process -> Sink: Unnamed (4/6)#0] ERROR com.youyuan.flink.app.YouyueDataAllocation - Occur Exception: java.net.SocketTimeoutException: Read timed out data: {"user_id":1060491087,"event_ts":1647716917585,"heart_time":1647716914132,"sex":1}
03:08:55.508 [Curator-Framework-0] ERROR org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl - Background exception was not retry-able or retry gave up
org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /flink/application_1639366579669_896581/leader
	at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:114) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:308) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CreateBuilderImpl$9.performBackgroundOperation(CreateBuilderImpl.java:801) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.OperationAndData.callPerformBackgroundOperation(OperationAndData.java:84) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:965) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_181]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
03:08:55.627 [Curator-Framework-0] ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor - Fatal error occurred in TaskExecutor akka.tcp://flink@192-168-16-13.yyipa.com:40673/user/rpc/taskmanager_0.
org.apache.flink.runtime.leaderretrieval.LeaderRetrievalException: Unhandled error in ZooKeeperLeaderRetrievalDriver:Background exception was not retry-able or retry gave up
	at org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalDriver.unhandledError(ZooKeeperLeaderRetrievalDriver.java:184) [youyue-data-allocation-1.0.jar:?]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$6.apply(CuratorFrameworkImpl.java:713) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$6.apply(CuratorFrameworkImpl.java:709) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:92) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.logError(CuratorFrameworkImpl.java:708) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.handleBackgroundOperationException(CuratorFrameworkImpl.java:924) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:1001) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_181]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /flink/application_1639366579669_896581/leader
	at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:114) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:308) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CreateBuilderImpl$9.performBackgroundOperation(CreateBuilderImpl.java:801) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.OperationAndData.callPerformBackgroundOperation(OperationAndData.java:84) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:965) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	... 9 more
03:08:55.629 [Curator-Framework-0] ERROR org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Fatal error occurred while executing the TaskManager. Shutting it down...
org.apache.flink.runtime.leaderretrieval.LeaderRetrievalException: Unhandled error in ZooKeeperLeaderRetrievalDriver:Background exception was not retry-able or retry gave up
	at org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalDriver.unhandledError(ZooKeeperLeaderRetrievalDriver.java:184) [youyue-data-allocation-1.0.jar:?]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$6.apply(CuratorFrameworkImpl.java:713) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$6.apply(CuratorFrameworkImpl.java:709) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:92) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.logError(CuratorFrameworkImpl.java:708) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.handleBackgroundOperationException(CuratorFrameworkImpl.java:924) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:1001) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_181]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /flink/application_1639366579669_896581/leader
	at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:114) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:308) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CreateBuilderImpl$9.performBackgroundOperation(CreateBuilderImpl.java:801) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.OperationAndData.callPerformBackgroundOperation(OperationAndData.java:84) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:965) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
	... 9 more

End of LogType:taskmanager.out
********************************************************************************
htmljsp
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 1
    评论
专栏目录
CDH平台YARN日志查看和问题排查
weixin_43497829的博客
02-20 6733
1.查看YARN中作业的web端口 方式一： 通过ResourceManagerWeb UI 查看日志：( 这种方式可以查看当前正在执行以及历史所有任务） 1.1 先找到 YARN ResourceManager 角色所在服务器的IP地址，可通过如下图方式进行查看。 CM操作路径：集群-YARN-实例-ResourceManage角色-主机。 1.2 然后查找端口， 通过查找yarn-site配置项 yarn.resourcemanger.webapp.address 的值，默认是8088.
Flink 1.13 cdc 任务 on yarn 日志查询详解
qq_31866793的博客
01-15 4926
目录 一，正常部署之后，flink on yarn 任务查看 二，shell命令查看具体日志 三，开启historyserver 四，使用yarn的restfulApi查看信息 五，总结： 一，正常部署之后，flink on yarn 任务查看 1，正在运行任务： 2，杀死任务之后： 3，我们点击History按钮 进入页面： 这并不是我们想要的效果，so，我们要看看具体的操作。 二，shell命令查看具体日志 在shell命令执行：..
Flink任务获取yarn上的运行日志，腾讯大数据开发面试题
最新发布
2401_84160361的博客
04-15 221
flink任务在yarn失败，需要从yarn的归档服务器当中获取运行日志用来排查异常原因，正在运行中的任务日志从当前运行的Node节点中读取，方便监控任务运行信息。
Flink Yarn Per Job - Yarn应用
hyunbar的博客
08-01 1117
决定后面什么类型的执行器提交任务yarn-session、yarn-per-job。是否指定为per-job模式，即指定”-myarn-cluster”;是否存在flink在yarn的appID，即yarn-session模式是否启动。executor的名字为“yarn-session”或“yarn-per-job”Generic、Yarn、Default三种命令行客户端。依次是Generic、Yarn、Default。匹配一个“-”的参数或者两个“–”的参数。...
yarn 怎么查看有多个job在跑_flink on yarn 模式下提示yarn资源不足问题分析
weixin_39771987的博客
12-11 870
背景在实时计算平台上通过YarnClient向yarn上提交flink任务时一直卡在那里，并在client端一直输出如下日志：(YarnClusterDescriptor.java:1036)- Deployment took more than 60 seconds. Please check if the requested resources are available in th...
kylin启动报错Background exception was not retry-able or retry gave up java.net.UnknownHostException
张不帅
07-06 5875
2020-05-29 13:41:50,902 INFO [main] zookeeper.ZooKeeper:438 : Initiating client connection, connectString=note01:2181:2181,note02:2181:2181,note03:2181:2181 sessionTimeout=120000 watcher=org.apache.curator.ConnectionState@12cd9150 2020-05-29 13:41:55,964
cdh6.3.1 FLINK_ON_YARN-1.14.0.jar
02-14
CDH6.3.1集成安装flink on yarn服务（通过parcel制作生成的csd文件） 文件名：FLINK_ON_YARN-1.14.0.jar 相关信息如下： 1. flink版本号：1.14.0 2. 系统版本：CentOS7.7 3. CDH版本：6.3.1 4. 扩展版本号：BIN-...
flink-on-yarn-1.13.2.jar
09-28
本人搭建cdh6.3.2环境时候集成flink1.13.2组件使用的flink_on_yarn-1.13.2.jar资源包，已验证可以使用，是本人制作parcel的时候的成品
FLINK_ON_YARN-1.14.3.jar
04-05
CDH6.2 FLINK on yarn : FLINK_ON_YARN-1.14.3.jar
FLINK_ON_YARN-1.12.0.jar
11-01
适用范围CDH (>= 5.2), CDH (),系统版本 el7
hive sql 卡在提交JOB
只是甲的博客
03-12 1519
备注: Hive 版本 2.1.1 文章目录一.问题描述二.问题解决2.1 首先想到的是重启大法2.1 查看正在执行的job三.验证参考 一.问题描述 之前开始归档，然后job卡住，我直接其它CTRL + C退出，后面运行mr job都卡在提交job的环节 hive> select * from emp where empno = 7369; Query ID = root_20201224123252_93210ca3-928f-49e7-9cf4-d442d2a6ca3e Total jobs
Centos 搭建Hadoop
爱是与世界平行
06-12 644
Centos搭建Hadoop一、搭建Hadoop需要JDK环境，首先配置JDK二、下载haoop三、在Centos服务器上解压下载好的安装包四、修改配置文件4.1 hadoop-env.sh4.2 core-site.xml4.3 hdfs-site.xml4.4 mapred-site.xml4.5 yarn-site.xml五、配置环节变量/etc/profile六、格式化NameNode七、...
Yarn日志 - Log Aggregation
weixin_44129801的博客
01-19 4021
1. 概述 大数据任务的日志，首先保存于本地，但本地存储能力有限，且检索时需要切不同服务器，不够方便 Yarn Log Aggregation的主要功能，是将本地日志移动到HDFS或者云存储 日志的重要性 错误定位 历史分析 2. 聚合开关 yarn.log-aggregation-enable，yarn日志聚合功能开关 true表示开启 3. 存储路径 3.1. 本地存储 yarn.nodemanager.local-dirs，本地化文件存储目录，如JAR包、配置文件，参考值/hado
yarn任务信息的几种获取方式
hncscwc的博客
06-02 6687
在问题定位、日常巡检、特定开发任务中，都会涉及查看yarn任务的相关信息，包括applicaiton的ID、类型、名称、起始时间、app所包含的container、以及每个container的日志文件内容等信息。本文就来聊聊查询查看这些信息的几种方式。【通过RM的WEB查看】这个是最简单的方式，直接在浏览器中打开RM的web网页就可以查看所有的App信息以及对应的日志了。...
Flink日志清洗案例
cts618
03-03 881
package com.cnic.flink.log.coding; import com.alibaba.fastjson.JSON; import java.util.HashMap; import java.util.Map; public class ContactsEntity { public static String LOG = "00:00:05 INFO [tid:2607,Q:W183CWbcBewrShAYJOeH] [L-Android]func=pab:se.
完美解决WARNING: YARN_LOG_DIR has been replaced by HADOOP_LOG_DIR. Using value of YARN_LOG_DIR.
qq_45773650的博客
07-20 3786
如上述清理，启动正常，就是报警告，看着心里痒痒的。 解决方案： 主要是/etc/profile里文件的错误，用命令：vim /etc/profile ,之后把报错的所在一行名字注销了，如我的YARN_LOG_DIR,如图 之后source /etc/profile，是文件立即生效，注意从机不要忘记操作，每一个都需要这样做，之后重新启动，命令： start-all.sh 就ok了 若是还报错，就没有虚拟操作系统都重启一下，我的就是这样好的。 ...
Hadoop集群笔记（一）
YlanHds的博客
04-28 1239
Hadoop 集群搭建 vim hadoop-env.sh export JAVA_HOME=/usr/java/jdk1.8.0_191-amd64 vim core-site.xml <configuration> <!-- 指定HDFS中NameNode的地址 --> <property> <name>fs.de...
flink taskmanager
06-06
Flink TaskManager是Flink分布式计算框架的核心组件之一，它是Flink集群中运行Flink任务的工作进程。每个TaskManager都可以运行多个任务，每个任务都是一个或多个并行的Flink算子。TaskManager负责将任务分配到适当的线程池中进行执行，并通过网络与其他TaskManager交换数据和状态信息，从而实现分布式计算。在Flink集群中，TaskManager的数量直接影响着任务的并发度和整个应用程序的吞吐量。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

htmljsp

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值