入坑Flink - Flink on Yarn提交任务异常

最新推荐文章于 2024-01-05 11:54:52 发布
最新推荐文章于 2024-01-05 11:54:52 发布
阅读量1.7k 收藏 1
点赞数 1
分类专栏： flink 文章标签： flink
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/lfish001/article/details/113127368
版权

入坑Flink - Flink on Yarn提交任务异常


在通过 flink run -m yarn-cluster 提交任务过程中任务一直created
在这里插入图片描述
查看jobManager日志发现jar包冲突 

2021-01-25 16:43:51,735 INFO  org.apache.flink.yarn.YarnResourceManager                     - Request slot with profile ResourceProfile{UNKNOWN} for job 82a3646f98fefd88eb9e3b25a6defd12 with allocation id 58a3ee1c12974c0bad093e6c32fade38.
2021-01-25 16:43:51,742 INFO  org.apache.flink.yarn.YarnResourceManager                     - Requesting new TaskExecutor container with resources <memory:2048, vCores:1>. Number pending requests 1.
2021-01-25 16:43:57,339 INFO  org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl         - Received new token for : test001:8041
2021-01-25 16:43:57,342 INFO  org.apache.flink.yarn.YarnResourceManager                     - Received 1 containers with 1 pending container requests.
2021-01-25 16:43:57,342 INFO  org.apache.flink.yarn.YarnResourceManager                     - Removing container request Capability[<memory:2048, vCores:1>]Priority[1]. Pending container requests 0.
2021-01-25 16:43:57,358 INFO  org.apache.flink.yarn.YarnResourceManager                     - TaskExecutor container_1604371635842_0055_01_000002 will be started on test001 with TaskExecutorProcessSpec {cpuCores=1.0, frameworkHeapSize=128.000mb (134217728 bytes), frameworkOffHeapSize=128.000mb (134217728 bytes), taskHeapSize=537.600mb (563714445 bytes), taskOffHeapSize=0 bytes, networkMemSize=158.720mb (166429984 bytes), managedMemorySize=634.880mb (665719939 bytes), jvmMetaspaceSize=256.000mb (268435456 bytes), jvmOverheadSize=204.800mb (214748368 bytes)}.
2021-01-25 16:43:57,360 ERROR org.apache.flink.yarn.YarnResourceManager                     - Could not start TaskManager in container container_1604371635842_0055_01_000002.
java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;
	at org.apache.flink.runtime.entrypoint.parser.CommandLineOptions.<clinit>(CommandLineOptions.java:28)
	at org.apache.flink.runtime.clusterframework.BootstrapTools.lambda$getDynamicPropertiesAsString$0(BootstrapTools.java:646)
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
	at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1553)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:545)
	at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
	at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438)
	at org.apache.flink.runtime.clusterframework.BootstrapTools.getDynamicPropertiesAsString(BootstrapTools.java:652)
	at org.apache.flink.yarn.YarnResourceManager.createTaskExecutorLaunchContext(YarnResourceManager.java:578)
	at org.apache.flink.yarn.YarnResourceManager.startTaskExecutorInContainer(YarnResourceManager.java:384)
	at java.lang.Iterable.forEach(Iterable.java:75)
	at org.apache.flink.yarn.YarnResourceManager.lambda$onContainersAllocated$1(YarnResourceManager.java:366)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
	at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
	at akka.actor.ActorCell.invoke(ActorCell.scala:561)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2021-01-25 16:43:57,362 INFO  org.apache.flink.yarn.YarnResourceManager                     - Requesting new TaskExecutor container with resources <memory:2048, vCores:1>. Number pending requests 1.
2021-01-25 16:43:58,348 INFO  org.apache.flink.yarn.YarnResourceManager                     - Received 1 containers with 1 pending container requests.
2021-01-25 16:43:58,348 INFO  org.apache.flink.yarn.YarnResourceManager                     - Removing container request Capability[<memory:2048, vCores:1>]Priority[1]. Pending container requests 0.
2021-01-25 16:43:58,348 INFO  org.apache.flink.yarn.YarnResourceManager                     - TaskExecutor container_1604371635842_0055_01_000003 will be started on test001 with TaskExecutorProcessSpec {cpuCores=1.0, frameworkHeapSize=128.000mb (134217728 bytes), frameworkOffHeapSize=128.000mb (134217728 bytes), taskHeapSize=537.600mb (563714445 bytes), taskOffHeapSize=0 bytes, networkMemSize=158.720mb (166429984 bytes), managedMemorySize=634.880mb (665719939 bytes), jvmMetaspaceSize=256.000mb (268435456 bytes), jvmOverheadSize=204.800mb (214748368 bytes)}.
2021-01-25 16:43:58,348 ERROR org.apache.flink.yarn.YarnResourceManager                     - Could not start TaskManager in container container_1604371635842_0055_01_000003.
java.lang.NoClassDefFoundError: Could not initialize class org.apache.flink.runtime.entrypoint.parser.CommandLineOptions
	at org.apache.flink.runtime.clusterframework.BootstrapTools.lambda$getDynamicPropertiesAsString$0(BootstrapTools.java:646)
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
	at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1553)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:545)
	at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
	at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438)
	at org.apache.flink.runtime.clusterframework.BootstrapTools.getDynamicPropertiesAsString(BootstrapTools.java:652)
	at org.apache.flink.yarn.YarnResourceManager.createTaskExecutorLaunchContext(YarnResourceManager.java:578)
	at org.apache.flink.yarn.YarnResourceManager.startTaskExecutorInContainer(YarnResourceManager.java:384)
	at java.lang.Iterable.forEach(Iterable.java:75)
	at org.apache.flink.yarn.YarnResourceManager.lambda$onContainersAllocated$1(YarnResourceManager.java:366)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
	at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
	at akka.actor.ActorCell.invoke(ActorCell.scala:561)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2021-01-25 16:43:58,349 INFO  org.apache.flink.yarn.YarnResourceManager                     - Requesting new TaskExecutor container with resources <memory:2048, vCores:1>. Number pending requests 1.
2021-01-25 16:43:59,352 INFO  org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl         - Received new token for : test002:8041
2021-01-25 16:43:59,353 INFO  org.apache.flink.yarn.YarnResourceManager                     - Received 1 containers with 1 pending container requests.
2021-01-25 16:43:59,353 INFO  org.apache.flink.yarn.YarnResourceManager                     - Removing container request Capability[<memory:2048, vCores:1>]Priority[1]. Pending container requests 0.
2021-01-25 16:43:59,353 INFO  org.apache.flink.yarn.YarnResourceManager                     - TaskExecutor container_1604371635842_0055_01_000004 will be started on test002 with TaskExecutorProcessSpec {cpuCores=1.0, frameworkHeapSize=128.000mb (134217728 bytes), frameworkOffHeapSize=128.000mb (134217728 bytes), taskHeapSize=537.600mb (563714445 bytes), taskOffHeapSize=0 bytes, networkMemSize=158.720mb (166429984 bytes), managedMemorySize=634.880mb (665719939 bytes), jvmMetaspaceSize=256.000mb (268435456 bytes), jvmOverheadSize=204.800mb (214748368 bytes)}.
2021-01-25 16:43:59,354 ERROR org.apache.flink.yarn.YarnResourceManager                     - Could not start TaskManager in container container_1604371635842_0055_01_000004.
java.lang.NoClassDefFoundError: Could not initialize class org.apache.flink.runtime.entrypoint.parser.CommandLineOptions
	at org.apache.flink.runtime.clusterframework.BootstrapTools.lambda$getDynamicPropertiesAsString$0(BootstrapTools.java:646)
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
	at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1553)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:545)
	at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
	at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438)
	at org.apache.flink.runtime.clusterframework.BootstrapTools.getDynamicPropertiesAsString(BootstrapTools.java:652)
	at org.apache.flink.yarn.YarnResourceManager.createTaskExecutorLaunchContext(YarnResourceManager.java:578)
	at org.apache.flink.yarn.YarnResourceManager.startTaskExecutorInContainer(YarnResourceManager.java:384)
	at java.lang.Iterable.forEach(Iterable.java:75)
	at org.apache.flink.yarn.YarnResourceManager.lambda$onContainersAllocated$1(YarnResourceManager.java:366)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
	at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
	at akka.actor.ActorCell.invoke(ActorCell.scala:561)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2021-01-25 16:43:59,354 INFO  org.apache.flink.yarn.YarnResourceManager                     - Requesting new TaskExecutor container with resources <memory:2048, vCores:1>. Number pending requests 1.
2021-01-25 16:44:00,359 INFO  org.apache.flink.yarn.YarnResourceManager                     - Received 1 containers with 1 pending container requests.

解决思路：找到冲突jar，干掉！！！

通过排查发现是程序内jar包与集群内jar包冲突，为避免类似问题的发生，将自己的pom文件中将所有关于hadoop、flink的包全改为 provided。

自此！重新提交任务，问题解决！！！
在这里插入图片描述

机智的大脚猴
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
【Flink】Flink 资源不足 导致 提交卡死 一直处于 创建中 Memory Reserved 预留的内存
九师兄
08-13 2167
1.概述 一个环境因为资源不足，运行2个任务后，第二个任务一直处于创建中。 查看有个节点是 Memory Reserved 预留的内存，但是后来好像不是这个原因。 然后我们经过研究发现是flink任务申请的资源不足。但是申请资源不足不应该会一直报错 资源无法申请到吗，我们的是任务一直处于创建中，日志没有报错。。 ...
Flink on yarn的问题：Invalid AMRMToken
weixin_30912051的博客
11-13 1744
目前采用的Flink的版本是1.4.2，运行在yarn上，总是时不时的报错“Invalid AMRMToken from appattempt”，导致AM挂掉。 简而言之，就是AM和RM沟通的过程中，突然AM提供的Token不被认可，导致拒绝连接，进而AM挂掉。 后来发现早期版本的yarn-client是存在问题的，至少在2.6.0还是存在问题，在2.7.0之后解决了。 具...
flink提交任务状态一直在created的原因
wppwpp1的专栏
11-17 4828
1，yarn的资源不够，导致slot数不够flinkjob设置的并发度。导致数据失败。 2,hdfs存储空间不够。导致checkpoint一直失败，提交不了任务。 3.
Flink1.16 on Yarn hadoop3.1.1 提交任务失败解决
weixin_41917987的博客
05-24 478
查看了页面接口调用，返回为空数组，猜测是 Jobmanager 没有采集到 Metrics 信息，于是查看 Jobmanager 日志，报了一个 hadoop-common 包下 commons-math3 一个类找不到，说明hadoop-common 冲突了 flink 的类，将 hadoop-common 下的 commons-math3 排除掉就可以了。分析了 hadoop-common 开源和华为版本的区别，源代码改了很多，无奈只能老实用华为版本的包了。
flink采用分离模式提交报错：java.lang.NoSuchMethodError
bradyM的博客
08-20 1917
问题背景 最近在写完一个flink项目后打包到集群运行，提交时因为满脑子想着周末怎么去浪，结果提交完发现提交命令忘记了-d参数，无奈只能手动kill掉任务，然后加上-d参数重新提交，结果问题就出现了，flink任务刚刚提交到yarn就会报如下错误： 排查问题 没有办法，只能顺着错误日志去寻找问题，第一步，先看大致错误： java.lang.NoSuchMethodError: org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.newIn
Flink on Yarn启动问题
^_^TaeYang-Jade^_^
05-28 984
一、Fink on YARN启动后自动关闭 使用./bin/yarn-session.sh -n 2 -s 2 -jm 1024 -tm 1024命令启动，过几秒就关闭了 经测试发现，是由于分配的内存太大导致的，把分配的内存调小，尝试改为800 即可正常启动。很可能你修改为800的内存后，进程还是会被kill掉。 这个报错是虚拟内存（virtual memory）超出限制，被yarn ...
flink-on-yarn-1.13.2.jar
09-28
本人搭建cdh6.3.2环境时候集成flink1.13.2组件使用的flink_on_yarn-1.13.2.jar资源包，已验证可以使用，是本人制作parcel的时候的成品
flink-1.16.2下载安装包，flink-1.16.2-bin-scala-2.12.zip
02-29
- **提交作业**：通过`bin/flink run`命令提交用户定义的Flink程序。 6. **监控与调试** - **Web UI**：Flink提供了一个Web界面，用于查看任务状态、日志和监控性能指标。 - **REST API**：允许通过HTTP接口与...
flink-1.19.0-bin-scala-2.12.tgz flink-1.16.3-bin-scala-2.12.tgz
最新发布
05-21
标题和描述中提到的"flink-1.19.0-bin-scala-2.12.tgz"和"flink-1.16.3-bin-scala-2.12.tgz"是Apache Flink的两个不同版本的二进制发行版。Flink是一个开源的流处理和批处理框架，它提供了低延迟、高吞吐量的数据...
flink-1.12.0-bin-scala_2.12.tgz
05-03
此外，通过`bin`目录中的脚本，可以轻松地启动本地单节点模式、独立集群模式或在YARN、Kubernetes等资源管理系统上运行Flink。 总结来说，"flink-1.12.0-bin-scala_2.12.tgz"是一个完整的Apache Flink 1.12.0发行版...
flink-1.13.6-bin-scala_2.11.tgz
07-14
Flink 可以在多种环境中运行，包括本地模式、独立集群模式以及 YARN、Kubernetes 等资源管理器上的部署。在 "flink-1.13.6" 文件夹中，包含了启动这些模式所需的所有配置和脚本。 四、Flink 的生态系统 Flink 与...
Flink笔记4：作业提交
Cape_sir
12-02 369
提交作业前先准备好项目jar包 1、Flink UI提交 1）添加jar包 2）填写配置信息 3）运行与查看结果 使用nc命令发送消息 查看结果 4）slot与parallelism 将parallelism设置为2（这里指的是页面填写配置信息时） 此时提交任务，一直卡在created状态，说明资源不够（slot太少） 我们在部署时设置的numberOfTaskSlots为1，并行度为2时就需要两个slot。 超时报错 另外说一下parallelism的优先级 代码中设置的>页
Flink案例系列2-提交到yarn集群报错java.lang.NoClassDefFoundError FlinkKafkaConsumer
只是甲的博客
11-16 3867
文章目录一.问题描述二.解决方案2.1 代码的问题？2.2 maven依赖的问题2.3 手工上传jar包2.3.1 上传FlinkKafkaConsumer相关的jar包2.3.2 上传FlinkKafkaConsumerBase相关的jar包 一.问题描述 环境介绍: CDH 6.3.1集群 Flink版本 1.9.0 Kafka版本 2.12 本地运行无问题，提交到yarn集群报错 代码: package org.example; /* @author 只是甲 * @date 2021-
Flink in yarn 提交任务报错 created一直等待 This usually indicates the requested resource is larger than Yar
qq_45472675的博客
01-05 663
Flink on yarn 用 yarn-session.sh 启动之后 向flink提交任务时 taskManger一直在created 等待超时之后报错 This usually indicates the requested resource is larger than Yarn's max container resource limit.。注意 这块最小内存为1600MB 改小了启动会报错。这块是我改过的 本来配置4090 超过虚拟机内存。我是用虚拟机启动了三台 所以查看内存分配。
详解Flink运行架构
huahuaxiaoshao的博客
05-06 2063
1 Flink运行时的组件       Flink运行时架构主要包括四个不同的组件，它们会在运行流处理应用程序时协同工作：作业管理器（JobManager）、资源管理器（ResourceManager）、任务管理器（TaskManager）和分发器（Dispatcher）。因为Flink是用Java和Scala实现的，所以所有组件都会运行...
1.Apache Flink 1.12.0 wordcount
终回首的博客
01-27 1720
Apache Flink wordcount java案例，scala案例，python案例 所有案例都基于win10系统 版本 JDK：1.8 Scala：2.11 Python：3.7.4 Flink：1.12.0 一、Batch Job(批量计算) BatchAPI 一般用于离线计算 1 java wordcount 1 创建项目 我已经创建好，有需要的可以直接下载 https://github.com/m769963249/flink_test_java.git git clone https://
flink学习（加餐）——job任务状态变化
Yohohaha的博客
12-03 3569
我们来看一下flink官网的图： 光看图就头晕了，我们可以来整理一下。 首先flink job任务状态总共有： Created Running Finished Failed Canceled Failing Canceling Restarting Suspended 根据个人理解，我们可以把这些状态分为四大类： 起始态 中间态 最终态 过渡态 起始态表示任务启动时的状态，中间态表示任务运行时的状态，最终态表示任务最后的状态，过渡态表示任务在上述这些状态转化时中间过渡的状态。 这样我们就把九个状
The health test result for DATA_NODE_FREE_SPACE_REMAINING has become bad: Space free on data volume
萌兔兔MMQ！！
04-16 2875
告警原文： The health test result for DATA_NODE_FREE_SPACE_REMAINING has become bad: Space free on data volumes: 128.8 MiB. Capacity of data volumes: 4.2 TiB. Percentage of capacity free: 0.00%. Critical threshold: 10.00%. This health test is currently suppres
Flink on yarn 问题梳理
Jhon_yh的博客
06-30 2827
文章目录1. Caused by: org.apache.flink.client.deployment.ClusterDeploymentException: Could not deploy Yarn job cluster. 1. Caused by: org.apache.flink.client.deployment.ClusterDeploymentException: Could not deploy Yarn job cluster. 解决办法： 在 flink 配置文件里 flink-co
flink-shaded-hadoop3 和 flink-shaded-hadoop3-uber
09-03
flink-shaded-hadoop3和flink-shaded-hadoop3-uber是Apache Flink项目中与Hadoop 3.x版本集成相关的两个模块。 首先，Hadoop是一个分布式计算框架，用于处理大规模数据。而Flink是一个快速而可扩展的流式处理引擎，它可以在实时和批处理任务之间无缝切换。为了与Hadoop集成，并且能够在Flink中使用Hadoop生态系统的各种功能和工具，例如HDFS、YARN和MapReduce等，Flink提供了与Hadoop版本兼容的特殊模块。 flink-shaded-hadoop3模块是Flink所提供的一个可防止与Hadoop 3.x版本依赖冲突的模块。在Flink应用程序中，当需要使用Hadoop 3.x相关功能时，可以将flink-shaded-hadoop3模块添加到项目的依赖中。该模块会将特定版本的Hadoop 3.x依赖项重新打包，以避免与Flink自身或其他依赖项产生冲突。这样一来，Flink就能够与Hadoop 3.x版本协同工作，平滑地使用Hadoop的功能。 而flink-shaded-hadoop3-uber模块则是更加完整和庞大的用于集成Hadoop 3.x版本的模块。它将包含Hadoop 3.x依赖的所有必需库和资源等，以便于使用和编译。相比于flink-shaded-hadoop3模块，flink-shaded-hadoop3-uber模块更像是一个“全能版”，其中包含了实现与Hadoop 3.x版本深度集成所需的所有组件。这使得开发人员能够方便地构建和部署Flink应用程序，并且在与Hadoop生态系统进行交互时更加方便。 总的来说，flink-shaded-hadoop3和flink-shaded-hadoop3-uber模块都是Flink为了与Hadoop 3.x版本无缝集成，提供的两个特殊模块。它们通过重新打包Hadoop依赖，解决了可能产生的冲突问题，使得Flink能够顺利使用并利用Hadoop的功能和工具。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值