本文介绍了使用Flink 1.12.5与Hive 1.1.0集成进行批处理数据迁移的步骤，包括创建Hive表、配置依赖、编写读写Hive的代码以及遇到的两个主要问题：1. SQL执行错误，原因是缺少Hive相关jar包；2. 资源分配超时，解决办法是适当设置Flink读取Hive的并行度。提供了问题解决方案和完整代码示例。
Flink 读写hive离不开flink官方文档， 官方文档地址：https://nightlies.apache.org/flink/flink-docs-release-1.12/dev/table/connectors/hive/
目录
实现目标：flink读取 hive 的 fs_table_source表数据 写入到fs_table_dst表，并统计数量
版本：
flink 1.12.5
hive 1.1.0
第一步: 环境准备
在hive上创建表
hive shell
create table IF NOT EXISTS fs_table_source ( userName STRING, age STRING, height STRING) PARTITIONED BY ( dt STRING, hr STRING);
create table IF NOT EXISTS fs_table_dst ( userName STRING, age STRING, height STRING) PARTITIONED BY ( dt STRING, hr STRING);
往表里插入数据
insert into table fs_table_source partition(dt=“2021.12.01”, hr=“12”) select “yu”, “23”, “175”;
insert into table fs_table_source partition(dt=“2021.12.01”, hr=“12”) select "zhang, “45”, “180”;
insert into table fs_table_source partition(dt=“2021.12.01”, hr=“13”) select “li”, “15”, “140”;
第二步: 加载所需的jar包
<properties>
<scala.version>2.11.8</scala.version>
<flink.version>1.12.5</flink.version>
<scala.binary.version>2.11</scala.binary.version>
</properties>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-scala_${scala.binary.version}</artifactId>
<!-- <scope>provided</scope>-->
<version>${flink.version}</version>
</dependency>
<!-- Flink Dependency -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-hive_2.11</artifactId>
<version>${flink.version}</version>
<!-- <scope>provided</scope>-->
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-planner-blink_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-api-scala-bridge_2.11</artifactId>
<version>${flink.version}</version>
<!-- <scope>provided</scope>-->
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-json</artifactId>
<version>${flink.version}</version>
</dependency>
<!-- Hive Dependency -->
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>1.1.0</version>
<!-- <scope>provided</scope>-->
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-metastore</artifactId>
<version>1.1.0</version>
<!-- <scope>provided</scope>-->
</dependency>
<dependency>
<groupId>org.apache.thrift</groupId>
<artifactId>libfb303</artifactId>
<version>0.9.2</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
flink官方文档介绍如下：
使用flink与hive集成的功能，需要将依赖的jar包添加到Flink的lib目录中如：
hive-exec-1.1.0.jar
hive-metastore-1.1.0.jar
libfb303-0.9.2.jar
flink-table-api-scala-bridge_2.11-1.12.5.jar
flink-connector-hive_2.11-1.12.5.jar
可以把idea中的jar包复制到服务器
点击Project Structrue
本地电脑的目录：
第三步 flink 批处理读写hive代码实现
package TestFlinkBatchHive
import org.apache.flink.table.api._
import org.apache.flink.table.catalog.hive.HiveCatalog
import org.apache.flink.table.api.{EnvironmentSettings, TableEnvironment}
/**
* flink 批量读写hive
*/
object TestBatchHive extends App {
override def main(args: Array[String]): Unit = {
val settings: EnvironmentSettings = EnvironmentSettings
.newInstance()
.useBlinkPlanner()
.inBatchMode()
.build()
val tableEnv: TableEnvironment = TableEnvironment.create(settings)
// 测试flink sql 好不好用
val res = tableEnv.sqlQuery("select '1', '1', 1, '1', '1', '1'")
val r = res.execute()
r.print()
val name = "myhive" // Catalog名称，定义一个唯一的名称表示
val defaultDatabase = "default" // 默认数据库名称
val hiveConfDir = "/etc/hive/conf" // hive-site.xml路径
val version = "1.1.0" // Hive版本号
val hive = new HiveCatalog(name, defaultDatabase, hiveConfDir, version)
val statementSet: StatementSet = tableEnv.createStatementSet()
tableEnv.registerCatalog(name, hive)
tableEnv.useCatalog(name)
tableEnv.getConfig.setSqlDialect(SqlDialect.HIVE)
tableEnv.useDatabase("default")
// 设置读取hive的并行度
val configuration = tableEnv.getConfig.getConfiguration
configuration.setString("table.exec.hive.infer-source-parallelism", "false")
configuration.setString("table.exec.hive.infer-source-parallelism.max", "15")
configuration.setString("table.exec.hive.fallback-mapred-reader", "true")
val sqlResult: Table = tableEnv.sqlQuery(
"""
| select username, age, height, dt, hr from fs_table_source
|""".stripMargin)
// 这种方式也可以
// statementSet.addInsert("fs_table_dst", sqlResult)
// statementSet.execute()
// register the view named 'fs_table_source.View' in the catalog named 'myhive'
// in the database named 'default'
tableEnv.createTemporaryView("fs_table_source.View", sqlResult)
// 统计总的数量
val total = tableEnv.executeSql("select count(*) from fs_table_source.View")
total.print()
// 写入hive的fs_table_dst表
tableEnv.executeSql("insert into fs_table_dst select * from fs_table_source.View")
}
}
flink 1.12连接hive方式:
设置读取hive时的并行度， 如果不设置的话最大值就是1000个并行度
val configuration = tableEnv.getConfig.getConfiguration
configuration.setString("table.exec.hive.infer-source-parallelism.max", "15")
Flink 官方文档介绍并行度设置
代码git地址
https://github.com/victor9309/TestFlinkBatchHive
第四步 打包提交到服务器执行
/usr/local/bigdata/flink-1.12.5/bin/flink run -c TestFlinkBatchHive.TestBatchHive -m yarn-cluster -p 5 ./TestFlink-1.0-SNAPSHOT.jar
执行结果（成功把hive 的 fs_table_source表数据 写入到fs_table_dst表）：
在执行过程遇到的问题
问题1：The main method caused an error: Failed to execute sql
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to execute sql
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:366)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
at org.apache.flink.client.cli.CliFrontend.lambda$main
10
(
C
l
i
F
r
o
n
t
e
n
d
.
j
a
v
a
:
1132
)
a
t
j
a
v
a
.
s
e
c
u
r
i
t
y
.
A
c
c
e
s
s
C
o
n
t
r
o
l
l
e
r
.
d
o
P
r
i
v
i
l
e
g
e
d
(
N
a
t
i
v
e
M
e
t
h
o
d
)
a
t
j
a
v
a
x
.
s
e
c
u
r
i
t
y
.
a
u
t
h
.
S
u
b
j
e
c
t
.
d
o
A
s
(
S
u
b
j
e
c
t
.
j
a
v
a
:
422
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
s
e
c
u
r
i
t
y
.
U
s
e
r
G
r
o
u
p
I
n
f
o
r
m
a
t
i
o
n
.
d
o
A
s
(
U
s
e
r
G
r
o
u
p
I
n
f
o
r
m
a
t
i
o
n
.
j
a
v
a
:
1692
)
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
r
u
n
t
i
m
e
.
s
e
c
u
r
i
t
y
.
c
o
n
t
e
x
t
s
.
H
a
d
o
o
p
S
e
c
u
r
i
t
y
C
o
n
t
e
x
t
.
r
u
n
S
e
c
u
r
e
d
(
H
a
d
o
o
p
S
e
c
u
r
i
t
y
C
o
n
t
e
x
t
.
j
a
v
a
:
41
)
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
c
l
i
e
n
t
.
c
l
i
.
C
l
i
F
r
o
n
t
e
n
d
.
m
a
i
n
(
C
l
i
F
r
o
n
t
e
n
d
.
j
a
v
a
:
1132
)
C
a
u
s
e
d
b
y
:
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
t
a
b
l
e
.
a
p
i
.
T
a
b
l
e
E
x
c
e
p
t
i
o
n
:
F
a
i
l
e
d
t
o
e
x
e
c
u
t
e
s
q
l
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
t
a
b
l
e
.
a
p
i
.
i
n
t
e
r
n
a
l
.
T
a
b
l
e
E
n
v
i
r
o
n
m
e
n
t
I
m
p
l
.
e
x
e
c
u
t
e
I
n
t
e
r
n
a
l
(
T
a
b
l
e
E
n
v
i
r
o
n
m
e
n
t
I
m
p
l
.
j
a
v
a
:
699
)
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
t
a
b
l
e
.
a
p
i
.
i
n
t
e
r
n
a
l
.
S
t
a
t
e
m
e
n
t
S
e
t
I
m
p
l
.
e
x
e
c
u
t
e
(
S
t
a
t
e
m
e
n
t
S
e
t
I
m
p
l
.
j
a
v
a
:
98
)
a
t
T
e
s
t
F
l
i
n
k
.
T
o
t
a
l
D
a
y
R
e
p
o
r
t
10(CliFrontend.java:1132) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692) at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132) Caused by: org.apache.flink.table.api.TableException: Failed to execute sql at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:699) at org.apache.flink.table.api.internal.StatementSetImpl.execute(StatementSetImpl.java:98) at TestFlink.TotalDayReport
10(CliFrontend.java:1132)atjava.security.AccessController.doPrivileged(NativeMethod)atjavax.security.auth.Subject.doAs(Subject.java:422)atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)atorg.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)atorg.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)Causedby:org.apache.flink.table.api.TableException:Failedtoexecutesqlatorg.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:699)atorg.apache.flink.table.api.internal.StatementSetImpl.execute(StatementSetImpl.java:98)atTestFlink.TotalDayReport.main(TotalDayReport.scala:43)
at TestFlink.TotalDayReport.main(TotalDayReport.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349)
… 11 more
Caused by: org.apache.flink.util.FlinkException: Failed to execute job ‘insert-into_myhive.default.fs_table_test122’.
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1918)
at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:135)
at org.apache.flink.table.planner.delegation.ExecutorBase.executeAsync(ExecutorBase.java:55)
at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:681)
… 19 more
Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph.
at org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob
9
(
R
e
s
t
C
l
u
s
t
e
r
C
l
i
e
n
t
.
j
a
v
a
:
405
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
C
o
m
p
l
e
t
a
b
l
e
F
u
t
u
r
e
.
u
n
i
E
x
c
e
p
t
i
o
n
a
l
l
y
(
C
o
m
p
l
e
t
a
b
l
e
F
u
t
u
r
e
.
j
a
v
a
:
870
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
C
o
m
p
l
e
t
a
b
l
e
F
u
t
u
r
e
9(RestClusterClient.java:405) at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870) at java.util.concurrent.CompletableFuture
9(RestClusterClient.java:405)atjava.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)atjava.util.concurrent.CompletableFutureUniExceptionally.tryFire(CompletableFuture.java:852)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay
9
(
F
u
t
u
r
e
U
t
i
l
s
.
j
a
v
a
:
390
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
C
o
m
p
l
e
t
a
b
l
e
F
u
t
u
r
e
.
u
n
i
W
h
e
n
C
o
m
p
l
e
t
e
(
C
o
m
p
l
e
t
a
b
l
e
F
u
t
u
r
e
.
j
a
v
a
:
760
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
C
o
m
p
l
e
t
a
b
l
e
F
u
t
u
r
e
9(FutureUtils.java:390) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture
9(FutureUtils.java:390)atjava.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)atjava.util.concurrent.CompletableFutureUniWhenComplete.tryFire(CompletableFuture.java:736)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at org.apache.flink.runtime.rest.RestClient.lambda$submitRequest
1
(
R
e
s
t
C
l
i
e
n
t
.
j
a
v
a
:
430
)
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
s
h
a
d
e
d
.
n
e
t
t
y
4.
i
o
.
n
e
t
t
y
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
n
o
t
i
f
y
L
i
s
t
e
n
e
r
0
(
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
j
a
v
a
:
577
)
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
s
h
a
d
e
d
.
n
e
t
t
y
4.
i
o
.
n
e
t
t
y
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
n
o
t
i
f
y
L
i
s
t
e
n
e
r
s
0
(
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
j
a
v
a
:
570
)
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
s
h
a
d
e
d
.
n
e
t
t
y
4.
i
o
.
n
e
t
t
y
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
n
o
t
i
f
y
L
i
s
t
e
n
e
r
s
N
o
w
(
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
j
a
v
a
:
549
)
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
s
h
a
d
e
d
.
n
e
t
t
y
4.
i
o
.
n
e
t
t
y
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
n
o
t
i
f
y
L
i
s
t
e
n
e
r
s
(
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
j
a
v
a
:
490
)
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
s
h
a
d
e
d
.
n
e
t
t
y
4.
i
o
.
n
e
t
t
y
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
s
e
t
V
a
l
u
e
0
(
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
j
a
v
a
:
615
)
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
s
h
a
d
e
d
.
n
e
t
t
y
4.
i
o
.
n
e
t
t
y
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
s
e
t
F
a
i
l
u
r
e
0
(
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
j
a
v
a
:
608
)
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
s
h
a
d
e
d
.
n
e
t
t
y
4.
i
o
.
n
e
t
t
y
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
t
r
y
F
a
i
l
u
r
e
(
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
j
a
v
a
:
117
)
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
s
h
a
d
e
d
.
n
e
t
t
y
4.
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
n
i
o
.
A
b
s
t
r
a
c
t
N
i
o
C
h
a
n
n
e
l
1(RestClient.java:430) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel
1(RestClient.java:430)atorg.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)atorg.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)atorg.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)atorg.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)atorg.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)atorg.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)atorg.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)atorg.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannelAbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap
2.
r
u
n
(
T
h
r
e
a
d
E
x
e
c
u
t
o
r
M
a
p
.
j
a
v
a
:
74
)
a
t
j
a
v
a
.
l
a
n
g
.
T
h
r
e
a
d
.
r
u
n
(
T
h
r
e
a
d
.
j
a
v
a
:
748
)
C
a
u
s
e
d
b
y
:
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
r
u
n
t
i
m
e
.
c
o
n
c
u
r
r
e
n
t
.
F
u
t
u
r
e
U
t
i
l
s
2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.flink.runtime.concurrent.FutureUtils
2.run(ThreadExecutorMap.java:74)atjava.lang.Thread.run(Thread.java:748)Causedby:org.apache.flink.runtime.concurrent.FutureUtilsRetryException: Could not complete the operation. Number of retries has been exhausted.
at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay
9
(
F
u
t
u
r
e
U
t
i
l
s
.
j
a
v
a
:
386
)
.
.
.
21
m
o
r
e
C
a
u
s
e
d
b
y
:
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
C
o
m
p
l
e
t
i
o
n
E
x
c
e
p
t
i
o
n
:
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
s
h
a
d
e
d
.
n
e
t
t
y
4.
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
9(FutureUtils.java:386) ... 21 more Caused by: java.util.concurrent.CompletionException: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel
9(FutureUtils.java:386)...21moreCausedby:java.util.concurrent.CompletionException:org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelAnnotatedConnectException: Connection refused: /10.1.10.211:8089
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943)
at java.util.concurrent.CompletableFuture
U
n
i
C
o
m
p
o
s
e
.
t
r
y
F
i
r
e
(
C
o
m
p
l
e
t
a
b
l
e
F
u
t
u
r
e
.
j
a
v
a
:
926
)
.
.
.
19
m
o
r
e
C
a
u
s
e
d
b
y
:
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
s
h
a
d
e
d
.
n
e
t
t
y
4.
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
UniCompose.tryFire(CompletableFuture.java:926) ... 19 more Caused by: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel
UniCompose.tryFire(CompletableFuture.java:926)...19moreCausedby:org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelAnnotatedConnectException: Connection refused: /10.1.10.211:8089
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:748)
解决方案： 缺少jar包
问题2：Could not allocate the required slot within slot request timeout
org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:118) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:80) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:233) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:224) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:215) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:666) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyTaskFailure(UpdateSchedulerNgOnInternalFailuresListener.java:56) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.executiongraph.ExecutionGraph.notifySchedulerNgAboutInternalTaskFailure(ExecutionGraph.java:1869) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1463) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1403) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.executiongraph.Execution.markFailed(Execution.java:1231) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.executiongraph.ExecutionVertex.markFailed(ExecutionVertex.java:758) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.scheduler.DefaultExecutionVertexOperations.markFailed(DefaultExecutionVertexOperations.java:41) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskDeploymentFailure(DefaultScheduler.java:522) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError
6
(
D
e
f
a
u
l
t
S
c
h
e
d
u
l
e
r
.
j
a
v
a
:
507
)
[
T
e
s
t
F
l
i
n
k
−
1.0
−
S
N
A
P
S
H
O
T
.
j
a
r
:
?
]
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
C
o
m
p
l
e
t
a
b
l
e
F
u
t
u
r
e
.
u
n
i
H
a
n
d
l
e
(
C
o
m
p
l
e
t
a
b
l
e
F
u
t
u
r
e
.
j
a
v
a
:
822
)
[
?
:
1.8.
0
1
44
]
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
C
o
m
p
l
e
t
a
b
l
e
F
u
t
u
r
e
6(DefaultScheduler.java:507) ~[TestFlink-1.0-SNAPSHOT.jar:?] at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) ~[?:1.8.0_144] at java.util.concurrent.CompletableFuture
6(DefaultScheduler.java:507) [TestFlink−1.0−SNAPSHOT.jar:?]atjava.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) [?:1.8.0144]atjava.util.concurrent.CompletableFutureUniHandle.tryFire(CompletableFuture.java:797) ~[?:1.8.0_144]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_144]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_144]
at org.apache.flink.runtime.scheduler.SharedSlot.cancelLogicalSlotRequest(SharedSlot.java:223) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.scheduler.SlotSharingExecutionSlotAllocator.cancelLogicalSlotRequest(SlotSharingExecutionSlotAllocator.java:168) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.scheduler.SharingPhysicalSlotRequestBulk.cancel(SharingPhysicalSlotRequestBulk.java:86) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkWithTimestamp.cancel(PhysicalSlotRequestBulkWithTimestamp.java:66) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda$schedulePendingRequestBulkWithTimestampCheck
0
(
P
h
y
s
i
c
a
l
S
l
o
t
R
e
q
u
e
s
t
B
u
l
k
C
h
e
c
k
e
r
I
m
p
l
.
j
a
v
a
:
91
)
[
T
e
s
t
F
l
i
n
k
−
1.0
−
S
N
A
P
S
H
O
T
.
j
a
r
:
?
]
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
E
x
e
c
u
t
o
r
s
0(PhysicalSlotRequestBulkCheckerImpl.java:91) ~[TestFlink-1.0-SNAPSHOT.jar:?] at java.util.concurrent.Executors
0(PhysicalSlotRequestBulkCheckerImpl.java:91) [TestFlink−1.0−SNAPSHOT.jar:?]atjava.util.concurrent.ExecutorsRunnableAdapter.call(Executors.java:511) ~[?:1.8.0_144]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_144]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:440) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:208) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[TestFlink-1.0-SNAPSHOT.jar:?]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [TestFlink-1.0-SNAPSHOT.jar:?]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [TestFlink-1.0-SNAPSHOT.jar:?]
at scala.PartialFunction
c
l
a
s
s
.
a
p
p
l
y
O
r
E
l
s
e
(
P
a
r
t
i
a
l
F
u
n
c
t
i
o
n
.
s
c
a
l
a
:
123
)
[
T
e
s
t
F
l
i
n
k
−
1.0
−
S
N
A
P
S
H
O
T
.
j
a
r
:
?
]
a
t
a
k
k
a
.
j
a
p
i
.
p
f
.
U
n
i
t
C
a
s
e
S
t
a
t
e
m
e
n
t
.
a
p
p
l
y
O
r
E
l
s
e
(
C
a
s
e
S
t
a
t
e
m
e
n
t
s
.
s
c
a
l
a
:
21
)
[
T
e
s
t
F
l
i
n
k
−
1.0
−
S
N
A
P
S
H
O
T
.
j
a
r
:
?
]
a
t
s
c
a
l
a
.
P
a
r
t
i
a
l
F
u
n
c
t
i
o
n
class.applyOrElse(PartialFunction.scala:123) [TestFlink-1.0-SNAPSHOT.jar:?] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [TestFlink-1.0-SNAPSHOT.jar:?] at scala.PartialFunction
class.applyOrElse(PartialFunction.scala:123)[TestFlink−1.0−SNAPSHOT.jar:?]atakka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)[TestFlink−1.0−SNAPSHOT.jar:?]atscala.PartialFunctionOrElse.applyOrElse(PartialFunction.scala:170) [TestFlink-1.0-SNAPSHOT.jar:?]
at scala.PartialFunction
O
r
E
l
s
e
.
a
p
p
l
y
O
r
E
l
s
e
(
P
a
r
t
i
a
l
F
u
n
c
t
i
o
n
.
s
c
a
l
a
:
171
)
[
T
e
s
t
F
l
i
n
k
−
1.0
−
S
N
A
P
S
H
O
T
.
j
a
r
:
?
]
a
t
s
c
a
l
a
.
P
a
r
t
i
a
l
F
u
n
c
t
i
o
n
OrElse.applyOrElse(PartialFunction.scala:171) [TestFlink-1.0-SNAPSHOT.jar:?] at scala.PartialFunction
OrElse.applyOrElse(PartialFunction.scala:171)[TestFlink−1.0−SNAPSHOT.jar:?]atscala.PartialFunctionOrElse.applyOrElse(PartialFunction.scala:171) [TestFlink-1.0-SNAPSHOT.jar:?]
at akka.actor.Actor
c
l
a
s
s
.
a
r
o
u
n
d
R
e
c
e
i
v
e
(
A
c
t
o
r
.
s
c
a
l
a
:
517
)
[
T
e
s
t
F
l
i
n
k
−
1.0
−
S
N
A
P
S
H
O
T
.
j
a
r
:
?
]
a
t
a
k
k
a
.
a
c
t
o
r
.
A
b
s
t
r
a
c
t
A
c
t
o
r
.
a
r
o
u
n
d
R
e
c
e
i
v
e
(
A
b
s
t
r
a
c
t
A
c
t
o
r
.
s
c
a
l
a
:
225
)
[
T
e
s
t
F
l
i
n
k
−
1.0
−
S
N
A
P
S
H
O
T
.
j
a
r
:
?
]
a
t
a
k
k
a
.
a
c
t
o
r
.
A
c
t
o
r
C
e
l
l
.
r
e
c
e
i
v
e
M
e
s
s
a
g
e
(
A
c
t
o
r
C
e
l
l
.
s
c
a
l
a
:
592
)
[
T
e
s
t
F
l
i
n
k
−
1.0
−
S
N
A
P
S
H
O
T
.
j
a
r
:
?
]
a
t
a
k
k
a
.
a
c
t
o
r
.
A
c
t
o
r
C
e
l
l
.
i
n
v
o
k
e
(
A
c
t
o
r
C
e
l
l
.
s
c
a
l
a
:
561
)
[
T
e
s
t
F
l
i
n
k
−
1.0
−
S
N
A
P
S
H
O
T
.
j
a
r
:
?
]
a
t
a
k
k
a
.
d
i
s
p
a
t
c
h
.
M
a
i
l
b
o
x
.
p
r
o
c
e
s
s
M
a
i
l
b
o
x
(
M
a
i
l
b
o
x
.
s
c
a
l
a
:
258
)
[
T
e
s
t
F
l
i
n
k
−
1.0
−
S
N
A
P
S
H
O
T
.
j
a
r
:
?
]
a
t
a
k
k
a
.
d
i
s
p
a
t
c
h
.
M
a
i
l
b
o
x
.
r
u
n
(
M
a
i
l
b
o
x
.
s
c
a
l
a
:
225
)
[
T
e
s
t
F
l
i
n
k
−
1.0
−
S
N
A
P
S
H
O
T
.
j
a
r
:
?
]
a
t
a
k
k
a
.
d
i
s
p
a
t
c
h
.
M
a
i
l
b
o
x
.
e
x
e
c
(
M
a
i
l
b
o
x
.
s
c
a
l
a
:
235
)
[
T
e
s
t
F
l
i
n
k
−
1.0
−
S
N
A
P
S
H
O
T
.
j
a
r
:
?
]
a
t
a
k
k
a
.
d
i
s
p
a
t
c
h
.
f
o
r
k
j
o
i
n
.
F
o
r
k
J
o
i
n
T
a
s
k
.
d
o
E
x
e
c
(
F
o
r
k
J
o
i
n
T
a
s
k
.
j
a
v
a
:
260
)
[
T
e
s
t
F
l
i
n
k
−
1.0
−
S
N
A
P
S
H
O
T
.
j
a
r
:
?
]
a
t
a
k
k
a
.
d
i
s
p
a
t
c
h
.
f
o
r
k
j
o
i
n
.
F
o
r
k
J
o
i
n
P
o
o
l
class.aroundReceive(Actor.scala:517) [TestFlink-1.0-SNAPSHOT.jar:?] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [TestFlink-1.0-SNAPSHOT.jar:?] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [TestFlink-1.0-SNAPSHOT.jar:?] at akka.actor.ActorCell.invoke(ActorCell.scala:561) [TestFlink-1.0-SNAPSHOT.jar:?] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [TestFlink-1.0-SNAPSHOT.jar:?] at akka.dispatch.Mailbox.run(Mailbox.scala:225) [TestFlink-1.0-SNAPSHOT.jar:?] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [TestFlink-1.0-SNAPSHOT.jar:?] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [TestFlink-1.0-SNAPSHOT.jar:?] at akka.dispatch.forkjoin.ForkJoinPool
class.aroundReceive(Actor.scala:517)[TestFlink−1.0−SNAPSHOT.jar:?]atakka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)[TestFlink−1.0−SNAPSHOT.jar:?]atakka.actor.ActorCell.receiveMessage(ActorCell.scala:592)[TestFlink−1.0−SNAPSHOT.jar:?]atakka.actor.ActorCell.invoke(ActorCell.scala:561)[TestFlink−1.0−SNAPSHOT.jar:?]atakka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)[TestFlink−1.0−SNAPSHOT.jar:?]atakka.dispatch.Mailbox.run(Mailbox.scala:225)[TestFlink−1.0−SNAPSHOT.jar:?]atakka.dispatch.Mailbox.exec(Mailbox.scala:235)[TestFlink−1.0−SNAPSHOT.jar:?]atakka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[TestFlink−1.0−SNAPSHOT.jar:?]atakka.dispatch.forkjoin.ForkJoinPoolWorkQueue.runTask(ForkJoinPool.java:1339) [TestFlink-1.0-SNAPSHOT.jar:?]
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [TestFlink-1.0-SNAPSHOT.jar:?]
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [TestFlink-1.0-SNAPSHOT.jar:?]
Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeout
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_144]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_144]
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) ~[?:1.8.0_144]
at java.util.concurrent.CompletableFuture
U
n
i
A
p
p
l
y
.
t
r
y
F
i
r
e
(
C
o
m
p
l
e
t
a
b
l
e
F
u
t
u
r
e
.
j
a
v
a
:
577
)
[
?
:
1.8.
0
1
44
]
.
.
.
31
m
o
r
e
C
a
u
s
e
d
b
y
:
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
r
u
n
t
i
m
e
.
j
o
b
m
a
n
a
g
e
r
.
s
c
h
e
d
u
l
e
r
.
N
o
R
e
s
o
u
r
c
e
A
v
a
i
l
a
b
l
e
E
x
c
e
p
t
i
o
n
:
S
l
o
t
r
e
q
u
e
s
t
b
u
l
k
i
s
n
o
t
f
u
l
f
i
l
l
a
b
l
e
!
C
o
u
l
d
n
o
t
a
l
l
o
c
a
t
e
t
h
e
r
e
q
u
i
r
e
d
s
l
o
t
w
i
t
h
i
n
s
l
o
t
r
e
q
u
e
s
t
t
i
m
e
o
u
t
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
r
u
n
t
i
m
e
.
j
o
b
m
a
s
t
e
r
.
s
l
o
t
p
o
o
l
.
P
h
y
s
i
c
a
l
S
l
o
t
R
e
q
u
e
s
t
B
u
l
k
C
h
e
c
k
e
r
I
m
p
l
.
l
a
m
b
d
a
UniApply.tryFire(CompletableFuture.java:577) ~[?:1.8.0_144] ... 31 more Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeout at org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda
UniApply.tryFire(CompletableFuture.java:577) [?:1.8.0144]...31moreCausedby:org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:Slotrequestbulkisnotfulfillable!Couldnotallocatetherequiredslotwithinslotrequesttimeoutatorg.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambdaschedulePendingRequestBulkWithTimestampCheck
0
(
P
h
y
s
i
c
a
l
S
l
o
t
R
e
q
u
e
s
t
B
u
l
k
C
h
e
c
k
e
r
I
m
p
l
.
j
a
v
a
:
86
)
[
T
e
s
t
F
l
i
n
k
−
1.0
−
S
N
A
P
S
H
O
T
.
j
a
r
:
?
]
.
.
.
24
m
o
r
e
C
a
u
s
e
d
b
y
:
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
T
i
m
e
o
u
t
E
x
c
e
p
t
i
o
n
:
T
i
m
e
o
u
t
h
a
s
o
c
c
u
r
r
e
d
:
300000
m
s
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
i
n
k
.
r
u
n
t
i
m
e
.
j
o
b
m
a
s
t
e
r
.
s
l
o
t
p
o
o
l
.
P
h
y
s
i
c
a
l
S
l
o
t
R
e
q
u
e
s
t
B
u
l
k
C
h
e
c
k
e
r
I
m
p
l
.
l
a
m
b
d
a
0(PhysicalSlotRequestBulkCheckerImpl.java:86) ~[TestFlink-1.0-SNAPSHOT.jar:?] ... 24 more Caused by: java.util.concurrent.TimeoutException: Timeout has occurred: 300000 ms at org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda
0(PhysicalSlotRequestBulkCheckerImpl.java:86) [TestFlink−1.0−SNAPSHOT.jar:?]...24moreCausedby:java.util.concurrent.TimeoutException:Timeouthasoccurred:300000msatorg.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambdaschedulePendingRequestBulkWithTimestampCheck$0(PhysicalSlotRequestBulkCheckerImpl.java:86) ~[TestFlink-1.0-SNAPSHOT.jar:?]
… 24 more
超时了， 读取hive的并行度太大了，没有设置默认值
解决方案：
val configuration = tableEnv.getConfig.getConfiguration
configuration.setString("table.exec.hive.infer-source-parallelism.max", "15")
configuration.setString("table.exec.hive.fallback-mapred-reader", "true")
Flink 官方文档介绍并行度设置
423
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
