resourcemanager.taskmanager-timeout 默认30000毫秒 taskmanager无任务运行就会被回收,下次提交任务时又会重新向yarn申请container启动taskmanager 耗时很大
resourcemanager.taskmanager-timeout 增加为86400000 保留一天
SlotManagerConfiguration.fromConfiguration(){
final Time taskManagerTimeout =
Time.milliseconds(
configuration.getLong(ResourceManagerOptions.TASK_MANAGER_TIMEOUT));
}
public DeclarativeSlotManager(){
new TaskExecutorManager(
slotManagerConfiguration.getDefaultWorkerResourceSpec(),
slotManagerConfiguration.getNumSlotsPerWorker(),
slotManagerConfiguration.getMaxSlotNum(),
slotManagerConfiguration.isWaitResultConsumedBeforeRelease(),
slotManagerConfiguration.getRedundantTaskManagerNum(),
slotManagerConfiguration.getTaskManagerTimeout(),
scheduledExecutor,
executor,
resourceActions);
}
TaskExecutorManager(){
taskManagerTimeoutsAndRedundancyCheck =
scheduledExecutor.scheduleWithFixedDelay(
() ->
mainThreadExecutor.execute(
this::checkTaskManagerTimeoutsAndRedundancy),
0L,
taskManagerTimeout.toMilliseconds(),
TimeUnit.MILLISECONDS);
}
taskManagerTimeout.toMilliseconds() 检查一次 checkTaskManagerTimeoutsAndRedundancy taskmanager空闲时就释放