Flink 心跳服务机制

心跳机制是用于检测客户端或者服务端是否存活的一种机制，通过定期向对方发送请求方法，常见的心跳检测有两种：

1. socket 套接字SO_KEEPALIVE本身带有的心跳机制，定期向对方发送心跳包，对方在收到心跳包后会自动回复；
2. 应用自身实现心跳机制，同样也是使用定期发送请求的方式；

        Flink对各组件服务状态的监控统一使用心跳服务来管理，如同其上诉2实现机制一样，其主要也是调用方通过周期性定时发送心跳请求，接收方接收到心跳请求后作出对应的心跳响应；其内部实现是通过RPC相互调用的方式，并重置对方超时线程的调度。在Flink的各个服务组件中，ResourceManager、JobMaster、TaskExecutor三者之间存在相互检测的心跳机制：ResourceManager会主动发送心跳请求探测JobMaster、TaskExecutor是否存活；JobMaster也会主动发送心跳请求探测TaskExecutor是否存活，以便进行任务重启或者失败处理。在flink心跳机制中，其主要心跳通信核心处理如下：

1. 心跳超时：心跳服务启动后，Flink会启动一个线程来处理心跳超时事件，在设定的心跳超时时间到达后才执行线程。如果接收到组件的心跳消息，会先将该线程取消而后重新开启，重置心跳超时事件的触发。心跳服务依赖于HeartbeatListener，当在timeout时间范围内未接收到心跳响应，则会触发超时处理线程，该线程通过调用HeartbeatListener的notifyHeartbeatTimeout方法做后续心跳超时处理操作(一般是尝试重连)。
2. 心跳请求：心跳检查是双向的，一方会主动发起心跳请求，而另一方则是对心跳做出响应，两者通过RPC相互调用，重置对方的超时线程。以JobManager和TaskManager为例，JM在启动时会开启周期调度，向已经注册到JM中的TM发起心跳检查，通过RPC调用TM的requestHeartbeat方法，重置对JM超时线程的调用，表示当前JM状态正常。在TM接受到来自JM的requestHeartbeat心跳请求方法后，TM会通过RPC调用JM的receiveHeartbeat，重置对TM超时线程的调用，表示TM状态正常。

Flink中心跳服务使用的主要接口和类如下图所示：

HeartbeatTarget类：

        心跳核心目标类：其主要用来发送心跳信息，也用来接收心跳响应。心跳发送者和接收者都是该接口的子类。两者都可以携带Payload负载信息。

public interface HeartbeatTarget<I> {
   /**
    * Sends a heartbeat response to the target. Each heartbeat response can carry a payload which
    * contains additional information for the heartbeat target.
    *
    * @param heartbeatOrigin Resource ID identifying the machine for which a heartbeat shall be reported.
    * @param heartbeatPayload Payload of the heartbeat. Null indicates an empty payload.
    */
   void receiveHeartbeat(ResourceID heartbeatOrigin, I heartbeatPayload);  // 接收监控目标发送来的心跳请求响应信息

   /**
    * Requests a heartbeat from the target. Each heartbeat request can carry a payload which
    * contains additional information for the heartbeat target.
    *
    * @param requestOrigin Resource ID identifying the machine issuing the heartbeat request.
    * @param heartbeatPayload Payload of the heartbeat request. Null indicates an empty payload.
    */
   void requestHeartbeat(ResourceID requestOrigin, I heartbeatPayload);   // 向监控目标发送心跳请求
}

HeartbeatManager：

        心跳管理器用来启动或停止监视HeartbeatTarget，并报告该目标心跳超时事件。通过monitorTarget来传递并监控HeartbeatTarget，这个方法可以看做是整个服务的输入，告诉心跳服务去管理哪些目标。

public interface HeartbeatManager<I, O> extends HeartbeatTarget<I> {
    // 开始监控心跳目标，当目标心跳超时，会报告给与HeartbeatManager关联的HeartbeatListener
    void monitorTarget(ResourceID resourceID, HeartbeatTarget<O> heartbeatTarget);
    // 取消监控心跳目标，ResourceID是心跳目标的标识
    void unmonitorTarget(ResourceID resourceID);
    // 停止当前心跳管理器
    void stop();
    // 返回最近一次心跳时间，如果心跳目标被移除了则返回-1
    long getLastHeartbeatFrom(ResourceID resourceId);
}

HeartbeatListener：

    和HeartbeatManager密切相关的接口，可以看做服务的输出。主要有以下作用：

• 心跳超时通知
• 接收心跳信息中的Payload
• 检索作为心跳响应输出的Payload

public interface HeartbeatListener<I, O> {
    // 心跳超时会调用该方法
    void notifyHeartbeatTimeout(ResourceID resourceID);
    // 接收到有关心跳的payload就会执行该方法
    void reportPayload(ResourceID resourceID, I payload);
    // 检索下一个心跳消息的Payload
    O retrievePayload(ResourceID resourceID);
}

心跳服务的创建：

        集群启动时会初始化一些服务，在ClusterEntrypoint#initializeServices方法中创建心跳管理服务。其会从配置文件中提取心跳间隔heartbeat.interval和心跳超时时间heartbeat.timeout配置，并创建HeartbeatServices；

heartbeatServices = createHeartbeatServices(configuration);

protected HeartbeatServices createHeartbeatServices(Configuration configuration) {
  return HeartbeatServices.fromConfiguration(configuration);
}

public static HeartbeatServices fromConfiguration(Configuration configuration) {
    // 心跳间隔,默认10s
    long heartbeatInterval = configuration.getLong(HeartbeatManagerOptions.HEARTBEAT_INTERVAL);
    // 心跳超时时间,50s
    long heartbeatTimeout = configuration.getLong(HeartbeatManagerOptions.HEARTBEAT_TIMEOUT);

    return new HeartbeatServices(heartbeatInterval, heartbeatTimeout);
}

createHeartbeatManager和createHeartbeatManagerSender核心方法：

        这两个方法使用的两个类HeartbeatManagerImpl、HeartbeatManagerSenderImpl是整个心跳服务的关键。

        HeartbeatManagerImpl由心跳接受方、响应者(例如TM)创建，接收来自心跳发起方、请求方(JM)的心跳发送请求，其主要包含两个重要属性heartbeatListener、heartbeatTargets。heartbeatTargets是一个Map集合，key代表要发送心跳组件(例如：TM)的ID，value则是为当前组件创建的触发心跳超时的线程HeartbeatMonitor，两者一一对应，心跳超时会触发对应heartbeatListener的notifyHeartbeatTimeout方法。注意：被发起方心跳接受者监控线程的开启是在接收到请求心跳(requestHeartbeat被调用后)以后才触发的，属于被动触发。

//  外部调用者传递heartbeatTarget，并为其创建一个HeartbeatMonitor
public void monitorTarget(ResourceID resourceID, HeartbeatTarget<O> heartbeatTarget) {
   if (!stopped) {
       if (heartbeatTargets.containsKey(resourceID)) {
           log.debug("The target with resource ID {} is already been monitored.", resourceID);
       } else {
           // HeartbeatMonitor中保存目标监控处理核心类heartbeatTarget，并且将其关联对应的超时处理监听器heartbeatListener
           HeartbeatManagerImpl.HeartbeatMonitor<O> heartbeatMonitor = new HeartbeatManagerImpl.HeartbeatMonitor<>(
               resourceID,
               heartbeatTarget,
               mainThreadExecutor,
               heartbeatListener,
               heartbeatTimeoutIntervalMs);

           heartbeatTargets.put(
               resourceID,
               heartbeatMonitor);

           // check if we have stopped in the meantime (concurrent stop operation)
           if (stopped) {
               heartbeatMonitor.cancel();
               heartbeatTargets.remove(resourceID);
           }
       }
   }
}

Heartbeat monitor管理心跳目标，在timeout时间内没有接收到心跳信号，则判定心跳超时,通知给HeartbeatListener，每次接收到心跳信号则重置当前timer。

static class HeartbeatMonitor<O> implements Runnable {
    private final ResourceID resourceID; /** Resource ID of the monitored heartbeat target. */
    private final HeartbeatTarget<O> heartbeatTarget; /** Associated heartbeat target. */
    private final ScheduledExecutor scheduledExecutor;
    private final HeartbeatListener<?, ?> heartbeatListener; /** Listener which is notified about heartbeat timeouts. */
    private final long heartbeatTimeoutIntervalMs; /** Maximum heartbeat timeout interval. */
    private volatile ScheduledFuture<?> futureTimeout;
    private final AtomicReference<State> state = new AtomicReference<>(State.RUNNING);
    private volatile long lastHeartbeat;  //  最近一次接收到心跳的时间

    HeartbeatMonitor(
        ResourceID resourceID,
        HeartbeatTarget<O> heartbeatTarget,
        ScheduledExecutor scheduledExecutor,
        HeartbeatListener<?, O> heartbeatListener,
        long heartbeatTimeoutIntervalMs) {
        this.resourceID = Preconditions.checkNotNull(resourceID); // 被监控的机器ID
        this.heartbeatTarget = Preconditions.checkNotNull(heartbeatTarget); // 心跳目录核心处理类
        this.scheduledExecutor = Preconditions.checkNotNull(scheduledExecutor);
        this.heartbeatListener = Preconditions.checkNotNull(heartbeatListener); // 心跳监听器

        Preconditions.checkArgument(heartbeatTimeoutIntervalMs > 0L, "The heartbeat timeout interval has to be larger than 0.");
        this.heartbeatTimeoutIntervalMs = heartbeatTimeoutIntervalMs;
        lastHeartbeat = 0L;
        resetHeartbeatTimeout(heartbeatTimeoutIntervalMs);
    }
    /....................
    // 报告心跳
    void reportHeartbeat() { 
        lastHeartbeat = System.currentTimeMillis();  // 保留最近一次接收心跳时间
        resetHeartbeatTimeout(heartbeatTimeoutIntervalMs);  // 接收心跳后, 重置timeout线程
    }
    // 重置TIMEOUT
    void resetHeartbeatTimeout(long heartbeatTimeout) {
        if (state.get() == State.RUNNING) {
            cancelTimeout(); //先取消线程，在重新开启
            futureTimeout = scheduledExecutor.schedule(this, heartbeatTimeout, TimeUnit.MILLISECONDS); // 启动超时线程

            // Double check for concurrent accesses (e.g. a firing of the scheduled future)
            if (state.get() != State.RUNNING) {
                cancelTimeout();
            }
        }
    }
    /................
    // 心跳超时，触发listener的notifyHeartbeatTimeout
    @Override
    public void run() {
        // The heartbeat has timed out if we're in state running
        if (state.compareAndSet(State.RUNNING, State.TIMEOUT)) {
            heartbeatListener.notifyHeartbeatTimeout(resourceID);
        }
    }
}

HeartbeatManagerSenderImpl是HeartbeatManagerImpl的子类，由心跳请求方(例如JM)创建，创建后立即开启周期调度线程，每次遍历自己管理的heartbeatTarget，触发heartbeatTarget.requestHeartbeat()心跳请求，属于主动触发。

this.heartbeatPeriod = heartbeatPeriod;
mainThreadExecutor.schedule(this, 0L, TimeUnit.MILLISECONDS);

public void run() {
   if (!stopped) {
      log.debug("Trigger heartbeat request.");
      for (HeartbeatMonitor<O> heartbeatMonitor : getHeartbeatTargets()) {
         CompletableFuture<O> futurePayload = getHeartbeatListener().retrievePayload(heartbeatMonitor.getHeartbeatTargetId()); // 重新创建当前负载信息
         final HeartbeatTarget<O> heartbeatTarget = heartbeatMonitor.getHeartbeatTarget(); // 心跳核心处理类

         if (futurePayload != null) {
            CompletableFuture<Void> requestHeartbeatFuture = FutureUtils.thenAcceptAsyncIfNotDone(
               futurePayload,
               getMainThreadExecutor(),
               payload -> heartbeatTarget.requestHeartbeat(getOwnResourceID(), payload)); // 使用 心跳核心处理类 去发送请求 

            requestHeartbeatFuture.exceptionally(
               (Throwable failure) -> {
                  log.warn("Could not request the heartbeat from target {}.", heartbeatTarget, failure);

                  return null;
               });
         } else {
            heartbeatTarget.requestHeartbeat(getOwnResourceID(), null);
         }
      }
      getMainThreadExecutor().schedule(this, heartbeatPeriod, TimeUnit.MILLISECONDS); // 周期调度
   }
}

 

心跳服务示例：

1、心跳发起、请求方：JM中HeartbeatManagerSenderImpl使用

1. 接收TM的注册后，将heartbeatTarget该加入到心跳目标的集合中，心跳发起请求方JM会使用自己的心跳管理发送器HeartbeatManagerSenderImpl来周期遍历调度自己管理的heartbeatTarget，触发heartbeatTarget.requestHeartbeat()心跳请求；在这就是触发JM针对TM的heartbeatTarget.requestHeartbeat()。
2. 在requestHeartbeat中通过RPC调用taskExecutor#heartbeatFromJobManager，从JM发送心跳请求信息至TM；最终TM接受到心跳请求信息后会调用HeartbeatManagerImpl中的requestHeartbeat，启动或重置超时线程，表示JM状态正常。在该方法中又通过RPC调用JM的receiveHeartbeat。

public CompletableFuture<RegistrationResponse> registerTaskManager(
      final String taskManagerRpcAddress,
      final TaskManagerLocation taskManagerLocation,
      final Time timeout) {
   final ResourceID taskManagerId = taskManagerLocation.getResourceID();

   if (registeredTaskManagers.containsKey(taskManagerId)) {
      final RegistrationResponse response = new JMTMRegistrationSuccess(resourceId);
      return CompletableFuture.completedFuture(response);
   } else {
      return getRpcService()
         .connect(taskManagerRpcAddress, TaskExecutorGateway.class)
         .handleAsync(
            (TaskExecutorGateway taskExecutorGateway, Throwable throwable) -> {
               if (throwable != null) {
                  return new RegistrationResponse.Decline(throwable.getMessage());
               }
               
               slotPoolGateway.registerTaskManager(taskManagerId); // TaskManager注册
               registeredTaskManagers.put(taskManagerId, Tuple2.of(taskManagerLocation, taskExecutorGateway));

               // monitor the task manager as heartbeat target    // 加入心跳目标
               taskManagerHeartbeatManager.monitorTarget(taskManagerId, new HeartbeatTarget<AllocatedSlotReport>() {
                  @Override
                  public void receiveHeartbeat(ResourceID resourceID, AllocatedSlotReport payload) {
                     // the task manager will not request heartbeat, so this method will never be called currently
                  }

                  @Override
                  public void requestHeartbeat(ResourceID resourceID, AllocatedSlotReport allocatedSlotReport) {
                     taskExecutorGateway.heartbeatFromJobManager(resourceID, allocatedSlotReport);  // JM要求TM发送心跳请求
                  }
               });

               return new JMTMRegistrationSuccess(resourceId);
            },
            getMainThreadExecutor());
   }
}

TM接收到JM的RPC心跳请求后，会最终调用TM上心跳接受处理器HeartbeatManagerImpl#requestHeartbeat来进行该心跳请求的处理：

//HeartbeatManagerImpl#requestHeartbeat()
public void requestHeartbeat(final ResourceID requestOrigin, I heartbeatPayload) {
   if (!stopped) {
      log.debug("Received heartbeat request from {}.", requestOrigin);
      final HeartbeatTarget<O> heartbeatTarget = reportHeartbeat(requestOrigin); // 启动超时线程, 并获取目标heartbeatTarget, 此时的目标是JM

      if (heartbeatTarget != null) {
         if (heartbeatPayload != null) {
            heartbeatListener.reportPayload(requestOrigin, heartbeatPayload); // 监听器汇报负载情况
         }
         CompletableFuture<O> futurePayload = heartbeatListener.retrievePayload(requestOrigin); // 监听器生成当前负载信息

         if (futurePayload != null) {
            CompletableFuture<Void> sendHeartbeatFuture = FutureUtils.thenAcceptAsyncIfNotDone(
               futurePayload,
               mainThreadExecutor,    // 心跳处理核心目标类 通过RPC调用 心跳请求发起方(JM)的receiveHeartbeat   
               retrievedPayload ->    heartbeatTarget.receiveHeartbeat(getOwnResourceID(), retrievedPayload)); // heartbeatTarget实例为下面monitorTarget所注册的

            sendHeartbeatFuture.exceptionally((Throwable failure) -> {
                  log.warn("Could not send heartbeat to target with id {}.", requestOrigin, failure);
                  return null;
               });
         } else {
            heartbeatTarget.receiveHeartbeat(ownResourceID, null);
         }
      }
   }
}

//TaskExecutor上的心跳核心处理类监控注册    TaskExecutor#establishJobManagerConnection
// monitor the job manager as heartbeat target
jobManagerHeartbeatManager.monitorTarget(jobManagerResourceID, new HeartbeatTarget<AccumulatorReport>() {
   @Override
   public void receiveHeartbeat(ResourceID resourceID, AccumulatorReport payload) {
      jobMasterGateway.heartbeatFromTaskManager(resourceID, payload);
   }

   @Override
   public void requestHeartbeat(ResourceID resourceID, AccumulatorReport payload) {
      // request heartbeat will never be called on the task manager side
   }
});

2、心跳接受、响应方：TM中HeartbeatManagerImpl的使用

• TM启动后会和JM建立连接，连接成功后会为JM创建HeartbeatTarget，并重写receiveHeartbeat方法。此时，HeartbeatManagerImpl中已经创建好对应的monitor线程，只有在JM执行requestHeartbeat后，才会触发该线程的执行。
• 在receiveHeartbeat方法内部，直接通过RPC调用JM的heartbeatFromTaskManager方法，最终进入JM侧的HeartbeatManagerSenderImpl#receiveHeartbeat中，在reportHeartbeat重置JM monitor线程的触发，代表TM正常执行。

//TaskExecutor#establishJobManagerConnection
private void establishJobManagerConnection(JobID jobId, final JobMasterGateway jobMasterGateway, JMTMRegistrationSuccess registrationSuccess) {
    /.............
    ResourceID jobManagerResourceID = registrationSuccess.getResourceID();
    // monitor the job manager as heartbeat target
    jobManagerHeartbeatManager.monitorTarget(jobManagerResourceID, new HeartbeatTarget<AccumulatorReport>() {
        // TM只接收心跳请求，接受到来自JM的请求信息后，会通过RPC回调jobMasterGateway.heartbeatFromTaskManager()
        @Override
        public void receiveHeartbeat(ResourceID resourceID, AccumulatorReport payload) {
            jobMasterGateway.heartbeatFromTaskManager(resourceID, payload);
        }

        @Override
        public void requestHeartbeat(ResourceID resourceID, AccumulatorReport payload) {
            // request heartbeat will never be called on the task manager side
        }
    });
    /..............
}

// jobMaster心跳请求方实例 
// taskManagerHeartbeatManager的创建 HeartbeatManagerSenderImpl继承自HeartbeatManagerImpl
taskManagerHeartbeatManager = heartbeatServices.createHeartbeatManagerSender(
    resourceId,
    new TaskManagerHeartbeatListener(),
    getMainThreadExecutor(),
    log);
    
// 接收到来自TM的心跳响应
public void heartbeatFromTaskManager(final ResourceID resourceID, AccumulatorReport accumulatorReport) {
    taskManagerHeartbeatManager.receiveHeartbeat(resourceID, accumulatorReport); // 
}

// JM接收到来自TM的心跳响应
public void receiveHeartbeat(ResourceID heartbeatOrigin, I heartbeatPayload) {
    if (!stopped) {
        log.debug("Received heartbeat from {}.", heartbeatOrigin);
        //接收到心跳后的操作
        reportHeartbeat(heartbeatOrigin);
    
        if (heartbeatPayload != null) {
            heartbeatListener.reportPayload(heartbeatOrigin, heartbeatPayload);
        }
    }
}