JStorm与Storm源码分析（二）--任务分配，assignment

mk-assignments主要功能就是产生Executor与节点+端口的对应关系,将Executor分配到某个节点的某个端口上，以及进行相应的调度处理。代码注释如下：

;;参数nimbus为nimbus-data对象,:scratch-topology-id为需要重新调度的Topology的id
(defnk mk-assignments [nimbus :scratch-topology-id nil]
  (let [conf (:conf nimbus);;分别从nimbus-data中获取conf,storm-cluster-state和inimbus对象,并将其保存为临时变量
        storm-cluster-state (:storm-cluster-state nimbus)
        ^INimbus inimbus (:inimbus nimbus) 
        ;;从zk中读取所有活跃的Topologies,获取他们id的集合
        topology-ids (.active-storms storm-cluster-state)
        ;;根据前面得到的Topology-id的集合,对每一个id调用read-topology-details方法
        ;;从参数nimbus-data中获取topology-details信息,并以<topology-id,topology-details>保存在集合中
        topologies (into {} (for [tid topology-ids] {tid (read-topology-details nimbus tid)}))
        ;;利用前面得到的<topology-id,topology-details>集合创建Topologies对象
        topologies (Topologies. topologies)
        ;;读取所有已经分配资源的Topology的id的集合。
        assigned-topology-ids (.assignments storm-cluster-state nil)
        existing-assignments (into {} (for [tid assigned-topology-ids] ;; 对于那些已经分配资源但需要重新调度的Topology(由scratch-topology-id指定), ;; 我们忽略其之前的分配，故之前分配占用的所有slot将被视为空闲slot(空闲资源),可重新被调度使用。 (when (or (nil? scratch-topology-id) (not= tid scratch-topology-id)) {tid (.assignment-info storm-cluster-state tid nil)})))
        ;; 调用compute-new-topology->executor->node+port方法为所有Topologies计算新的调度,
        ;; 并返回topology->executor->node+port
        topology->executor->node+port (compute-new-topology->executor->node+port
                                       nimbus
                                       existing-assignments
                                       topologies
                                       scratch-topology-id)

        ;;获取当前系统时间(秒)
        now-secs (current-time-secs)
        ;;调用basic-supervisor-details-map方法获取ZooKeeper中所有的SupervisorInfo信息,
        ;;然后将其转换为<supervisor-id,SupervisorDetails>集合,具体操作看1
        basic-supervisor-details-map (basic-supervisor-details-map storm-cluster-state)

        ;; 对topology->executor->node+port中各项进行处理,通过添加开始时间等构建最终的作业
        ;; 返回得到<topology-id Assignment>集合
        new-assignments (into {} (for [[topology-id executor->node+port] topology->executor->node+port ;;根据topology-id获取Topology的任务分配情况 :let [existing-assignment (get existing-assignments topology-id) ;;从executor->node+port信息中提取所有的节点信息 all-nodes (->> executor->node+port vals (map first) set) ;;根据all-nodes获取每个节点的主机名信息,并返回一个<node hostname>集合 node->host (->> all-nodes (mapcat (fn [node] (if-let [host (.getHostName inimbus basic-supervisor-details-map node)] [[node host]] ))) (into {})) ;;将上述获取到的<node, hostname>集合和<node, host>集合,得到所有<node host>关系. ;;如果存在相同的node,则与其对应的主机名将采用<node,hostname>集合中的值 all-node->host (merge (:node->host existing-assignment) node->host) ;;调用changed-executors,通过将executor->node+port信息同existing-assignment中的信息进行比对, ;;计算出所有被重新分配的Executor reassign-executors (changed-executors (:executor->node+port existing-assignment) executor->node+port) ;;通过将已经存在的assignment中的executor->start-time-secs信息 ;;与所有被重新分配的通过将已经存在的assignment中的executor->start-time-secs进行合并, ;;获得最新的所有<executor,start-time-secs>集合 start-times (merge (:executor->start-time-secs existing-assignment) (into {} (for [id reassign-executors] [id now-secs] )))]] ;;创建Assignment对象,参数分别为该Topology在Nimbus服务器上的root文件夹路径、 ;;<node,host>集合、新的executor->node+port映射关系以及新的<executor,start-time-secs>集合 {topology-id (Assignment. (master-stormdist-root conf topology-id) (select-keys all-node->host all-nodes) executor->node+port start-times)}))]

    ;; 对于新计算的<topology-id,assignment>集合中的每一项,比较其新的调度与当前运行时的调度之间是否发生了变化
    ;; 如果没有发生变化,就打印一条记录;否则将该Topology在ZooKeeper中保存的调度结果更新assignment
    (doseq [[topology-id assignment] new-assignments
            :let [existing-assignment (get existing-assignments topology-id) topology-details (.getById topologies topology-id)]]
      (if (= existing-assignment assignment)
        (log-debug "Assignment for " topology-id " hasn't changed")
        (do (log-message "Setting new assignment for topology id " topology-id ": " (pr-str assignment)) (.set-assignment! storm-cluster-state topology-id assignment) )))
    ;;对于前面得到的new-assignments中的每一项,首先计算出新增的slot,
    ;;再将其转换化为worker-slot对象,返回的是<topology-id,worker-slot>集合,
    ;;最后调用inimbus的assignSlots方法来分配slot
    (->> new-assignments
          (map (fn [[topology-id assignment]] (let [existing-assignment (get existing-assignments topology-id)] [topology-id (map to-worker-slot (newly-added-slots existing-assignment assignment))] )))
          (into {})
          (.assignSlots inimbus topologies))
    ))

在该过程中，如果某个Slot不存在Executor的超时,而Supervisor的ZooKeeper心跳超时时,
认为当前Slot依然有效,可以分配认为.最坏的情况就是这些分配过去的Executor会超时,在下一轮的分配过程中,则不会分配。

mk-assignments的详细过程如下:
1.从ZooKeeper中读取所有活跃的Topologies
2.从ZooKeeper中读取当前的assignments,获取所有已经分配资源的Topology的id的集合。
3.对Topologies进行新assignments
3.1通过调用computer-topology->executors取出所有已经assignment的topology的executors
3.2 update-all-heartbeats,对每一个Topology进行更新心跳
3.3调用compute-topology->alive-executors过滤topology->executors,保留alive的executors
3.4调用compute->supervisor->dead-ports找出dead ports
3.5调用compute-topology->scheduler-assignment转换ZooKeeper中的assignment为SchedulerAssignment
3.6通过调用missing-assignment-topologies找出需要从新assign的Topology
3.7通过调用all-scheduling-slots得到所有Supervisor节点中可用的slot数量
3.8调用read-all-supervisor-details得到所有的Supervisor节点SupervisorDetails
3.9获取backtype.storm.scheduler.Cluster
3.10调用scheduler.schedule分配所有的Topologies
3.11通过调用compute-topology->executor->node_port转换SchedulerAssignment为Assignment,输出ressign日志
4.通过将已经存在的assignment中的executor->start-time-secs信息与所有被重新分配的通过将已经存在的assignment中的executor->start-time-secs进行合并,获得最新的所有<executor,start-time-secs>集合,补充start-times等信息,获得new-assignments。
5.调用set-assignment!将新的assignment结果写入ZooKeeper.

mk-assignments负责对当前集群中所有Topology进行新一轮的任务调度。先检查已运行的Topology所占用的资源，判断它们是否有问题以及重新分配；根据系统当前的可用资源,为新提交的Topology分配任务。mk-assignments会将所有assignment信息更新到ZooKeeper中，Supervisor周期性地检查这些分配信息，并根据这些分配信息做相应的调度处理。

注：学习李明老师Storm源码分析和陈敏敏老师Storm技术内幕与大数据实现的笔记的整理。
欢迎关注下面二维码进行技术交流：