Yarn的状态机框架分析

Yarn的状态机框架分析

什么是状态机

状态机(State Machine)，是有限状态自动机的简称。简单解释：给定一个状态机，同时给定它的当前状态和输入，那么输出状态是可以明确的运算出来的。

使用状态机的目的

当时实体与其相关的事件越多，它的状态和处理就越复杂，状态机负责合理地组织状态转换的流程，快速找到指定初始状态和事件类型对应的状态转换方法。

yarn中的状态机

YARN将各种处理逻辑抽象成事件和对应事件调度器， 并将每类事件的处理过程分割成多个步骤， 用有限状态机表示。

在Yarn中，App、AppAttempt、Container、Node都可以使用状态机表示。其中，

• RMApp:用于维护一个Application的生命周期；
• RMAppAttempt:用于维护一次尝试运行的生命周期；
• RMContainer:用于维护一个已分配的资源最小单位Container的生命周期；
• RMNode:用于维护一个NodeManager的生命周期。

关于yarn的处理过程

yarn中的状态机是与事件驱动模型结合使用的，在Service框架中注册调度处理器，然后在处理器使用状态机。

整个处理过程大致为： 处理请求会作为事件进入系统， 由异步调度器（AsyncDispatcher） 负责传递给相应事件调度器（Event Handler） 。 该事件调度器可能将该事件转发给另外一个事件调度器， 也可能交给一个带有有限状态机的事件处理器， 其处理结果也以事件的形式输出给中央异步调度器。 而新的事件会再次被中央异步调度器转发给下一个事件调度器， 直至处理完成（达到终止条件） 。

状态机的使用流程分为两步：

第一步：Service注册Handler。

第二步：Handler使用状态机。

状态机举例

例如，对于一个应用RMApp而言，RMApp存在一个初始状态，处理事件时，会根据事件类型匹配对应的转换类Transition，将RMApp从初始状态转化成目标状态。RMApp经历的流程为：初始状态–>转换方法–>目标状态，将其所有流程汇总起来，就是状态机。

下图为RMApp状态的转换流程。乍看还是比较复杂的。

各个使用状态机的事件处理类（实现EventHandler接口）依赖状态工厂类StateMachineFactory，完成了状态机的初始化。将状态机的处理流程，通过链表结构TransitionsListNode进行组织起来。以处理RMAppEventType事件来说，在RMActiveServices的serviceInit方法中注册RMAppEventType事件的调度处理器ApplicationEventDispatcher

// Register event handler for RmAppEvents
rmDispatcher.register(RMAppEventType.class,
    new ApplicationEventDispatcher(rmContext));

ApplicationEventDispatcher中去让RMApp的实现类RMAppImpl处理事件。

@Override
public void handle(RMAppEvent event) {
  ApplicationId appID = event.getApplicationId();
  RMApp rmApp = this.rmContext.getRMApps().get(appID);
  if (rmApp != null) {
    try {
      rmApp.handle(event);
    } catch (Throwable t) {
      LOG.error("Error in handling event type " + event.getType()
          + " for application " + appID, t);
    }
  }
}

RMAppImpl的handle方法中通过调用状态的的转换方法doTransition来改变RMApp对象状态。

public void handle(RMAppEvent event) {

    this.writeLock.lock();

    try {
      ApplicationId appID = event.getApplicationId();
      LOG.debug("Processing event for " + appID + " of type "
          + event.getType());
      final RMAppState oldState = getState();
      try {
        /* keep the master in sync with the state machine */
        this.stateMachine.doTransition(event.getType(), event);
      } catch (InvalidStateTransitionException e) {
        LOG.error("App: " + appID
            + " can't handle this event at current state", e);
        onInvalidStateTransition(event.getType(), oldState);
      }

      // Log at INFO if we're not recovering or not in a terminal state.
      // Log at DEBUG otherwise.
      if ((oldState != getState()) &&
          (((recoveredFinalState == null)) ||
            (event.getType() != RMAppEventType.RECOVER))) {
        LOG.info(String.format(STATE_CHANGE_MESSAGE, appID, oldState,
            getState(), event.getType()));
      } else if ((oldState != getState()) && LOG.isDebugEnabled()) {
        LOG.debug(String.format(STATE_CHANGE_MESSAGE, appID, oldState,
            getState(), event.getType()));
      }
    } finally {
      this.writeLock.unlock();
    }
  }

状态机的核心StateMachineFactory类

StateMachineFactory类中的stateMachineTable对象就是状态机

它的类型是两层Map：Map<STATE, Map<EVENTTYPE, Transition<OPERAND, STATE, EVENTTYPE, EVENT>>>，

外层Map的key表示旧状态，内层Map的key表示事件类型；

内层Map的value是Transition<OPERAND, STATE, EVENTTYPE, EVENT>接口，OPERAND表示操作对象，STATE表示目的状态，EVENTTYPE表示事件类型，EVENT表示事件。

stateMachineTable起到的作用是：RMAppImpl可能有多种旧状态，每种旧状态可以对应多种事件类型，根据旧状态和要处理事件的类型，就能找到处理这种情形的状态转换方法和目的状态，同时状态转换方法包含对事件的处理。

核心成员

• transitionsListNode：就是将状态机的一个个过渡的ApplicableTransition实现串联为一个链表，每个节点包含一个ApplicableTransition实现及指向下一个节点的引用
• stateMachineTable : 状态拓扑表,为了提高检索状态对应的过渡map而冗余的数据结构，此结构在optimized为真时，通过对transitionsListNode链表进行处理产生
• defaultInitialState：对象创建时，内部有限状态机的默认初始状态。比如：RMAppImpl的内部状态机默认初始状态是RMAppState.NEW。
• optimized：布尔类型，用于标记当前状态机是否需要优化性能，即构建状态拓扑表stateMachineTable。

状态转换

• 元素：当前状态，目标状态，事件，回调函数。

• 简单转换：一个当前状态、 一个目标状态、 一种触发事件、 零/一个回调

• 多事件转换：一个初始状态、 一个最终状态、 多种事件 、 零/一个回调

• 多状态转换：一个初始状态、 多个最终状态、 一种事件、多个回调

  // 简单转换
  public StateMachineFactory
           <OPERAND, STATE, EVENTTYPE, EVENT>
        addTransition(STATE preState, STATE postState, EVENTTYPE eventType) {
  return addTransition(preState, postState, eventType, null);
  }
  
  // 多事件单一状态转换
  public StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT> addTransition(
    STATE preState, STATE postState, Set<EVENTTYPE> eventTypes,
    SingleArcTransition<OPERAND, EVENT> hook) {
  StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT> factory = null;
  for (EVENTTYPE event : eventTypes) {
    if (factory == null) {
      factory = addTransition(preState, postState, event, hook);
    } else {
      factory = factory.addTransition(preState, postState, event, hook);
    }
  }
  return factory;
  }
  
  // 简单转换
  public StateMachineFactory
             <OPERAND, STATE, EVENTTYPE, EVENT>
          addTransition(STATE preState, STATE postState,
                        EVENTTYPE eventType,
                        SingleArcTransition<OPERAND, EVENT> hook){
    return new StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT>
        (this, new ApplicableSingleOrMultipleTransition<OPERAND, STATE, EVENTTYPE, EVENT>
           (preState, eventType, new SingleInternalArc(postState, hook)));
  }
  
  // 单事件多状态转换 MultipleInternalArc类中的转换方法会将回调的状态与预存的状态们进行正确性匹配，不匹配将抛出异常。
  public StateMachineFactory
               <OPERAND, STATE, EVENTTYPE, EVENT>
            addTransition(STATE preState, Set<STATE> postStates,
                          EVENTTYPE eventType,
                          MultipleArcTransition<OPERAND, EVENT, STATE> hook){
      return new StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT>
          (this,
           new ApplicableSingleOrMultipleTransition<OPERAND, STATE, EVENTTYPE, EVENT>
             (preState, eventType, new MultipleInternalArc(postStates, hook)));
  }
  

InternalStateMachine

状态机的具体实现类，主要成员操作对象operand，当前状态currentState，状态转换的监控器StateTransitionListener（有点类似AOP的处理方式）,主要事件处理方法doTransition。

private class InternalStateMachine
        implements StateMachine<STATE, EVENTTYPE, EVENT> {
    private final OPERAND operand;
    private STATE currentState;
    private final StateTransitionListener<OPERAND, EVENT, STATE> listener;

    ...
    ...
    ...

    @Override
    public synchronized STATE doTransition(EVENTTYPE eventType, EVENT event)
         throws InvalidStateTransitionException  {
      // 前置监听处理
      listener.preTransition(operand, currentState, event);
      STATE oldState = currentState;
      currentState = StateMachineFactory.this.doTransition
          (operand, currentState, eventType, event);
      // 后驱监听处理
      listener.postTransition(operand, oldState, currentState, event);
      return currentState;
    }
  }

简化实验

简化类图

事件类型

package com.donny.state;

/**
 * @author 1792998761@qq.com
 * @description
 * @date 2023/9/27
 */
public enum MyEventType {
    START,
    Change1,
    Change2
}

任务状态

package com.donny.state;

/**
 * @author 1792998761@qq.com
 * @description
 * @date 2023/9/27
 */
public enum MyState {
    NEW,
    State_1,
    State_2,
    FINISHED
}

状态机框架相关

package com.donny.state;

/**
 * @author 1792998761@qq.com
 * @description
 * @date 2023/9/27
 */
public interface SingleArcTransition<OPERAND, EVENT> {
    public void transition(OPERAND operand, EVENT event);
}

package com.donny.state;

/**
 * @author 1792998761@qq.com
 * @description
 * @date 2023/9/27
 */
public interface MultipleArcTransition<OPERAND, EVENT, STATE extends Enum<STATE>> {
    public STATE transition(OPERAND operand, EVENT event);
}

package com.donny.state;

/**
 * @author 1792998761@qq.com
 * @description
 * @date 2023/9/27
 */
public interface ApplicableTransition<OPERAND, STATE extends Enum<STATE>,
        EVENTTYPE extends Enum<EVENTTYPE>, EVENT> {
    void apply(StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT> subject);
}

package com.donny.state;

import java.util.HashMap;
import java.util.Map;

/**
 * @author 1792998761@qq.com
 * @description
 * @date 2023/9/27
 */
public class ApplicableSingleOrMultipleTransition<OPERAND, STATE extends Enum<STATE>,
        EVENTTYPE extends Enum<EVENTTYPE>, EVENT> implements ApplicableTransition<OPERAND, STATE, EVENTTYPE, EVENT> {

    final STATE preState;
    final EVENTTYPE eventType;
    final Transition<OPERAND, STATE, EVENTTYPE, EVENT> transition;

    ApplicableSingleOrMultipleTransition
            (STATE preState, EVENTTYPE eventType,
             Transition<OPERAND, STATE, EVENTTYPE, EVENT> transition) {
        this.preState = preState;
        this.eventType = eventType;
        this.transition = transition;
    }

    @Override
    public void apply
            (StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT> subject) {
        Map<EVENTTYPE, Transition<OPERAND, STATE, EVENTTYPE, EVENT>> transitionMap
                = subject.getStateMachineTable().get(preState);
        if (transitionMap == null) {
            transitionMap = new HashMap<EVENTTYPE,
                    Transition<OPERAND, STATE, EVENTTYPE, EVENT>>();
            subject.getStateMachineTable().put(preState, transitionMap);
        }
        transitionMap.put(eventType, transition);
    }
}

package com.donny.state;

import java.util.Set;

/**
 * @author 1792998761@qq.com
 * @description
 * @date 2023/9/27
 */
public class MultipleInternalArc<OPERAND, STATE extends Enum<STATE>,
        EVENTTYPE extends Enum<EVENTTYPE>, EVENT> implements Transition<OPERAND, STATE, EVENTTYPE, EVENT> {
    private Set<STATE> validPostStates;
    private MultipleArcTransition<OPERAND, EVENT, STATE> hook;  // transition hook

    MultipleInternalArc(Set<STATE> postStates,
                        MultipleArcTransition<OPERAND, EVENT, STATE> hook) {
        this.validPostStates = postStates;
        this.hook = hook;
    }

    @Override
    public STATE doTransition(OPERAND operand, STATE oldState,
                              EVENT event, EVENTTYPE eventType)
            throws RuntimeException {
        STATE postState = hook.transition(operand, event);

        if (!validPostStates.contains(postState)) {
            throw new RuntimeException("oldState:" + oldState + " ,eventType:" + eventType);
        }
        return postState;
    }
}

public class MyTransition implements SingleArcTransition {
    @Override
    public void transition(Object o, Object o2) {
        System.out.println("do transition");
    }
}

package com.donny.state;

/**
 * @author 1792998761@qq.com
 * @description
 * @date 2023/9/27
 */
public class SingleInternalArc<OPERAND, STATE extends Enum<STATE>,
        EVENTTYPE extends Enum<EVENTTYPE>, EVENT> implements Transition<OPERAND, STATE, EVENTTYPE, EVENT> {
    private STATE postState;
    private SingleArcTransition<OPERAND, EVENT> hook; // transition hook

    SingleInternalArc(STATE postState,
                      SingleArcTransition<OPERAND, EVENT> hook) {
        this.postState = postState;
        this.hook = hook;
    }

    @Override
    public STATE doTransition(OPERAND operand, STATE oldState,
                              EVENT event, EVENTTYPE eventType) {
        if (hook != null) {
            hook.transition(operand, event);
        }
        return postState;
    }
}

package com.donny.state;

/**
 * @author 1792998761@qq.com
 * @description
 * @date 2023/9/27
 */
public interface Transition<OPERAND, STATE extends Enum<STATE>,
        EVENTTYPE extends Enum<EVENTTYPE>, EVENT> {

    STATE doTransition(OPERAND operand, STATE oldState, EVENT event, EVENTTYPE eventType);
}

package com.donny.state;

/**
 * @author 1792998761@qq.com
 * @description
 * @date 2023/9/27
 */
public interface StateMachine<STATE extends Enum<STATE>,
        EVENTTYPE extends Enum<EVENTTYPE>, EVENT> {
    public STATE getCurrentState();

    public STATE doTransition(EVENTTYPE eventType, EVENT event)
            throws RuntimeException;
}

package com.donny.state;

/**
 * @author 1792998761@qq.com
 * @description
 * @date 2023/9/27
 */
public class TransitionsListNode<OPERAND, STATE extends Enum<STATE>,
        EVENTTYPE extends Enum<EVENTTYPE>, EVENT> {
    final ApplicableTransition<OPERAND, STATE, EVENTTYPE, EVENT> transition;
    final TransitionsListNode next;

    TransitionsListNode
            (ApplicableTransition<OPERAND, STATE, EVENTTYPE, EVENT> transition,
             TransitionsListNode next) {
        this.transition = transition;
        this.next = next;
    }
}

package com.donny.state;

import java.util.*;

/**
 * @author 1792998761@qq.com
 * @description
 * @date 2023/9/27
 */
public class StateMachineFactory<OPERAND, STATE extends Enum<STATE>,
        EVENTTYPE extends Enum<EVENTTYPE>, EVENT> {
    private final TransitionsListNode transitionsListNode;

    private Map<STATE, Map<EVENTTYPE,
            Transition<OPERAND, STATE, EVENTTYPE, EVENT>>> stateMachineTable;

    private STATE defaultInitialState;

    private final boolean optimized;

    public StateMachineFactory(STATE defaultInitialState) {
        this.transitionsListNode = null;
        this.defaultInitialState = defaultInitialState;
        this.optimized = false;
        this.stateMachineTable = null;
    }

    private StateMachineFactory
            (StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT> that,
             ApplicableTransition<OPERAND, STATE, EVENTTYPE, EVENT> t) {
        this.defaultInitialState = that.defaultInitialState;
        this.transitionsListNode
                = new TransitionsListNode(t, that.transitionsListNode);
        this.optimized = false;
        this.stateMachineTable = null;
    }

    public StateMachineFactory
            <OPERAND, STATE, EVENTTYPE, EVENT>
    installTopology() {
        return new StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT>(this, true);
    }

    private StateMachineFactory
            (StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT> that,
             boolean optimized) {
        this.defaultInitialState = that.defaultInitialState;
        this.transitionsListNode = that.transitionsListNode;
        this.optimized = optimized;
        if (optimized) {
            makeStateMachineTable();
        } else {
            stateMachineTable = null;
        }
    }

    private void makeStateMachineTable() {
        Stack<ApplicableTransition<OPERAND, STATE, EVENTTYPE, EVENT>> stack =
                new Stack<ApplicableTransition<OPERAND, STATE, EVENTTYPE, EVENT>>();

        Map<STATE, Map<EVENTTYPE, Transition<OPERAND, STATE, EVENTTYPE, EVENT>>>
                prototype = new HashMap<STATE, Map<EVENTTYPE, Transition<OPERAND, STATE, EVENTTYPE, EVENT>>>();

        prototype.put(defaultInitialState, null);
        stateMachineTable
                = new EnumMap<STATE, Map<EVENTTYPE,
                Transition<OPERAND, STATE, EVENTTYPE, EVENT>>>(prototype);

        for (TransitionsListNode cursor = transitionsListNode;
             cursor != null;
             cursor = cursor.next) {
            stack.push(cursor.transition);
        }

        while (!stack.isEmpty()) {
            stack.pop().apply(this);
        }
    }

    public Map<STATE, Map<EVENTTYPE, Transition<OPERAND, STATE, EVENTTYPE, EVENT>>> getStateMachineTable() {
        return stateMachineTable;
    }

    private STATE doTransition
            (OPERAND operand, STATE oldState, EVENTTYPE eventType, EVENT event)
            throws RuntimeException {
        Map<EVENTTYPE, Transition<OPERAND, STATE, EVENTTYPE, EVENT>> transitionMap
                = stateMachineTable.get(oldState);
        if (transitionMap != null) {
            Transition<OPERAND, STATE, EVENTTYPE, EVENT> transition
                    = transitionMap.get(eventType);
            if (transition != null) {
                return transition.doTransition(operand, oldState, event, eventType);
            }
        }
        throw new RuntimeException("RuntimeException: " + oldState + eventType);
    }

    public StateMachineFactory
            <OPERAND, STATE, EVENTTYPE, EVENT>
    addTransition(STATE preState, STATE postState,
                  EVENTTYPE eventType,
                  SingleArcTransition<OPERAND, EVENT> hook) {
        return new StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT>
                (this, new ApplicableSingleOrMultipleTransition<OPERAND, STATE, EVENTTYPE, EVENT>
                        (preState, eventType, new SingleInternalArc(postState, hook)));
    }

    public StateMachineFactory
            <OPERAND, STATE, EVENTTYPE, EVENT>
    addTransition(STATE preState, Set<STATE> postStates,
                  EVENTTYPE eventType,
                  MultipleArcTransition<OPERAND, EVENT, STATE> hook) {
        return new StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT>
                (this,
                        new ApplicableSingleOrMultipleTransition<OPERAND, STATE, EVENTTYPE, EVENT>
                                (preState, eventType, new MultipleInternalArc(postStates, hook)));
    }

    private synchronized void maybeMakeStateMachineTable() {
        if (stateMachineTable == null) {
            makeStateMachineTable();
        }
    }

    private class InternalStateMachine
            implements StateMachine<STATE, EVENTTYPE, EVENT> {
        private final OPERAND operand;
        private STATE currentState;

        InternalStateMachine(OPERAND operand, STATE initialState) {
            this.operand = operand;
            this.currentState = initialState;
            if (!optimized) {
                maybeMakeStateMachineTable();
            }
        }

        @Override
        public synchronized STATE getCurrentState() {
            return currentState;
        }

        @Override
        public synchronized STATE doTransition(EVENTTYPE eventType, EVENT event)
                throws RuntimeException {
            STATE oldState = currentState;
            currentState = StateMachineFactory.this.doTransition
                    (operand, currentState, eventType, event);
            return currentState;
        }
    }

    public StateMachine<STATE, EVENTTYPE, EVENT> make(OPERAND operand) {
        return new InternalStateMachine(operand, defaultInitialState);
    }
}

package com.donny.state;

/**
 * @author 1792998761@qq.com
 * @description
 * @date 2023/9/27
 */
public class Test {

    private final StateMachine stateMachine;

    private static final StateMachineFactory stateMachineFactory
            = new StateMachineFactory(MyState.NEW)
            .addTransition(MyState.NEW, MyState.State_1, MyEventType.START, new MyTransition())
            .addTransition(MyState.State_1, MyState.State_2, MyEventType.Change1, new MyTransition())
            .addTransition(MyState.State_2, MyState.FINISHED, MyEventType.Change2, new MyTransition())
            .installTopology();

    public Test() {
        this.stateMachine = stateMachineFactory.make(this);
    }


    public static void main(String[] args) {
        Test t = new Test();
        System.out.println(t.stateMachine.getCurrentState());
        Object event = new Object();
        t.stateMachine.doTransition(MyEventType.START, event);
        System.out.println(t.stateMachine.getCurrentState());
        t.stateMachine.doTransition(MyEventType.Change1, event);
        System.out.println(t.stateMachine.getCurrentState());
        t.stateMachine.doTransition(MyEventType.Change2, event);
        System.out.println(t.stateMachine.getCurrentState());
    }
}

实验结果

NEW
do transition
State_1
do transition
State_2
do transition
FINISHED