Rollout Algorithm

Monte Carlo Tree Search

Construct a search tree by node based on the output of the simulation.

Construction of a search tree

(1) Selection: Starting from root node $R$，the optimal child node is selected recursively until the leaf node $L$ is reached.
(2) Expansion: If $L$ is not a terminating node then create one or more word child nodes and select one of them, $C$.
(3) Simulation: Run the output of a simulation starting from $C$ until the end of the gaming game → Rollout policy：usually a uniform random distribution function.
(4) Backpropagation: Update the current action sequence with the resultant output of the simulation. Motivation: update the total simulation gain $Q(v)$ and total number of visits $N(s)$.
(5) UTC: Upper Confidence Bound applied to Trees. Function to select the next node for traversal from the visited nodes.
$$\mathbb{UCT}(v_i,v)=\frac{Q(v_i)}{N(v_i)}+c\sqrt{\frac{log(N(v))}{N(v_i)}}$$
where, $\frac{Q(v_i)}{N(v_i)}$ is exploitation component, can be viewed as the win rate estimate of the sub-node $v_i$, $\sqrt{\frac{log(N(v))}{N(v_i)}}$ is exploration component, $c$ is the discount factor between exploitation and exploration. (Greedy algorithm when $c=0$)

Essence of Rollout

Rollout Algorithm is a decision planning algorithm based on MC control. Different from MC, which estimates all the value function in order to find the optimal strategy ${\pi }^{\ast }$, Rollout algorithm estimate only the value of each current state (Planning at Decision Time).
For each state, Rollout policy chooses the action correspongding to the maximum estimate (new policy ${\pi }^{\prime }$), which satisfies:
$$q_{\pi }(s,{\pi }^{\prime }(s))\ge v_{\pi }(s)$$
Therefore, the essence of Rollout algorithm is to improve the current strategy, but not find the optimal strategy ${\pi }^{\ast }$.

The Efficiency of Rollout

Constrained by one-step decision time.
(1) Number of possible actions $|\mathcal{A}(s)|$.
(2) The length of the simulated trajectory.
(3) Execution time of the policy.
(4) The number of simulated trajectories for better value estimation.

Geometric interpretation of Rollout

According to Bellman’s equation, each policy $\mu$ defines the linear function $T_{\mu }J$, which has value at $x$ given by:
( T μ J ) ( x ) = E { g ( x , u , w ) + α J ( f ( x , u , w ) ) } , f o r   a l l   x (T_{\mu}J)(x)=E\left \{ g(x,u,w) + \alpha J(f(x,u,w)) \right \} ,for \: all \: x (Tμ​J)(x)=E{g(x,u,w)+αJ(f(x,u,w))},forallx
And value at state $x$ is given by:
( T J ) ( x ) = m i n u ∈ U ( x ) E { g ( x , u , w ) + α J ( f ( x , u , w ) ) } , f o r   a l l   x (TJ)(x)=min_{u \in U(x)}E\left \{ g(x,u,w) + \alpha J(f(x,u,w)) \right \} ,for \: all \: x (TJ)(x)=minu∈U(x)​E{g(x,u,w)+αJ(f(x,u,w))},forallx
can also be written as $TJ=mi{n}_{\mu }{T}_{\mu }J$¹.

Truncated Rollout

Truncated Rollout = $m$ steps value iterations + base policy $\mu$ + terminal cost function approximation $\tilde{J}\to J_{\mu }$
(1) Truncated Rollout with one-step lookahead
$$T_{\tilde{\mu }}(T_{\mu }^m\tilde{J})=T(T_{\mu }^m\tilde{J})$$
that is, $\tilde{\mu }$ attains the minimum of $\tilde{J}$.

(2) Truncated Rollout with l-step lookahead
$$T_{\tilde{\mu }}(T^{l-1}{T}_{\mu }^m\tilde{J})=T(T^{l-1}{T}_{\mu }^m\tilde{J})$$

value $l$: starting point $T^{l-1}J\to J^{\ast }$
value $m$: starting point $\to J_{\mu }$

Receding horizon in MPC

Problem formulation

Consider a finite horizon $l$-stage optimal control problem involving the same cost function and the requirement that the state after $l$ steps is driven to 0. This is the problem:
$$\underset{u_t,t=k,\dots ,k+l-1}{\min }\sum _{t=k}^{k+l-1}g(x_t,u_t)$$
System equation constraines:
$$x_{t+1}=f(x_t,u_t),t=k,\dots ,k+l-1$$
The control constraints:
$$u_t\in U(x_t),t=k,\dots ,k+l-1$$
And the terminal state constraints:
$$x_{k+l}=0$$
If {${\tilde{u}}_k,\dots ,{\tilde{u}}_{k+l-1}$} is the optimal control sequence of this problem, we apply ${\tilde{u}}_k$ and we apply ${\tilde{u}}_k$ and we discard the other controls ${\tilde{u}}_{k+1},\dots ,{\tilde{u}}_{k+l-1}$.
At the next stage, repeat the process, once the next state $x_{k+1}$ is revealed.

In summary, the receding horizon in MPC is equivalent to the $l$-step lookahead rollout.

Reference:
8.10 rollout算法
蒙特卡洛树搜索 Monte Carlo Tree Search
Bertsekas, Dimitri. “Newton’s method for reinforcement learning and model predictive control.” Results in Control and Optimization 7 (2022): 100121.

---

1. $T,T_{\mu }$ are the Bellman operators defined to give a compact expression. ↩︎