Lect3_Dynamic_Programming

Planning by Dynamic Programming

Introduction

Dynamic sequential or temporal component to the problem Programming optimising a “program”

Requirements for DP

1. Optimal substructure: 将问题分解成多个子问题，寻找子问题的最优解而后组合起来得到大问题的最优解
2. Overlapping subproblems: 子问题出现多次并且其解决方案可以重复利用

MDP satisfy both requirements:

1. Bellman equation gives recursive decomposition.
2. Value function stores and reuses solutions

DP used for planning in an MDP

Prediction:

Control:

Policy Evaluation

---

Iterative Policy Evaluation

Iterative application of [$\text{Bellman expectation backup}$](#Bellman expectation backup): $v_1\to v_2\to \dots \to v_{\pi }$

• Using Synchornous backups:
  • at each iteration $k+1$
  • for all states $s\in \mathcal{S}$
  • update $v_{k+1}(s)$ from $v_k(s^{\prime })$
  • where $s^{\prime }$​ is a successor state of $s$​
• Convergence to $v_{\pi }$ can be proved

How to update：

$$v_{k+1}(s)=\sum _{a\in \mathcal{A}}\pi (a\mid s)\left({\mathcal{R}}_s^a+\sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a{v}_k(s^{\prime })\right)$$
不由得想起数值分析里面迭代求解 $x=f(x)$​，即 $\text{initial}{x}_0=\text{number},\text{then loop}{x}_{k+1}=f(x_k)$​

Matrix From：
$${\mathbf{v}}^{k+1}={\mathcal{R}}^{\pi }+\gamma {\mathcal{P}}^{\pi }{\mathbf{v}}^k$$
where
$$\begin{array}{rl}{\mathcal{R}}^{\pi }& =\sum _{a\in \mathcal{A}}\pi (a\mid s){\mathcal{R}}_s^a\\ {\mathcal{P}}^{\pi }& =\sum _{s^{\prime }\in s}{\mathcal{P}}_{s{s}^{\prime }}^{\pi }=\sum _{s^{\prime }\in \mathcal{S}}\sum _{a\in \mathcal{A}}\pi (a\mid s){\mathcal{P}}_{s{s}^{\prime }}^a=\sum _{a\in \mathcal{A}}\pi (a\mid s)\sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a\end{array}$$

---

Example

• $\gamma =1$
• Nonterminal states 1, …, 14; One terminal state (shown twice as shaded squares)
• Actions leading out of the grid leave state unchanged, e.g. when $s=4,a=\text{west}$, the next state will be $s^{\prime }=4$
• Transition is deterministic given the action, e.g. ${\mathcal{P}}_{62}^{\text{north}}=\mathcal{P}\left[s^{\prime }=2\mid s=6,a=\text{north}\right]=1$
• Reward is -1 until the terminal state is reached
• Uniform random policy, $\pi (n\mid \cdot )=\pi (e\mid \cdot )=\pi (w\mid \cdot )=\pi (s\mid \cdot )=0.25$​

一开始给每一个状态的价值函数都初始化为0，不断地进行迭代就好了。如下：

计算过程如下：

For k=0:
$$v_0(s)=0\forall s$$
For k = 1: e.g.
$$\begin{array}{rl}{v}_1(s=4)& =\pi (a=n\mid s=4)\ast \left(R_{s=4}^{a=n}+{\mathcal{P}}_{s=4,s^{\prime }=\text{terminal}}^{a=n}{v}_0(s^{\prime }=\text{terminal})\right)+\\ & =\pi (a=w\mid s=4)\ast \left(R_{s=4}^{a=w}+{\mathcal{P}}_{s=4,s^{\prime }=4}^{a=w}{v}_0(s^{\prime }=4)\right)+\\ & =\pi (a=s\mid s=4)\ast \left(R_{s=4}^{a=s}+{\mathcal{P}}_{s=4,s^{\prime }=8}^{a=s}{v}_0(s^{\prime }=8)\right)+\\ & =\pi (a=e\mid s=4)\ast \left(R_{s=4}^{a=e}+{\mathcal{P}}_{s=4,s^{\prime }=5}^{a=e}{v}_0(s^{\prime }=5)\right)+\\ & =0.25\ast (-1+0)+0.25\ast (-1+0)+0.25\ast (-1+0)+0.25\ast (-1+0)=-1.0\end{array}$$
For k = 2: e.g.
$$\begin{array}{rl}{v}_2(s=4)& =\pi (a=n\mid s=4)\ast \left(R_{s=4}^{a=n}+{\mathcal{P}}_{s=4,s^{\prime }=\text{terminal}}^{a=n}{v}_1(s^{\prime }=\text{terminal})\right)+\\ & =\pi (a=w\mid s=4)\ast \left(R_{s=4}^{a=w}+{\mathcal{P}}_{s=4,s^{\prime }=4}^{a=w}{v}_1(s^{\prime }=4)\right)+\\ & =\pi (a=s\mid s=4)\ast \left(R_{s=4}^{a=s}+{\mathcal{P}}_{s=4,s^{\prime }=8}^{a=s}{v}_1(s^{\prime }=8)\right)+\\ & =\pi (a=e\mid s=4)\ast \left(R_{s=4}^{a=e}+{\mathcal{P}}_{s=4,s^{\prime }=5}^{a=e}{v}_1(s^{\prime }=5)\right)+\\ & =0.25\ast (-1+0)+0.25\ast (-1-1)+0.25\ast (-1-1)+0.25\ast (-1-1)=-1.75\end{array}$$
$$\begin{array}{rl}{v}_2(s=8)& =\pi (a=n\mid s=8)\ast \left(R_{s=8}^{a=n}+{\mathcal{P}}_{s=8,s^{\prime }=4}^{a=n}{v}_1(s^{\prime }=4)\right)+\\ & =\pi (a=w\mid s=8)\ast \left(R_{s=8}^{a=w}+{\mathcal{P}}_{s=8,s^{\prime }=8}^{a=w}{v}_1(s^{\prime }=8)\right)+\\ & =\pi (a=s\mid s=8)\ast \left(R_{s=8}^{a=s}+{\mathcal{P}}_{s=8,s^{\prime }=12}^{a=s}{v}_1(s^{\prime }=12)\right)+\\ & =\pi (a=e\mid s=8)\ast \left(R_{s=8}^{a=e}+{\mathcal{P}}_{s=8,s^{\prime }=9}^{a=e}{v}_1(s^{\prime }=9)\right)+\\ & =0.25\ast (-1-1)+0.25\ast (-1-1)+0.25\ast (-1-1)+0.25\ast (-1-1)=-2\end{array}$$
For k=3: e.g.
$$\begin{array}{rl}{v}_3(s=4)& =\pi (a=n\mid s=4)\ast \left(R_{s=4}^{a=n}+{\mathcal{P}}_{s=4,s^{\prime }=\text{terminal}}^{a=n}{v}_2(s^{\prime }=\text{terminal})\right)+\\ & =\pi (a=w\mid s=4)\ast \left(R_{s=4}^{a=w}+{\mathcal{P}}_{s=4,s^{\prime }=4}^{a=w}{v}_2(s^{\prime }=4)\right)+\\ & =\pi (a=s\mid s=4)\ast \left(R_{s=4}^{a=s}+{\mathcal{P}}_{s=4,s^{\prime }=8}^{a=s}{v}_2(s^{\prime }=8)\right)+\\ & =\pi (a=e\mid s=4)\ast \left(R_{s=4}^{a=e}+{\mathcal{P}}_{s=4,s^{\prime }=5}^{a=e}{v}_2(s^{\prime }=5)\right)+\\ & =0.25\ast (-1+0)+0.25\ast (-1-1.75)+0.25\ast (-1-2)+0.25\ast (-1-2)=-2.4375\end{array}$$
$$\begin{array}{rl}{v}_3(s=8)& =\pi (a=n\mid s=8)\ast \left(R_{s=8}^{a=n}+{\mathcal{P}}_{s=8,s^{\prime }=4}^{a=n}{v}_2(s^{\prime }=4)\right)+\\ & =\pi (a=w\mid s=8)\ast \left(R_{s=8}^{a=w}+{\mathcal{P}}_{s=8,s^{\prime }=8}^{a=w}{v}_2(s^{\prime }=8)\right)+\\ & =\pi (a=s\mid s=8)\ast \left(R_{s=8}^{a=s}+{\mathcal{P}}_{s=8,s^{\prime }=12}^{a=s}{v}_2(s^{\prime }=12)\right)+\\ & =\pi (a=e\mid s=8)\ast \left(R_{s=8}^{a=e}+{\mathcal{P}}_{s=8,s^{\prime }=9}^{a=e}{v}_2(s^{\prime }=9)\right)+\\ & =0.25\ast (-1-1.75)+0.25\ast (-1-2)+0.25\ast (-1-2)+0.25\ast (-1-2)=-2.9375\end{array}$$
$$\begin{array}{rl}{v}_3(s=12)& =\pi (a=n\mid s=12)\ast \left(R_{s=12}^{a=n}+{\mathcal{P}}_{s=12,s^{\prime }=8}^{a=n}{v}_2(s^{\prime }=8)\right)+\\ & =\pi (a=w\mid s=12)\ast \left(R_{s=12}^{a=w}+{\mathcal{P}}_{s=12,s^{\prime }=12}^{a=w}{v}_2(s^{\prime }=12)\right)+\\ & =\pi (a=s\mid s=12)\ast \left(R_{s=12}^{a=s}+{\mathcal{P}}_{s=12,s^{\prime }=12}^{a=s}{v}_2(s^{\prime }=12)\right)+\\ & =\pi (a=e\mid s=12)\ast \left(R_{s=12}^{a=e}+{\mathcal{P}}_{s=12,s^{\prime }=13}^{a=e}{v}_2(s^{\prime }=13)\right)+\\ & =0.25\ast (-1-2)+0.25\ast (-1-2)+0.25\ast (-1-2)+0.25\ast (-1-2)=-3.0\end{array}$$

Policy Iteration

Algorithm:

1. Given a policy $\pi$

2. Loop forever until stopping condition

   1. Evaluate the policy $\pi$​
      $$v_{\pi }(s)=\mathbb{E}\left[R_{t+1}+\gamma R_{t+2}+\dots \mid S_t=s\right]$$

   2. Improve the policy by acting greedily with respect to $v_{\pi }$​​
      $${\pi }^{\prime }=\mathrm{greedy}(v_{\pi })$$

Policy improvement

Proof of why ${\pi }^{\prime }\ge \pi$​ when use greedy:

1. Consider a deterministic policy, $a=\pi (s)$

2. improve the policy by acting greedily
   $${\pi }^{\prime }(s)=\underset{a\in \mathcal{A}}{\mathrm{argmax}}{q}_{\pi }(s,a)$$

3. This improves the value from any state s over one step
   $$q_{\pi }\left(s,{\pi }^{\prime }(s)\right)=\underset{a\in \mathcal{A}}{\max }{q}_{\pi }(s,a)\ge q_{\pi }\left(s,\pi (s)\right)=v_{\pi }(s)\forall s$$

4. It therefore improves the value function, $v_{{\pi }^{\prime }}(s)\ge v_{\pi }(s)$​
   $$\begin{array}{rl}{v}_{\pi }(s)& \le q_{\pi }\left(s,{\pi }^{\prime }(s)\right)={\mathbb{E}}_{{\pi }^{\prime }}\left[R_{t+1}+\gamma v_{\pi }\left(S_{t+1}\right)\mid S_t=s\right]\\ & \le {\mathbb{E}}_{{\pi }^{\prime }}\left[R_{t+1}+\gamma q_{\pi }\left(S_{t+1},{\pi }^{\prime }(S_{t+1})\right)\mid S_t=s\right]\text{because step 3.}\forall s\\ & \le {\mathbb{E}}_{{\pi }^{\prime }}\left[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{q}_{\pi }\left(S_{t+2},{\pi }^{\prime }(S_{t+2})\right)\mid S_t=s\right]\\ & \le \cdots \le {\mathbb{E}}_{{\pi }^{\prime }}\left[R_{t+1}+\gamma R_{t+2}+\cdots \mid S_t=s\right]=v_{{\pi }^{\prime }}(s)\end{array}$$
   Why ${\pi }^{\prime }$ and $\pi$​ ?

   得回到 action-value function 的定义来看待这个问题
   $$q_{\pi }(s,a)={\mathbb{E}}_{\pi }\left[R_{t+1}+\gamma v_{\pi }(s_{t+1})\right]=\sum _{a\in \mathcal{A}}\pi (a\mid s)\left(R_{t+1}+\gamma v_{\pi }(s_{t+1})\right)$$
   因此可以看出求期望时，即加权平均值的 加权为 $\pi (a\mid s)$ 和动作策略是有关的，当我们使用了 greedy 之后，每一个状态下各动作选取的概率会改变，而 state-value funtion 还是由 $\pi$ 算出来的，没进行更新。所以有 ${\pi }^{\prime }$ and $\pi$ 的区别

Proof of why converge to ${\pi }^{\ast }$:

1. If improvements stop
   $$q_{\pi }\left(s,{\pi }^{\prime }(s)\right)=\underset{a\in \mathcal{A}}{\max }{q}_{\pi }(s,a)=q_{\pi }\left(s,\pi (s)\right)=v_{\pi }(s)\forall s$$

2. Then the Bellman optimality equation has been satisfied
   $$v_{\pi }(s)=\underset{a\in \mathcal{A}}{\max }{q}_{\pi }(s,a)$$

3. Therefore now $v_{\pi }(s)=v_{\ast }(s)\forall s\in \mathcal{S}$​

4. so $\pi$ is an optimal policy

Value Iteration

Principle of Optimality

A policy $\pi (a\mid s)$ achieves the optimal value from state s, i.e. $v_{\pi }(s)=v_{\ast }(s)$, if and only if

• for any state s’ reachable from s
• $\pi$ achieves the optimal value from state s’, i.e. $v_{\pi }(s^{\prime })=v_{\ast }(s^{\prime })$

---

Deterministic Value Iteration

Compare to Iterative Policy Evaluation, please click here, actually the difference between them is how to update (max or expectation)

• If we know the solution to subproblems $v_{\ast }(s^{\prime })$

• Then solution $v_{\ast }(s^{\prime })$ can be found by one-step lookhead
  $$v_{\ast }(s)\leftarrow \underset{a\in \mathcal{A}}{\max }\left({\mathcal{R}}_S^a+\gamma \sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a{v}_{\ast }(s^{\prime })\right)$$

• The idea of value iteration is to apply these updates iteratively

• Intuition: start with final rewards and work backwards

Iterative application of [$\text{Bellman optimality backup}$​](#Bellman optimality backup): $v_1\to v_2\to \dots \to v_{\ast }$​

• Using Synchornous backups:
  • at each iteration $k+1$
  • for all states $s\in \mathcal{S}$
  • update $v_{k+1}(s)$ from $v_k(s^{\prime })$​
• Intermediate value function may not correspond to any policy

How to update:

$$v_{k+1}(s)=\underset{a\in \mathcal{A}}{\max }\left({\mathcal{R}}_s^a+\gamma \sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a{v}_k(s^{\prime })\right)$$
Matrix From:
$${\mathbf{v}}_{k+1}=\underset{a\in \mathcal{A}}{\max }\left({\mathcal{R}}^{\mathbf{a}}+\gamma {\mathcal{P}}^{\mathbf{a}}{v}_k\right)$$

---

A live demo

GridWorld: Dynamic Programming Demo

Summary of DP Algorithms

Problem	bEllman equation	Algorithm
Prediction	Bellman Expectation Equation	Iterative Policy Evaluation
Control	Bellman Expectation Equation + Greedy Policy Improvement	Policy Iteration
Control	Bellman Optimality Equation	Value Iteration

• Algorithms are based on state-value function $v_{\pi }(s)$ or $v_{\ast }(s)$​
• m actions and n states, so complexity = O(m*n(s)*n(s’) ) = O(mn²) per iteration