Lect2_MDPs

Markov Decision Processes

MPDs formally describe an environment for RL

Markov Processes

Definition

无记忆的随机过程

i.e. a sequence of random states $S_1,S_2,\dots$​ with the [Markov property](#Markov Property)

A Markov Process (or Markov Chain) is a tuple $⟨\mathcal{S},\mathcal{P}⟩$​​​

• $\mathcal{S}$​ is a (finite) set of states
• $\mathcal{P}$ is a state [transition probability matrix](#state transition matrix)
  ${\mathcal{P}}_{s{s}^{\prime }}=\mathbb{P}\left[S_{t+1}=s^{\prime }\mid S_t=s\right]$​

Markov Property

“The future is independent of the past given the present”
$$\mathbb{P}\left[S_{t+1}\mid S_t\right]=\mathbb{P}\left[S_{t+1}\mid S_1,\dots ,S_t\right]$$
The state is a sufficient statistic of the future

State Transition Matrix

state transition probability ${\mathcal{P}}_{source\_destination}$​
$${\mathcal{P}}_{s{s}^{\prime }}=\mathbb{P}\left[S_{t+1}=s^{\prime }\mid S_t=s\right]$$
state transition matrix $\mathcal{P}$​​

defines transition probabilities from all state s to all successor state s’
$$\mathcal{P}=\left[\begin{array}{ccc}{\mathcal{P}}_{11}& \cdots & {\mathcal{P}}_{1n}\\ ⋮& & \\ {\mathcal{P}}_{n1}& \cdots & {\mathcal{P}}_{nn}\end{array}\right]$$
根据概率的性质，一定有：
$$\sum _{j=1}^n{\mathcal{P}}_{ij}=1\forall i=1,\dots ,n$$
Example:

Markov Reward Process

Definition

a Markov chain with values

A Markov Reward Process is a tuple $⟨\mathcal{S},\mathcal{P},\mathcal{R},\gamma ⟩$

• $\mathcal{S}$ is a finite set of states
• $\mathcal{P}$​ is a state transition probability matrix
  ${\mathcal{P}}_{s{s}^{\prime }}=\mathbb{P}\left[S_{t+1}=s^{\prime }\mid S_t=s\right]$​
• $\mathcal{R}$​ is a reward function, $\mathcal{R}s = \mathbb{E}[R{t+1} \mid S_t =s] $​
• $\gamma$ is a discount factor, $\gamma \in [0,1]$​

注意这时的reward只和状态有关 ${\mathcal{R}}_s$​​​，例如下图的 class 1 无论去 Facebook 还是 class 2 都是 $R=-2$​

Return

the retrun $G_t$ is the total discounted reward from time-step t
$$G_t=R_{t+1}+\gamma R_{t+2}+\cdots =\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}$$

• $\gamma \in [0,1]$​ 代表了未来的奖励在现在这一时刻起的作用大小,更希望得到现有的奖励，未来的奖励就要把它打折扣
  • $\gamma$ 代表“近视”，比较重视短期利益
  • $\gamma$ 代表“远视”，对未来的奖励也一样看待

Why discount

• 有些马尔可夫过程是带环的，它并没有终结，我们想避免这个无穷的奖励。
• 我们并没有建立一个完美的模拟环境的模型，也就是说，我们对未来的评估不一定是准确的，我们不一定完全信任我们的模型，因为这种不确定性，所以我们对未来的预估增加一个折扣。我们想把这个不确定性表示出来，希望尽可能快地得到奖励，而不是在未来某一个点得到奖励。
• 如果这个奖励是有实际价值的，我们可能是更希望立刻就得到奖励，而不是后面再得到奖励（现在的钱比以后的钱更有价值）。
• 在人的行为里面来说的话，大家也是想得到即时奖励。
• 有些时候可以把这个系数设为 0：我们就只关注了它当前的奖励。我们也可以把它设为 1：对未来并没有折扣，未来获得的奖励跟当前获得的奖励是一样的。

Value Function

The state value function v(s) of an MRP is the expected return starting from state s
$$v(s)=\mathbb{E}[G_t\mid S_t=s]$$

Bellman Equation

记 $S_{t+1}$​ 为 $s^{\prime }$​​，根据期望的定义：

$$v(s)={\mathcal{R}}_s+\gamma \sum _{s^{\prime }\in S}{\mathcal{P}}_{s{s}^{\prime }}v(s^{\prime })\text{\hspace{1em}(4)}$$
由 equation 1. 到式 equation 2. 并不是那么的直观，还需要进一步证明 $\mathbb{E}[G_{t+1}\mid S_t]=\mathbb{E}\left[v(S_{t+1})\mid S_t\right]=\mathbb{E}\left[\mathbb{E}[G_{t+1}\mid S_{t+1}]\mid S_t\right]$

先回顾一下全概率公式：$\mathbb{E}(X)={\sum }_{x}xP(X=x\mid Y=y)$​​​​

记 $G_{t+1}=g^{\prime },S_{t+1}=s^{\prime },S_t=s$​

$$\begin{array}{rl}\mathbb{E}\left[\mathbb{E}[G_{t+1}\mid S_{t+1}]\mid S_t\right]& =\mathbb{E}\left[\mathbb{E}[g^{\prime }\mid S^{\prime }]\mid S_t\right]\\ & =\mathbb{E}\left[\sum _{g^{\prime }}{g}^{\prime }p(g^{\prime }\mid s^{\prime })\mid s\right]\\ & =\sum _{s^{\prime }}\left(\sum _{g^{\prime }}{g}^{\prime }p(g^{\prime }\mid s^{\prime },s)\right)p(s^{\prime }\mid s)\\ & =\sum _{s^{\prime }}\sum _{g^{\prime }}{g}^{\prime }\frac{p(g^{\prime },s^{\prime },s)}{p(s^{\prime },s)}\frac{p(s^{\prime },s)}{p(s)}\\ & =\sum _{s^{\prime }}\sum _{g^{\prime }}{g}^{\prime }p(g^{\prime },s^{\prime }\mid s)\\ & =\sum _{g^{\prime }}{g}^{\prime }p(g^{\prime }\mid s)\\ & =\mathbb{E}[G_{t+1}\mid s_t]\end{array}$$
Equation 3. in Matrix form

根据矩阵形式，直接可求 value function 的解析解：$\mathbf{v}={\left(I-\gamma (P)\right)}^{-1}\mathcal{R}$

但是对于含n个状态的矩阵，计算复杂度为 $O(n^3)$，因此解析解只适合小型的MRPs，对于大型的MRPs，采取迭代的方法：

• Dynamic programming
• Monte-Carlo evaluation
• Temporal-Difference learning

Markov Decision Processes

Definition

A MDP is a MRP with decisions. It is a environment in which all states are Markov

A Markov Decision Process is a tuple $⟨\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma ⟩$

• $\mathcal{S}$​ is a finite set of states
• $\mathcal{A}$ is a finite set of actions
• $\mathcal{P}$​​ is a state transition probability matrix
  ${\mathcal{P}}_{s{s}^{\prime }}^a=\mathbb{P}\left[S_{t+1}=s^{\prime }\mid S_t=s,A_t=a\right]$​​
• $\mathcal{R}$ is a reward function, $\mathcal{R}s^{\color{red}a} = \mathbb{E}[R{t+1} \mid S_t =s ,{\color{red} A_t = a}] $
• $\gamma$ is a discount factor, $\gamma \in [0,1]$​

Policy

A policy $\pi$ is a distribution over actions given states
$$\pi (a\mid s)=\mathbb{P}\left[A_t=a\mid s_t=s\right]$$
Given an MDP $\mathcal{M}=⟨\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma ⟩$ and a policy $\pi$

• The state sequence $S_1,S_2,\dots$ is a Markkov process $⟨\mathcal{S},{\mathcal{P}}^{\pi }⟩$

• The state and reward sequence $S_1,R_2,S_2,\dots$ is a Markkov reward process $⟨\mathcal{S},{\mathcal{P}}^{\pi },{\mathcal{R}}^{\pi },\gamma ⟩$​​

• where
  $$\begin{gathered} {\mathcal{P}}_{s,s^{\prime }}^{\pi }=\sum _{a\in \mathcal{A}}\pi (a\mid s){\mathcal{P}}_{s{s}^{\prime }}^a \\ {R}_s^{\pi }=\sum _{a\in \mathcal{A}}\pi (a\mid s){\mathcal{R}}_s^a \end{gathered}$$

Value Function

• State-value funtion $v_{\pi }(s)=\mathbb{E}\left[G_t\mid S_t=s\right]$​​
• Action-value function $q_{\pi }(s,a)=\mathbb{E}\left[G_t\mid S_t=s,A_t=a\right]$​

Bellman Expectation Equation

1. value function can be decomposed into immediate reward plus discounted value of successor state
   $$v_{\pi }(s)={\mathbb{E}}_{\pi }\left[R_{t+1}+\gamma v_{\pi }(S_{t+1})\mid S_t=s\right]\text{\hspace{1em}(5)}$$

   $$q_{\pi }(s,a)={\mathbb{E}}_{\pi }\left[R_{t+1}+\gamma q_{\pi }(S_{t+1,}{A}_{t+1})\mid S_t=s,A_t=a\right]\text{\hspace{1em}(6)}$$
   equation 5. 和 equation 6. 表明了当前状态和未来状态之间的 value function 的关系

2. 考虑 state-value function 和 action-value function 之间的关系

$$v_{\pi }(s)=\sum _{a\in \mathcal{A}}\pi (a\mid s)q_{\pi }(s,a)\text{对所有用黑色实心圆代表的 action: a 求和 }\text{\hspace{1em}(7)}$$
$$q_{\pi }(s,a)={\mathcal{R}}_s^a+\gamma \sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a{v}_{\pi }(s^{\prime })\text{对所有用空心圆代表的 state: s 求和}\text{\hspace{1em}(8)}$$

3. 把 equation 7. 和 equation 8. 互相代入就可以得出 equation 5. 和 equation 6. 的取去掉 $\mathbb{E}[]$​ 的形式

$$v_{\pi }(s)=\sum _{a\in \mathcal{A}}\pi (a\mid s)\left({\mathcal{R}}_s^a+\gamma \sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a{v}_{\pi }(s^{\prime })\right)\text{\hspace{1em}(9)}$$

$$q_{\pi }(s,a)={\mathcal{R}}_s^a+\gamma \sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a\sum _{a^{\prime }\in \mathcal{A}}\pi (a^{\prime }\mid s^{\prime })q_{\pi }(s^{\prime },a^{\prime })\text{\hspace{1em}(10)}$$
直接由 equation 5. 6. 分别推 equation 9. 10.
$$\begin{array}{rl}{v}_{\pi }(s)& ={\mathbb{E}}_{\pi }\left[R_{t+1}+\gamma v_{\pi }(S_{t+1})\mid S_t=s\right]\\ & ={\mathbb{E}}_{\pi }\left[R_{t+1}\mid S_t=s\right]+\gamma {\mathbb{E}}_{\pi }\left[v_{\pi }(S_{t+1})\mid S_t=s\right]\\ & =\sum _{a\in \mathcal{A}}\pi (a\mid s){\mathcal{R}}_s^a+\gamma \sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^{\pi }{v}_{\pi }(s^{\prime })\\ & =\sum _{a\in \mathcal{A}}\pi (a\mid s){\mathcal{R}}_s^a+\gamma \sum _{s^{\prime }\in \mathcal{S}}\left[\sum _{a\in \mathcal{A}}\pi (a\mid s){\mathcal{P}}_{s{s}^{\prime }}^a\right]v_{\pi }(s^{\prime })\\ & =\sum _{a\in \mathcal{A}}\pi (a\mid s)\left({\mathcal{R}}_s^a+\gamma \sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a{v}_{\pi }(s^{\prime })\right)\triangleq \text{Equation 9.}\end{array}$$

$$\begin{array}{rl}{q}_{\pi }(s,a)& ={\mathbb{E}}_{\pi }\left[R_{t+1}+\gamma q_{\pi }(S_{t+1,}{A}_{t+1})\mid S_t=s,A_t=a\right]\\ & ={\mathbb{E}}_{\pi }\left[R_{t+1}\mid S_t=s,A_t=a\right]+\gamma {\mathbb{E}}_{\pi }\left[q_{\pi }(S_{t+1,}{A}_{t+1})\mid S_t=s,A_t=a\right]\\ & ={\mathcal{R}}_s^a+\gamma \sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a\sum _{a^{\prime }\in \mathcal{A}}\pi (a^{\prime }\mid s^{\prime })q_{\pi }(s^{\prime },a^{\prime })\triangleq \text{Equation 10.}\end{array}$$

直接使用 Equation 9. 计算如下例子：

Optimal Value Function

记：$v_{\ast }(s)=\underset{\pi }{\max }{v}_{\pi }(s)$, $q_{\ast }(s,a)=\underset{\pi }{\max }{q}_{\pi }(s,a)$

An MDP is “solved” when we know the optimal value function

如何比较 policies 的好坏（大小）：$\pi \ge {\pi }^{\prime }\mathrm{if}{v}_{\pi }(s)\ge v_{{\pi }^{\prime }}(s),\forall s$​

Finding an optimal Policy

If we know $q_{\ast }(s,a)$, we immediately have the optimal policy
$${\pi }_{\ast }(a\mid s)=\{\begin{array}{l}1,\text{if}a=\underset{a\in \mathcal{A}}{\mathrm{argmax}}{q}_{\ast }(s,a)\\ 0,otherwise\end{array}$$

Bellman Optimaltiy Equation

$$v_{\ast }(s)=\underset{a}{\max }{q}_{\ast }(s,a)\text{\hspace{1em}(11)}$$

$$q_{\ast }(s,a)={\mathcal{R}}_s^a+\gamma \sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a{v}_{\ast }(s^{\prime })\text{\hspace{1em}(12)}$$

Equation 11. 和 Equation 12. 互相代入
$$v_{\ast }(s)=\underset{a}{\max }{\mathcal{R}}_s^a+\gamma \sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a{v}_{\ast }(s^{\prime })\text{\hspace{1em}(11)}$$
$$q_{\ast }(s,a)={\mathcal{R}}_s^a+\gamma \sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a\underset{a^{\prime }}{\max }{q}_{\ast }(s^{\prime },a^{\prime })\text{\hspace{1em}(12)}$$