RL(2)：马尔科夫决策过程

把扫地机器人简化成以下条件：
状态序列：
{ 0 , 1 , 2 , 3 , 4 , 5 } \{0,1,2,3,4,5\} {0,1,2,3,4,5}

行为序列：
{ − 1 , + 1 } \{-1,+1\} {−1,+1}

转移函数：
$\bar{f}(0,\pm 1)=0,\bar{f}(1,+1)=2,\bar{f}(1,-1)=0,\bar{f}(2,+1)=3,\bar{f}(2,-1)=1$
$\bar{f}(3,+1)=4,\bar{f}(3,-1)=2,\bar{f}(4,+1)=5,\bar{f}(4,-1)=4,\bar{f}(5,\pm 1)=5$

奖励函数：
$\rho (0,\pm 1,0)=0,\rho (1,+1,2)=0,\rho (1,-1,0)=1,\rho (2,+1,3)=0,\rho (2,-1,1)=0$
$\rho (3,+1,4)=0,\rho (3,-1,2)=0,\rho (4,+1,5)=5,\rho (4,-1,3)=0,\rho (5,\pm 1,5)=0$

1. 定义

定义1.1：

Argument of the maximum——假设给定函数$f(x)$有最大$M$，则$f(x)$达到$M$的$x$值集合表示为
$$\underset{x}{\mathrm{argmax}}f(x)$$

定义1.2：

一个随机变量的期望值——假设随机变量$X$取$x_1$的概率为$p_1$,$x_2$的概率为$p_2$，……，则$X$的期望为
$$\mathbb{E}[X]=x_1{p}_1+x_2{p}_2+...+x_k{p}_k$$
$\mathbb{E}$被称为exception operator，并具有以下性质：
$$\begin{gathered} \mathbb{E}[X+Y]=\mathbb{E}[X]+\mathbb{E}[Y] \\ \mathbb{E}[X+c]=\mathbb{E}[X]+c \\ \mathbb{E}[cX]=c\mathbb{E}[X] \end{gathered}$$

定义1.3：

$L^{\infty }$-norm:向量$\text{x}=[x_1,x_2,...,x_n{]}^T$的$L^{\infty }$-norm，用$||\text{x}|{|}_{\infty }$表示，是$\text{x}$中最大的元素。
$$||\text{x}|{|}_{\infty }\stackrel{\Delta }{=}\underset{i}{\max }|x_i|$$

2. 马尔科夫决策的要素

2.1 状态，动作，转换和奖励

状态：

S = { s 1 , s 2 , . . . , s ∣ S ∣ } S=\{s_1,s_2,...,s_{|S|}\} S={s1​,s2​,...,s∣S∣​}

动作：

A ( s ) = { a 1 , a 2 , . . . , a ∣ A ∣ } A(s)=\{a_1,a_2,...,a_|A|\} A(s)={a1​,a2​,...,a∣​A∣}

转换函数：

确定的转换函数：
$$\begin{gathered} \bar{f}:S\times A\to S \\ \bar{f}(s,a)=s^{\prime } \end{gathered}$$
不确定的转换函数：
$$\begin{gathered} \bar{f}:S\times A\times S\to [0,1] \\ f(s,a,s^{\prime })=\mathbb{P}(S_{t+1}=s^{\prime }|S_t=s,A_t=a)=p_{s{s}^{\prime }}^a \end{gathered}$$

奖励函数：

确定的奖励函数：
$$\begin{gathered} R_t=\bar{\rho }(S_{t-1},A_{t-1},S_t|S_{t-1}=s,A_{t-1}=a,S_t=s^{\prime })=\bar{\rho }(s,a,s^{\prime })=r \\ \bar{\rho }:S\times A\times S\to R \end{gathered}$$

状态决定的奖励函数：$\rho (r|s)$
$$\rho :R\times S\to [0,1]$$
$$r(s)=\mathbb{E}[R_t|S_{t-1}=s]=r_1\rho (r_1|s)+...+r_m\rho (r_m|s)=\sum _{r}r\rho (r|s)$$

状态和动作决定的奖励函数：$\rho (r|s,a)$
$$\rho :R\times S\times A\to [0,1]$$
$$\begin{array}{rl}r(s,a)& =\mathbb{E}[R_t|S_{t-1}=s,A_{t-1}=a]\\ & =r_1\rho (r_1|s,a)+...+r_m\rho (r_m|s,a)\\ & =\sum _{r}r\rho (r|s,a)\end{array}$$

2.2 四参数的$p$函数

$$\begin{gathered} p(s^{\prime },r|s,a)=f(s,a,s^{\prime })\cdot \rho (r|s^{\prime }) \\ p:S\times R\times S\times A\to [0,1] \end{gathered}$$

利用上式可以推导：

概率转换函数

$$f(s,a,s^{\prime })=\mathbb{P}{S}_t=s^{\prime }|S_{t-1}=s,A_{t-1}=a=\sum _{r\in R}p(s^{\prime },r|s,a)$$

状态决定的奖励函数

$$\rho (r|s^{\prime })=f(s,a,s^{\prime })\cdot \rho (r|s^{\prime })=\sum _{r\in R}p(s^{\prime },r|s,a)\cdot \rho (r|s^{\prime })$$
$$\rho (r|s^{\prime })=\frac{p(s^{\prime },r|s,a)}{{\sum }_{r\in R}p(s^{\prime },r|s,a)}$$

状态和动作决定的奖励函数

$$\rho (r|s,a)=\sum _{s^{\prime }\in S}p(s^{\prime },r|s,a)$$

基于状态的期望奖励

$$r(s)=\mathbb{E}[R_t|S_t=s]=\sum _{r}r\rho (r|s)$$

基于状态和动作的期望奖励

$$r(s,a)=\mathbb{E}[R_t|S_{t-1}=s,A_{t-1}=a]=\sum _{r\in R}\sum _{s^{\prime }\in S}rp(s^{\prime },r|s,a)$$

基于状态-动作-下一状态的期望奖励

$$\begin{array}{rl}r(s,a,s^{\prime })& =\mathbb{E}[R_t|S_{t-1}=s,A_{t-1}=a,S_t=s^{\prime }]\\ & =\sum _{r}r\rho (r|s^{\prime })\\ & =\sum _{r}r\frac{p(s^{\prime },r|s,a)}{{\sum }_{r\in R}p(s^{\prime },r|s,a)}\end{array}$$

2.3 Return

奖励的总和：
$$G_t\stackrel{\Delta }{=}{R}_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+...=\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}$$

2.4 策略

π : S × A → [ 0 , 1 ] π ( a ∣ s ) = P { A t = a ∣ S t = s } \pi:S×A\rightarrow[0,1]\\ \pi(a|s)=\mathbb P\{A_t=a|S_t=s\} π:S×A→[0,1]π(a∣s)=P{At​=a∣St​=s}

3. 价值函数和贝尔曼方程

状态价值函数：

$$\begin{array}{rl}{v}_{\pi }& \stackrel{\Delta }{=}{\mathbb{E}}_{\pi }[G_t|S_t=s]\\ & ={\mathbb{E}}_{\pi }[\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}|S_t=s],对于所有的s\in S\end{array}$$

动作价值函数：

$$\begin{array}{rl}{q}_{\pi }(s,a)& ={\mathbb{E}}_{\pi }[G_t|S_t=s,A_t=a]\\ & ={\mathbb{E}}_{\pi }[\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}|S_t=s,A_t=a]\end{array}$$

贝尔曼方程：

$$\begin{array}{rl}{v}_{\pi }& \stackrel{\Delta }{=}{\mathbb{E}}_{\pi }[G_t|S_t=s]\\ & ={\mathbb{E}}_{\pi }[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+...|S_t=s]\\ & ={\mathbb{E}}_{\pi }[R_{t+1}|S_t=s]+\gamma {\mathbb{E}}_{\pi }[(R_{t+2}+\gamma R_{t+3}+...)|S_t=s]\\ & ={\mathbb{E}}_{\pi }[R_{t+1}|S_t=s]+\gamma {\mathbb{E}}_{\pi }[G_{t+1}|S_t=s]\end{array}$$
$${\mathbb{E}}_{\pi }[R_{t+1}|S_t=s]=\sum _a\pi (a|s)\sum _{s^{\prime }}\sum _{r}r\cdot p(s^{\prime },r|s,a)$$
$$\begin{array}{rl}{\mathbb{E}}_{\pi }[G_{t+1}|S_t=s]& =\sum _a\pi (a|s)\sum _{s^{\prime }}\sum _{r}p(s^{\prime },r|s,a){\mathbb{E}}_{\pi }[G_{t+1}|S_{t+1}=s^{\prime }]\\ & =\sum _{a)}\pi (a|s)\sum _{s^{\prime }}\sum _{r}p(s^{\prime },r|s,a)v_{\pi }(s^{\prime })\end{array}$$

$$\begin{array}{rl}{v}_{\pi }(s)& ={\mathbb{E}}_{\pi }[R_{t+1}|S_t=s]+\gamma {\mathbb{E}}_{\pi }[G_{t+1}|S_t=s]\\ & =\sum _a\pi (a|s)\sum _{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma v_{\pi }(s^{\prime })]\end{array}$$
$$\begin{array}{rl}{q}_{\pi }(s,a)& \stackrel{\Delta }{=}{\mathbb{E}}_{\pi }[G_t|S_t,A_t=a]\\ & ={\mathbb{E}}_{\pi }[R_{t+1}|S_t=s,A_t=a]+\gamma {\mathbb{E}}_{\pi }[G_{t+1}|S_t=s,A_t=a]\\ {\mathbb{E}}_{\pi }[R_{t+1}|S_t=s,A_t=a]& =\sum _{s^{\prime }}\sum _{r}r\cdot p(s^{\prime },r|s,a)\\ {\mathbb{E}}_{\pi }[G_{t+1}|S_t=s,A_t=a]& =\sum _{s^{\prime }}\sum _{r}p(s^{\prime },r|s,a)\pi (a^{\prime }|s^{\prime }){\mathbb{E}}_{\pi }[G_{t+1}|S_{t+1}=s^{\prime },A_{t+1}=a^{\prime }]\\ & =\sum _{s^{\prime }}\sum _{r}p(s^{\prime },r|s,a)\pi (a^{\prime }|s^{\prime })q_{\pi }(s^{\prime },a^{\prime })\end{array}$$
$$q_{\pi }(s,a)=\sum _{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma \pi (a^{\prime }|s^{\prime })q_{\pi }(s^{\prime },a^{\prime })]$$
$q_{\pi }$贝尔曼方程：
$$\begin{array}{rl}{q}_{\pi }(s,a)& ={\mathbb{E}}_{\pi }[R_{t+1}|S_t=s,A_t=a]+\gamma {\mathbb{E}}_{\pi }[G_{t+1}|S_t=s,A_t=a]\\ & =\sum _{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma q_{\pi }(s^{\prime },a^{\prime })]\end{array}$$

$v_{\pi }(s)$和$q_{\pi }(s,a)$的关系：
$$v_{\pi }(s)=\sum _a\pi (a|s)q_{\pi }(s,a)$$

如果策略$\pi$是确定的，即$\pi (a|s)=1$
$$v_{\pi }(s)=q_{\pi }(s,\pi (s))$$
可以得出
$$q_{\pi }(s,a)=\sum _{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma v_{\pi }(s^{\prime })]$$

4. 最优策略和最优值函数

最优状态函数和最优动作函数：

$$\begin{gathered} v_{\ast }(s)\stackrel{\Delta }{=}\underset{\pi }{\max }{v}_{\pi }(s) \\ {q}_{\ast }(s,a)\stackrel{\Delta }{=}\underset{\pi }{\max }{q}_{\pi }(s,a) \end{gathered}$$

贝尔曼最优方程:

$$\begin{array}{rl}{v}_{\ast }(s)& =\underset{a\in A(s)}{\max }{q}_{\ast }(s,a)\\ & =\underset{a}{\max }\sum _{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma v_{\ast }(s^{\prime })]\\ q_{\ast }(s,a)& =\sum _{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma v_{\ast }(s^{\prime })]\\ & =\sum _{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s^{\prime },a^{\prime })]\end{array}$$