深度强化学习(二)

深度强化学习(二)(贝尔曼方程)

一.贝尔曼方程（将 $Q_{\pi }$ 表示成 $Q_{\pi }$ ）

Theorem :假设 $R_t$ 是 $S_t、A_t、S_{t+1}$ 的函数。那么
$$Q_{\pi }\left(s_t,a_t\right)={\mathbb{E}}_{S_{t+1},A_{t+1}}\left[R_t+\gamma \cdot Q_{\pi }\left(S_{t+1},A_{t+1}\right)\mid S_t=s_t,A_t=a_t\right].\text{\hspace{1em}(1.1)}$$

proof:令 S t + 1 : = { S t + 1 , S t + 2 , ⋯   } \mathcal{S}_{t+1:}=\left\{S_{t+1}, S_{t+2}, \cdots\right\} St+1:​={St+1​,St+2​,⋯}, A t + 1 : = { A t + 1 , A t + 2 , ⋯   } \mathcal{A}_{t+1:}=\left\{A_{t+1}, A_{t+2}, \cdots\right\} At+1:​={At+1​,At+2​,⋯},由$U_t$的定义知$U_t=R_t+\gamma \cdot U_{t+1}$
$$\begin{array}{rl}{Q}_{\pi }\left(s_t,a_t\right)& ={\mathbb{E}}_{{\mathcal{S}}_{t+1:},{\mathcal{A}}_{t+1:}}\left[U_t\mid S_t=s_t,A_t=a_t\right]\\ & ={\mathbb{E}}_{{\mathcal{S}}_{t+1:},{\mathcal{A}}_{t+1:}}\left[R_t+\gamma \cdot U_{t+1}\mid S_t=s_t,A_t=a_t\right]\\ & =\underset{(1)}{\underbrace{{\mathbb{E}}_{{\mathcal{S}}_{t+1},{\mathcal{A}}_{t+1}}\left[R_t|S_t=s_t,A_t=a_t\right]}}+\gamma \cdot \underset{(2)}{\underbrace{{\mathbb{E}}_{{\mathcal{S}}_{t+1:},{\mathcal{A}}_{t+1:}}\left[U_{t+1}\mid S_t=s_t,A_t=a_t\right]}}\end{array}$$
其中,$t$时刻的回报$R_t$只与$t+1$时刻的状态$S_{t+1}$有关，而$S_{t+1}$只与$S_t,A_t$有关,则
$$\begin{array}{rl}(1)& ={\mathbb{E}}_{{\mathcal{S}}_{t+1},{\mathcal{A}}_{t+1}}\left[R_t|S_t=s_t,A_t=a_t\right]\\ & ={\mathbb{E}}_{S_{t+1}}\left[R_t|S_t=s_t,A_t=a_t\right]\\ & ={\mathbb{E}}_{S_{t+1},A_{t+1}}\left[R_t|S_t=s_t,A_t=a_t\right]\end{array}$$
对$(2)$中的式子变形可得
$$\begin{array}{rl}(2)& ={\mathbb{E}}_{{\mathcal{S}}_{t+1:},{\mathcal{A}}_{t+1:}}\left[U_{t+1}\mid S_t=s_t,A_t=a_t\right]\\ & ={\mathbb{E}}_{S_{t+1},A_{t+1},{\mathcal{S}}_{t+2},{\mathcal{A}}_{t+2}}\left[U_{t+1}|S_t=s_t,A_t=a_t\right]\\ & ={\mathbb{E}}_{S_{t+1},A_{t+1}}\left[{\mathbb{E}}_{{\mathcal{S}}_{t+2},{\mathcal{A}}_{t+2}}\left[U_{t+1}|S_{t+1},A_{t+1},S_t=s_t,A_t=a_t\right]|S_t=s_t,A_t=a_t\right]利用马尔可夫性\\ & ={\mathbb{E}}_{S_{t+1},A_{t+1}}\left[{\mathbb{E}}_{{\mathcal{S}}_{t+2},{\mathcal{A}}_{t+2}}\left[U_{t+1}|S_{t+1},A_{t+1}\right]|S_t=s_t,A_t=a_t\right]\\ & ={\mathbb{E}}_{S_{t+1},A_{t+1}}\left[Q_{\pi }\left(S_{t+1},A_{t+1}\right)\mid S_t=s_t,A_t=a_t\right]\end{array}$$
由此证毕。

二.贝尔曼方程$\text{ (将 }{Q}_{\pi }\text{ 表示成 }{V}_{\pi }\text{ ) }$

Theorem :假设 $R_t$ 是 $S_t、A_t、S_{t+1}$ 的函数。那么
$$Q_{\pi }\left(s_t,a_t\right)={\mathbb{E}}_{S_{t+1}}\left[R_t+\gamma \cdot V_{\pi }\left(S_{t+1}\right)\mid S_t=s_t,A_t=a_t\right]\text{\hspace{1em}(1.2)}$$

proof:$\text{ 由于 }{V}_{\pi }\left(S_{t+1}\right)={\mathbb{E}}_{A_{t+1}\sim \pi \left(\cdot \mid S_{t+1}\right)}\left[Q\left(S_{t+1},A_{t+1}\right)\right]={\mathbb{E}}_{A_{t+1}}\left[Q_{\pi }(S_{t+1},A_{t+1})|S_{t+1}\right]$
$$\begin{array}{rl}(2)=& {\mathbb{E}}_{S_{t+1},A_{t+1}}\left[Q_{\pi }\left(S_{t+1},A_{t+1}\right)\mid S_t=s_t,A_t=a_t\right]\\ =& {\mathbb{E}}_{S_{t+1}}\left[{\mathbb{E}}_{A_{t+1}}\left[Q_{\pi }(S_{t+1},A_{t+1})|S_{t+1}\right]|S_t=s_t,A_t=a_t\right]\\ =& {\mathbb{E}}_{S_{t+1}}\left[V_{\pi }\left(S_{t+1}\right)|S_t=s_t,A_t=a_t\right]\end{array}$$
证毕

三.贝尔曼方程(将 $V_{\pi }$ 表示成 $V_{\pi }$ )

Theorem :假设 $R_t$ 是 $S_t、A_t、S_{t+1}$ 的函数。那么
$$V_{\pi }\left(s_t\right)={\mathbb{E}}_{A_t,S_{t+1}}\left[R_t+\gamma \cdot V_{\pi }\left(S_{t+1}\right)\mid S_t=s_t\right]\text{\hspace{1em}(1.3)}$$

proof:
$$\begin{array}{rl}{V}_{\pi }\left(s_t\right)& ={\mathbb{E}}_{A_t,{\mathcal{S}}_{t+1},{\mathcal{A}}_{t+1}}\left[U_t\mid S_t=s_t\right]\\ & ={\mathbb{E}}_{A_t,{\mathcal{S}}_{t+1},{\mathcal{A}}_{t+1}},\left[R_t+\gamma U_{t+1}|S_t=s_t\right]\\ & ={\mathbb{E}}_{A_t,{\mathcal{S}}_{t+1},{\mathcal{A}}_{t+1}}\left[R_t\mid S_t=s_t\right]+\gamma {\mathbb{E}}_{A_t,{\mathcal{S}}_{t+1},{\mathcal{A}}_{t+1}}\left[U_{t+1}\mid S_t=s_t\right]\\ & ={\mathbb{E}}_{A_t,S_{t+1}}\left[R_t\mid S_t=s_t\right]+\gamma {\mathbb{E}}_{S_{t+1}}\left[{\mathbb{E}}_{A_t{\mathcal{A}}_{t+1},{\mathcal{S}}_{t+2}}\left[U_{t+1}\mid S_{t+1},S_t=s_t\right]\mid S_t=s_t\right]\\ & ={\mathbb{E}}_{A_t,S_{t+1}}\left[R_t\mid S_t=s_t\right]+\gamma {\mathbb{E}}_{S_{t+1}}\left[E_{{\mathcal{A}}_{t+1},{\mathcal{S}}_{t+2}}\left[U_{t+1}\mid S_{t+1}\right]\mid S_t=s_t\right]马尔可夫性\\ & ={\mathbb{E}}_{A_t,S_{t+1}}\left[R_t\mid S_t=s_t\right]+\gamma {\mathbb{E}}_{S_{t+1}}\left[V_{\pi }(S_{t+1})\mid S_t=s_t\right]\\ & ={\mathbb{E}}_{A_t,S_{t+1}}\left[R_t\mid S_t=s_t\right]+\gamma {\mathbb{E}}_{A_t,S_{t+1}}\left[V_{\pi }(S_{t+1})\mid S_t=s_t\right]马尔可夫性\\ \text{证毕}& \end{array}$$
或者直接利用式$1.2$，两边同时对$A_t\sim \pi (\cdot |s_t)$求期望得
$$\begin{array}{rl}{\mathbb{E}}_{A_t\sim \pi (\cdot |s_t)}[Q_{\pi }\left(s_t,A_t\right)]& ={\mathbb{E}}_{A_t\sim \pi (\cdot |s_t)}[{\mathbb{E}}_{S_{t+1}}\left[R_t+\gamma \cdot V_{\pi }\left(S_{t+1}\right)\mid S_t=s_t,A_t\right]]\\ \Updownarrow & \\ {\mathbb{E}}_{A_t}[Q_{\pi }\left(S_t,A_t\right)\mid S_t=s_t]& ={\mathbb{E}}_{A_t}[{\mathbb{E}}_{S_{t+1}}\left[R_t+\gamma \cdot V_{\pi }\left(S_{t+1}\right)\mid S_t=s_t,A_t\right]\mid S_t=s_t]\\ & ={\mathbb{E}}_{S_{t+1},A_t}[R_t+\gamma \cdot V_{\pi }\left(S_{t+1}\right)\mid S_t=s_t]\end{array}$$

利用式$1.3$，进一步写出显示表达式可得
$$\begin{array}{rl}{V}_{\pi }(s_t)& ={\mathbb{E}}_{A_t,S_{t+1}}\left[R_t\mid S_t=s_t\right]+\gamma {\mathbb{E}}_{A_t,S_{t+1}}\left[V_{\pi }(S_{t+1})\mid S_t=s_t\right]\\ & ={\mathbb{E}}_{A_t}[{\mathbb{E}}_{S_{t+1}}[R_t\mid A_t,S_t=s_t]\mid S_t=s_t]+\gamma {\mathbb{E}}_{A_t}\left[{\mathbb{E}}_{S_{t+1}}\left[V_{\pi (S_{t+1})}\mid A_t,S_t=s_t\right]\mid S_t=s_t\right]\\ & =\sum _{A_t}\pi (a_t\mid s_t){\mathbb{E}}_{S_{t+1}}[R_t\mid A_t,S_t=s_t]+\gamma \sum _{A_t}\pi (a_t\mid s_t){\mathbb{E}}_{S_{t+1}}\left[V_{\pi (S_{t+1})}\mid A_t,S_t=s_t\right]\\ & =\sum _{A_t}\pi (a_t\mid s_t)\sum _{S_{t+1}}r\cdot p(s_{t+1}\mid s_t,a_t)+\gamma \sum _{A_t}\pi (a_t\mid s_t)\sum _{S_{t+1}}{V}_{\pi }(s_{t+1})\cdot p(s_{t+1}\mid s_t,a_t)\end{array}$$
其中$r=r(s_t,s_{t+1},a_t)$

四.最优贝尔曼方程

Theorem :假设 $R_t$ 是 $S_t、A_t、S_{t+1}$ 的函数。那么
$$Q_{\star }\left(s_t,a_t\right)={\mathbb{E}}_{S_{t+1}\sim p\left(\cdot \mid s_t,a_t\right)}\left[R_t+\gamma \cdot \underset{A\in \mathcal{A}}{\max }{Q}_{\star }\left(S_{t+1},A\right)\mid S_t=s_t,A_t=a_t\right]\text{\hspace{1em}(1.4)}$$

由贝尔曼方程可知
$$Q_{\star }\left(s_t,a_t\right)={\mathbb{E}}_{S_{t+1},A_{t+1}}\left[R_t+\gamma \cdot Q_{\star }\left(S_{t+1},A_{t+1}\right)\mid S_t=s_t,A_t=a_t\right]$$
因为动作 $A_{t+1}={\mathrm{argmax}}_A{Q}_{\star }\left(S_{t+1},A\right)$ 是状态 $S_{t+1}$ 的确定性函数, 所以
$$Q_{\star }\left(s_t,a_t\right)={\mathbb{E}}_{S_{t+1}}\left[R_t+\gamma \cdot \underset{A\in \mathcal{A}}{\max }{Q}_{\star }\left(S_{t+1},A\right)\mid S_t=s_t,A_t=a_t\right]$$

五.多步目标下的贝尔曼方程

设 $R_k$ 是 $S_k、A_k、S_{k+1}$ 的函数, $\forall k=1,\cdots ,n$ 。 那么
$$\underset{U_t\text{ 的期望 }}{\underbrace{Q_{\pi }\left(s_t,a_t\right)}}={\mathbb{E}}_{S_{t+1},A_{t+1},\cdots ,S_{t+m},A_{t+m}}[\left(\sum _{i=0}^{m-1}{\gamma }^i{R}_{t+i}\right)+{\gamma }^m\cdot \underset{U_{t+m}\text{ 的期望 }}{\underbrace{Q_{\pi }\left(S_{t+m},A_{t+m}\right)}}\mid S_t=s_t,A_t=a_t].$$

proof:设一局游戏的长度为 $n$ 。根据定义, $t$ 时刻的回报 $U_t$ 是 $t$ 时刻之后的所有奖励的加权和:
$$U_t=R_t+\gamma R_{t+1}+{\gamma }^2{R}_{t+2}+\cdots +{\gamma }^{n-t}{R}_n.$$

同理, $t+m$ 时刻的回报可以写成:
$$U_{t+m}=R_{t+m}+\gamma R_{t+m+1}+{\gamma }^2{R}_{t+m+2}+\cdots +{\gamma }^{n-t-m}{R}_n.$$

下面我们推导两个回报的关系。把 $U_t$ 写成:
$$\begin{array}{rl}{U}_t& =\left(R_t+\gamma R_{t+1}+\cdots +{\gamma }^{m-1}{R}_{t+m-1}\right)+\left({\gamma }^m{R}_{t+m}+\cdots +{\gamma }^{n-t}{R}_n\right)\\ & =\left(\sum _{i=0}^{m-1}{\gamma }^i{R}_{t+i}\right)+{\gamma }^m\underset{\text{等于 }{U}_{t+m}}{\underbrace{\left(R_{t+m}+\gamma R_{t+m+1}+\cdots +{\gamma }^{n-t-m}{R}_n\right)}}.\end{array}$$

因此, 回报可以写成这种形式:$U_t=\left({\sum }_{i=0}^{m-1}{\gamma }^i{R}_{t+i}\right)+{\gamma }^m{U}_{t+m}.$则
$$\begin{array}{rl}{Q}_{\pi }\left(s_t,a_t\right)& ={\mathbb{E}}_{{\mathcal{S}}_{t+1:},{\mathcal{A}}_{t+1:}}\left[U_t\mid S_t=s_t,A_t=a_t\right]\\ & ={\mathbb{E}}_{{\mathcal{S}}_{t+1:},{\mathcal{A}}_{t+1:}}\left[\left(\sum _{i=0}^{m-1}{\gamma }^i{R}_{t+i}\right)+{\gamma }^m{U}_{t+m}\mid S_t=s_t,A_t=a_t\right]\\ & =\underset{(1)}{\underbrace{{\mathbb{E}}_{{\mathcal{S}}_{t+1},{\mathcal{A}}_{t+1}}\left[\sum _{i=0}^{m-1}{\gamma }^i{R}_{t+i}|S_t=s_t,A_t=a_t\right]}}+{\gamma }^m\cdot \underset{(2)}{\underbrace{{\mathbb{E}}_{{\mathcal{S}}_{t+1:},{\mathcal{A}}_{t+1:}}\left[U_{t+m}\mid S_t=s_t,A_t=a_t\right]}}\end{array}$$

$$\begin{array}{rl}\text{其中(1)}& ={\mathbb{E}}_{{\mathcal{S}}_{t+1},{\mathcal{A}}_{t+1}}\left[\sum _{i=0}^{m-1}{\gamma }^i{R}_{t+i}|S_t=s_t,A_t=a_t\right]\\ & ={\mathbb{E}}_{S_{t+1},\cdots ,S_{t+m},A_{t+1},\cdots ,A_{t+m-1}}\left[\sum _{i=0}^{m-1}{\gamma }^i{R}_{t+i}|S_t=s_t,A_t=a_t\right]\text{最简形式，其他都与}{R}_t,\cdots ,R_{t+m-1}无关\\ & ={\mathbb{E}}_{S_{t+1},\cdots ,S_{t+m},A_{t+1},\cdots ,A_{t+m}}\left[\sum _{i=0}^{m-1}{\gamma }^i{R}_{t+i}|S_t=s_t,A_t=a_t\right]\end{array}$$

$$\begin{array}{rl}\text{其中(2)}& ={\mathbb{E}}_{{\mathcal{S}}_{t+1:},{\mathcal{A}}_{t+1:}}\left[U_{t+m}\mid S_t=s_t,A_t=a_t\right]\\ & ={\mathbb{E}}_{S_{t+1},A_{t+1},\cdots ,S_{t+m},A_{t+m},{\mathcal{S}}_{t+m+1},{\mathcal{A}}_{t+m+1}}\left[U_{t+m}\mid S_t=s_t,A_t=a_t\right]\\ & ={\mathbb{E}}_{S_{t+1},A_{t+1},\cdots ,S_{t+m},A_{t+m}}[{\mathbb{E}}_{{\mathcal{S}}_{t+m+1},{\mathcal{A}}_{t+m+1}}[U_{t+m}\mid S_{t+1},A_{t+1},\cdots ,S_{t+m},A_{t+m},S_t=s_t,A_t=a_t]\mid S_t=s_t,A_t=a_t]\\ & ={\mathbb{E}}_{S_{t+1},A_{t+1},\cdots ,S_{t+m},A_{t+m}}[{\mathbb{E}}_{{\mathcal{S}}_{t+m+1},{\mathcal{A}}_{t+m+1}}[U_{t+m}\mid S_{t+m},A_{t+m}]\mid S_t=s_t,A_t=a_t]\\ & ={\mathbb{E}}_{S_{t+1},A_{t+1},\cdots ,S_{t+m},A_{t+m}}[Q_{\pi }(S_{t+m},A_{t+m})\mid S_t=s_t,A_t=a_t]\end{array}$$

证毕