【RL】MDP(2)

前文链接

MDP(1)

策略迭代

策略评估

矩阵算法

在动态特性$\mathcal{P}(s^{\prime },r^{\prime }\mid s,a)$，给定$\pi$得到$q_{\pi }(s,a)$.
令
$$V_{\pi }(s)={\left(\begin{array}{c}{V}_{\pi }(s_1)\\ V_{\pi }(s_2)\\ ⋮\\ V_{\pi }(s_{|s|})\end{array}\right)}_{|s|\times 1}$$
根据$V_{\pi }(s)$表达式可以得到
$$\begin{array}{rl}{V}_{\pi }(s)& ={\mathbb{E}}_{\pi }[G_t\mid S_t=s]\\ & ={\mathbb{E}}_{\pi }[R_{t+1}+\gamma V_{\pi }{S}_{t+1}\mid S_t=s]\\ & =\sum _a\pi (a\mid s_i)\sum _{s^{\prime },r}\mathcal{P}(s^{\prime },r\mid s,a)[r+\gamma V_{\pi }(s^{\prime })]\end{array}$$
展开得到
$$\begin{array}{r}{V}_{\pi }(s)=\underset{(A)}{\underbrace{\sum _a\pi (a\mid s)\sum _{s^{\prime },r}\mathcal{P}(s^{\prime },r\mid s,a)r}}+\underset{(B)}{\underbrace{\gamma \sum _a\pi (a\mid s)\sum _{s^{\prime },r}\mathcal{P}(s^{\prime },r\mid s,a)V_{\pi }(s^{\prime })}}\end{array}$$
其中$(A)$部分可以表示为
$$\sum _a\pi (a\mid s)\sum _{r}r\mathcal{P}(r\mid s,a)=\sum _a\pi (a\mid s)\mathbb{E}[R_{t+1}\mid S_t=s,A_t=a]$$
令$r(s,a)\triangleq \mathbb{E}[R_{t+1}\mid S_t=s,A_t=a]$
得到
$$(A)=\sum _a\pi (a\mid s)r(s,a)=r_{\pi }(s)$$
引入向量$r_{\pi }$
$$r_{\pi }={\left(\begin{array}{c}{r}_{\pi }(s_1)\\ r_{\pi }(s_2)\\ ⋮\\ r_{\pi }(s_{|s|})\end{array}\right)}_{|s|\times 1}$$
$(B)$部分值为
$$\begin{array}{rl}(B)& =\gamma \sum _a\pi (a\mid s)\sum _{s^{\prime }}\mathcal{P}(s^{\prime }\mid s,a)V_{\pi }(s^{\prime })\\ & =\gamma \sum _{s^{\prime }}\underset{P_{\pi }(s,s^{\prime })}{\underbrace{\sum _a\pi (a\mid s)\mathcal{P}(s^{\prime }\mid s,a)}}{V}_{\pi }(s^{\prime })\\ & =\gamma \sum _{s^{\prime }}{P}_{\pi }(s,s^{\prime })V_{\pi }(s^{\prime })\end{array}$$
令$P_{\pi }\triangleq [P_{\pi }(s,s^{\prime })]$，得到
$$V_{\pi }=r_{\pi }+\gamma P_{\pi }{V}_{\pi }$$
解出
$$V_{\pi }=(\mathbf{I}-\gamma P_{\pi }{)}^{-1}{r}_{\pi }$$
算法时间复杂度为$\mathcal{O}(|s{|}^3)$.

迭代法

给定$\pi$，求出$V_{\pi }$.
$$V_{\pi }(s)=\sum _a\pi (a\mid s)\sum _{s^{\prime },r}\mathcal{P}(s^{\prime },r\mid s,a)(r+\gamma V_{\pi }(s^{\prime }))$$
构造一个数列 { V k } k = 1 ∞ → V π \{V_k\}_{k=1}^\infty\to V_\pi {Vk​}k=1∞​→Vπ​
迭代方程为
$$V_{k+1}(s)\triangleq \sum _a\pi (a\mid s)\sum _{s^{\prime },r}\mathcal{P}(s^{\prime },r\mid s,a)(r+\gamma V_k(s^{\prime }))$$

策略改进

求出$q_{\pi }(s,a)$构造出新的${\pi }^{\prime }$，使得$q_{{\pi }^{\prime }}(s,a)>q_{\pi }(s,a)$.

策略改进定理：给定$\pi ,{\pi }^{\prime }$如果$\forall s\in S$, $q_{\pi }(s,{\pi }^{\prime }(s))$，那么则有$\forall s\in S,V_{{\pi }^{\prime }}(s)\ge V_{\pi }(s)$

证明：
根据公式
$$\begin{array}{rl}{q}_{\pi }(s,a)& =\sum _{s^{\prime },r}\mathcal{P}(s^{\prime },r\mid s,a)[r+\gamma V_{\pi }(s^{\prime })]\\ & =\mathbb{E}[R_{t+1}+\gamma V_{\pi }(S_{t+1})\mid S_t=s,A_t=a]\\ & =\mathbb{E}[R_{t+1}+\gamma V_{\pi }(S_{t+1})\mid S_t=s,A_t={\pi }^{\prime }(s)]\\ & ={\mathbb{E}}_{{\pi }^{\prime }}[R_{t+1}+\gamma V_{\pi }(S_{t+1})\mid S_t=s]\\ & \le {\mathbb{E}}_{{\pi }^{\prime }}[R_{t+1}+\gamma q_{\pi }(S_{t+1},{\pi }^{\prime }(S_{t+1}))\mid S_t=s]\\ & ={\mathbb{E}}_{{\pi }^{\prime }}[R_{t+1}+\gamma {\mathbb{E}}_{{\pi }^{\prime }}[R_{t+2}+\gamma V_{\pi }(S_{t+2})\mid S_{t+1}]\mid S_t=s]\\ & ={\mathbb{E}}_{{\pi }^{\prime }}[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{V}_{\pi }(S_{t+2})\mid S_t=s]\\ & \dots \\ & \le {\mathbb{E}}_{{\pi }^{\prime }}[\underset{G_t}{\underbrace{R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\dots }}\mid S_t=s]\\ & =V_{{\pi }^{\prime }}(s)\end{array}$$

贪心策略(Greedy Policy): $\forall s\in S{\pi }^{\prime }(s)=\underset{a}{\mathrm{argmax}}{q}_{\pi }(s,a)$

$$V_{\pi }(s)\le \underset{a}{\max }{q}_{\pi }(s,a)=q_{\pi }(s,{\pi }^{\prime }(s))$$
由策略改进定理可知
$$\forall s\in S,V_{{\pi }^{\prime }}(s)\ge V_{\pi }(s)$$

如果$V_{{\pi }^{\prime }}=V_{\pi }$那么$V_{{\pi }^{\prime }}=V_{\pi }=V_{\ast }$

证明：
当$V_{{\pi }^{\prime }}=V_{\pi }\Rightarrow q_{{\pi }^{\prime }}=q_{\pi }$.
所以
$$\begin{array}{rl}& \forall s\in S\\ V_{{\pi }^{\prime }}(s)& =\sum _a{\pi }^{\prime }(a\mid s)q_{{\pi }^{\prime }}(s,a)q_{\pi }(s,a)\\ & =q_{\pi }(s,{\pi }^{\prime }(s))\\ & =\underset{a}{\max }{q}_{\pi }(s,a)\\ & =\underset{a}{\max }\sum _{s^{\prime },r}\mathcal{P}(s^{\prime },r\mid s,a)[r+\gamma V_{{\pi }^{\prime }}(s^{\prime })]\end{array}\text{\hspace{1em}(1)}$$
可以发现$(1)$是Bellman Optimality Equation.

价值迭代

价值迭代时极端情况下的策略迭代
$$V_{k+1}(s)=\underset{a}{\max }\sum _{s^{\prime },r}\mathcal{P}(s^{\prime },r\mid s,a)[r+\gamma V_k(s^{\prime })]$$