Policy Optimization

参考资料：https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html

Intro to Policy Optimization

本部分着重推导策略梯度的数学公式

关于Policy Gradient的简单求导

参数: ${\pi }_{\theta }$, 目标函数：最大化$J({\pi }_{\theta })=E_{\tau \sim {\pi }_{\theta }}[R(\tau )]$

梯度上升：
$${\theta }_{k+1}={\theta }_k+\alpha {▽}_{\theta }J({\pi }_{\theta }){|}_{{\theta }_k}$$

• ${▽}_{\theta }J({\pi }_{\theta })$被称作是Policy Gradient,以这种方式优化策略的方法被称作是policy gradient algorithms(包括Vanilla Policy Gradient和TRPO算法)

求导的具体步骤：

1.轨迹$\tau =(s_0,a_0,...,s_{T+1})$的概率
$$P(\tau |\theta )={\rho }_0(s_0)\prod _{t=0}^{T}P(s_{t+1}|s_t,a_t){\pi }_{\theta }(a_t|s_t)$$
2.求导技巧
$${▽}_{\theta }P(\tau |\theta )=P(\tau |\theta ){▽}_{\theta }logP(\tau |\theta )$$
3.关于轨迹的log概率
$$\begin{array}{rl}logP(\tau |\theta )& ={\rho }_0(s_0)\prod _{t=0}^{T}P(s_{t+1}|s_t,a_t){\pi }_{\theta }(a_t|s_t)\\ & =log{\rho }_0(s_0)+\sum _{t=0}^T(logP(s_{t+1}|s_t,a_t)+log{\pi }_{\theta }(a_t|s_t))\end{array}$$
4.关于log概率求导

• ${\rho }_0(s_0),P(s_{t+1}|s_t,a_t)与{\pi }_{\theta }无关，所以关于其求导为零$
  $$\begin{array}{rl}{▽}_{\theta }logP(\tau |\theta )& ={▽}_{\theta }{\rho }_0(s_0)\prod _{t=0}^{T}P(s_{t+1}|s_t,a_t){\pi }_{\theta }(a_t|s_t)\\ & ={▽}_{\theta }log{\rho }_0(s_0)+{▽}_{\theta }\sum _{t=0}^T(logP(s_{t+1}|s_t,a_t)+log{\pi }_{\theta }(a_t|s_t))\\ & =\sum _{t=0}^T{▽}_{\theta }log{\pi }_{\theta }(a_t|s_t)\end{array}$$

5.综上所述
$$\begin{array}{rl}{▽}_{\theta }J({\pi }_{\theta })& ={▽}_{\theta }{E}_{\tau \sim {\pi }_{\theta }}[R(\tau )]\\ & ={▽}_{\theta }{\int }_{\tau }P(\tau |\theta )R(\tau )\\ & ={\int }_{\tau }{▽}_{\theta }P(\tau |\theta )R(\tau )\\ & ={\int }_{\tau }P(\tau |\theta ){▽}_{\theta }logP(\tau |\theta )R(\tau )\\ & =E_{\tau \sim {\pi }_{\theta }}[{▽}_{\theta }log{\pi }_{\theta }(a_t|s_t)R(\tau )]\end{array}$$
$$\Rightarrow {▽}_{\theta }J({\pi }_{\theta })=E_{\tau \sim {\pi }_{\theta }}[\sum _{t=0}^T{▽}_{\theta }log{\pi }_{\theta }(a_t|s_t)R(\tau )]$$