Policy-based PPO

PPO

PPO在PG的基础上引入了重要性采样，反推得到损失函数

特点

1.策略${\pi }_{{\theta }^{\prime }}$

PPO引入了另一个策略${\pi }_{{\theta }^{\prime }}$来采样，并用${\pi }_{{\theta }^{\prime }}$采集的样本训练${\pi }_{\theta }$

2.重要性采样

$${\mathbb{E}}_{x\backsim p}[f(x)]={\mathbb{E}}_{x\backsim q}[f(x)\frac{p(x)}{q(x)}]$$
注：重要性采样和on-policy、off-policy无关

3.两个网络

critic网络：输出价值v
actor网络：输出动作

3.约束项

KL散度，用来度量两个概率分布相似度的指标

更新公式

PPO

$$\begin{gathered} J_{PPO}^{{\theta }^{\prime }}(\theta )=J^{{\theta }^{\prime }}(\theta )-\beta KL(\theta ,{\theta }^{\prime }) \\ {J}^{{\theta }^{\prime }}(\theta )={\mathbb{E}}_{(s_t,a_t)\backsim {\pi }_{{\theta }^{\prime }}}[A^{{\theta }^{\prime }}(s_t,a_t)\frac{p_{\theta }(a_t|s_t)}{p_{{\theta }^{\prime }}(a_t|s_t)}] \end{gathered}$$
优势函数也用${\theta }^{\prime }$的原因：跟环境互动的是${\pi }_{{\theta }^{\prime }}$，期望奖励和也是在这个策略下求的

技巧

1.惩罚

$$\begin{gathered} J_{PPO}^{{\theta }_k}(\theta )=J^{{\theta }_k}(\theta )-\beta KL(\theta ,{\theta }_k) \\ {J}^{{\theta }_k}(\theta )\approx \sum _{(s_t,a_t)}[A^{{\theta }_k}(s_t,a_t)\frac{p_{\theta }(a_t|s_t)}{p_{{\theta }^{\prime }}(a_t|s_t)}] \end{gathered}$$

2.裁剪

$$J_{PPO2}^{{\theta }_k}(\theta )\approx \sum _{(s_t,a_t)}min(\frac{p_{\theta }(a_t|s_t)}{p_{{\theta }^{\prime }}(a_t|s_t)}{A}^{{\theta }_k}(s_t,a_t),clip(\frac{p_{\theta }(a_t|s_t)}{p_{{\theta }^{\prime }}(a_t|s_t)},1-\epsilon ,1+\epsilon )A^{{\theta }_k}(s_t,a_t))$$