强化学习笔记-TRPO（1）基础理论部分

TRPO

1. Base-line

• policy- Gradient

  策略梯度算法，是一个强化学习中非常重要的算法，他描述的是$\pi (a|s,\theta )$,即，在网络参数为$\theta$ 时，当状态（环境）为s时候，采取哪种状态a
  $$\frac{\nabla V_{\pi }(s)}{\nabla \theta }=E_{A\pi }[\frac{\nabla ln\pi (A|S,\theta )}{\nabla \theta }\ast Q_{\pi }(S,A)]$$

• Baseline

​ 存在一个常数b，它不依赖于action A
$$\begin{gathered} E_{A\pi }[\frac{\nabla ln\pi (A|S,\theta )}{\nabla \theta }\ast b] \\ =b\ast E_{A\pi }[\frac{\nabla ln\pi (A|S,\theta )}{\nabla \theta }] \\ 根据期望的定义=b\ast \sum _a\pi (A|S,\theta )\ast [\frac{\nabla ln\pi (A|S,\theta )}{\nabla \theta }] \\ lnx的导数=b\ast \sum _a\frac{\nabla \pi (a|s,\theta )}{\nabla \theta }=0(求和与求导对象不同) \end{gathered}$$
所以存在一个不依赖于A的常数b,似的期望为0。所以考虑在梯度下降中添加bias ，虽然不会影响结果，但是会影响MC近似，让估计到的Q会和真实的数值更加接近
$$\frac{\nabla V_{\pi }(s)}{\nabla \theta }=E_{A\pi }[\frac{\nabla ln\pi (A|S,\theta )}{\nabla \theta }\ast （Q_{\pi }(S,A)-b)]$$

• 常见Baseline

  $b=V_{\pi }(s_t),V_{\pi }(s_t)S_t$,

  b = 0

2. Trust Region Policy Optimization（TRPO）

2.1 Optimization

Gradient Ascent
$$
Find \theta^* = argmax_\theta J(\theta)

$$

• $\theta_{old} $
  $$g={\frac{\nabla J(\theta )}{\nabla \theta }}_{\theta ={\theta }_{old}}$$

• ${\theta }_{new}\leftarrow {\theta }_{old}+\alpha \ast g$

但是梯度不太容易计算，所以采用随机梯度下降算法$J(\theta )=E_S[V(S;\theta )]$

Random Gradient

• $S \leftarrow $random sampling

• $g={\frac{\nabla J(\theta )}{\nabla \theta }}_{\theta ={\theta }_{old}}$

• ${\theta }_{new}\leftarrow {\theta }_{old}+\alpha \ast g$

2.2 Trust Region

$N({\theta }_{old})$表示${\theta }_{old}$ 的邻域
N ( θ o l d ) = { θ ∣ ∣ ∣ θ − θ o l d ∣ ∣ 2 ≤ Δ } N(\theta_{old}) = \{\theta | ||\theta-\theta_{old}||_2 \leq \Delta\} N(θold​)={θ∣∣∣θ−θold​∣∣2​≤Δ}
在邻域内，用$J(\theta )表示L(\theta |{\theta }_{old})$

步骤

• Approximation ：$J(\theta )=L(\theta |{\theta }_{old})$
• Maximization：${\theta }_{new}\leftarrow argama{x}_{\theta \in N({\theta }_{old})}L(\theta |{\theta }_{old})$

2.3 Policy-Based RL

• $\pi (a|s;\theta )$

$$\begin{gathered} V_{\pi }=E_{A-\pi }[Q_{\pi }(S,A)]=\sum _a\pi (a|s,\theta )\ast Q_{\pi }(s,a) \\ =\sum _a\pi (a|s,{\theta }_{old})\ast Q_{\pi }(s,a)\ast \frac{\pi (a|s,\theta )}{\pi (a|s,{\theta }_{old})} \\ {E}_{A-\pi (.|s,{\theta }_{old})}[Q_{\pi }(s,a)\ast \frac{\pi (a|s,\theta )}{\pi (a|s,{\theta }_{old})}] \end{gathered}$$

• $J(\theta )=E_s[V_{\pi }(s)]=E_s[E_{A-\pi (.|s,{\theta }_{old})}[Q_{\pi }(s,a)\ast \frac{\pi (a|s,\theta )}{\pi (a|s,{\theta }_{old})}]]$

利用重要性采样，我们通过旧的策略来找到新策略的状态转移函数

2.4 TRPO

policy gradient

• 学习不够稳定，超参数对结果影响很大，容易从局部最优走到不好的策略

• sample effiency：采样效率的利用很低。

two step

• $L(\theta |{\theta }_{old})$

  1.玩一局游戏，得到$Q_{\pi }(s_i,a_i):R:r_1,r_2,r_3,r_4,,,,r_n$

  2.重要性采样,估计出
  $$L(\theta |{\theta }_{old})=\sum _{i=1}^n{Q}_{\pi }(s,a)\ast \frac{\pi (a|s,\theta )}{\pi (a|s,{\theta }_{old})}$$
  ​ 这里的Q由采样的u决定

  3.$ argmax = L(\theta | \theta_{old})$

• ${\theta }_{new}\leftarrow argmax$
  $$\begin{gathered} {\theta }_{new}\leftarrow argma{x}_{\theta }L(\theta |{\theta }_{old}) \\ \theta \in N({\theta }_{old}) \end{gathered}$$

  1. $||\theta -{\theta }_{old}<\Delta ||$
  2. KL 散度，来衡量两个分布之间距离足够远