强化学习—— Trust Region Policy Optimization (TRPO算法)

1 Trust Region Algorithm 置信域算法

problem：
$${\theta }^{\star }=\underset{\theta }{\mathrm{argmax}}J(\theta )$$
repeat：

1. Approximation: 给定 ${\theta }_{old}$, 构建$L(\theta |{\theta }_{old})$去近似$J(\theta )$，其中$\theta$需要满足${\theta }_{old}$的置信域，即$N({\theta }_{old})$。
2. Maximization: 在置信域内，求取优化后的$\theta$：$${\theta }_{new}=\underset{\theta \in N({\theta }_{old})}{\mathrm{argmax}}L(\theta |{\theta }_{old})$$

2 Trust Region Policy Optimization (TRPO算法)

• state-value function:
  $$V_{\pi }(s)=\sum _a\pi (a|s;\theta )Q(s,a)=E_{A\pi }[Q_{\pi }(s,A)]$$
• objective function:
  $$J(\theta )=E_S[V_{\pi }(S)]$$
• approximation:
  $$V_{\pi }(s)=\sum _a\frac{\pi (a|s;\theta )}{\pi (a|,s;{\theta }_{old})}\cdot Q_{\pi }(s,a)\cdot \pi (a|s;{\theta }_{old})=E_{A\pi (\cdot |s;{\theta }_{old})}[\frac{\pi (A|s;\theta )}{\pi (A|s;{\theta }_{old})}\cdot Q_{\pi }(s,A)]$$
  $$J(\theta )=E_S[E_A[\frac{\pi (A|S;\theta )}{\pi (A|S;{\theta }_{old})}\cdot Q_{\pi }(S,A)]]$$
• trajectory from $\pi (a,|s;{\theta }_{old})$:
  $$s_1,a_1,r_1,s_2,a_2,r_2,...,s_n,a_n,r_n$$
• Montel Carlo Approximation:
  $$L(\theta |{\theta }_{old})=\frac{1}{n}\sum _{i=1}^n\frac{\pi (a_i|s_i;\theta )}{\pi (a_i|s_i;{\theta }_{old})}\cdot Q_{\pi }(s_i,a_i)$$
  $$Q_{\pi }(s_i,a_i)\approx u_i=r_i+\gamma r_{i+1}+...+{\gamma }^{n-i}{r}_n$$
  $$\tilde{L}(\theta |{\theta }_{old})=\frac{1}{n}\sum _{i=1}^n\frac{\pi (a_i|s_i;\theta )}{\pi (a_i|s_i;{\theta }_{old})}\cdot u_i$$
• $\theta \in N({\theta }_{old})$

option1:$||\theta -{\theta }_{old}||<\Delta$

option2: $\frac{1}{n}{\sum }_{i=1}^{n}KL[\pi (\cdot |s_i;\theta )||\pi (\cdot |s_i;{\theta }_{old})]<\Delta$

3. Summary for TRPO

1. trajectory from $\pi (\cdot |s;{\theta }_{old})$
   $$s_1,a_1,r_1,s_2,a_2,r_2,...,s_n,a_n,r_n$$
2. dicounted returns
   $$u_i=\sum _{k=i}^n{\gamma }^{k-i}\cdot r_k$$
3. approximation:
   $$\tilde{L}(\theta |{\theta }_{old})=\sum _{i=1}^n\frac{\pi (a_i|s_i;\theta )}{\pi (a_i|s_i;{\theta }_{old})}\cdot u_i$$
4. maximization:
   $${\theta }_{new}=\underset{\theta \in N({\theta }_{old})}{\mathrm{argmax}}\tilde{L}(\theta |{\theta }_{old})$$
   by CyrusMay 2022 07 14