[RL 9] Trust Region Policy Optimization (ICML, 2015)

Trust Region Policy Optimization (ICML, 2015)

1 Introduction

1. policy optimization categories
   1. policy iteration (GPI)
   2. PG (e.g. TRPO)
   3. derivative-free(无导数) optimization methods

2 Preliminaries

1. Consider an infinite-horizon discounted MDP
   1. instead an average reward one
2. objective measurement $\eta (\tilde{\pi })$ (Kakade, 2002, TODO)
   $$\begin{gathered} \eta (\tilde{\pi })=\eta (\pi )+{\mathbb{E}}_{s_0,a_0,\cdots \sim \tilde{\pi }}\left[\sum _{t=0}^{\infty }{\gamma }^t{A}_{\pi }\left(s_t,a_t\right)\right] \\ =\eta (\pi )+\sum _s{\rho }_{\tilde{\pi }}(s)\sum _a\tilde{\pi }(a\mid s)A_{\pi }(s,a) \end{gathered}$$
   1. improving the expectation on right hand side leads to policy improvment
   2. but ${\rho }_{\tilde{\pi }}(s)$ is hard to estimate
   3. so use local approximation instead
3. local approximation objective measurement $L_{\pi }(\tilde{\pi })$ (Kakade, 2002, TODO)
   $$L_{\pi }(\tilde{\pi })=\eta (\pi )+\sum _s{\rho }_{\pi }(s)\sum _a\tilde{\pi }(a\mid s)A_{\pi }(s,a)$$
   1. match $\eta (\tilde{\pi })$ to first order
   2. lower bound
   3. limited form of policy improvement

3 Monotonic Improvement Guarantee for General Stochastic Policies

1. Theorem 1: lower bound with General Stochastic Policies
   $$\begin{array}{rl}\eta (\tilde{\pi })& \ge L_{\pi }(\tilde{\pi })-C{D}_{\mathrm{K}\mathrm{L}}^{\max }(\pi ,\tilde{\pi })\\ & \text{ where }C=\frac{4ϵ\gamma }{(1-\gamma {)}^2},ϵ=\underset{s,a}{\max }\left|A_{\pi }(s,a)\right|\end{array}$$
   • KL can be seen as penalty
   • $C$ is a large constant, thus leads to a small step size
2. Algorithm 1: monotonic policy iteration
   • advantage should be accurate
   • can be extend to continunous state and action sapce

4 Optimization of Parameterized Policies

1. Approximations
   1. trust region constrain
      • step size in Algorithm 1 could be very small
      • change KL penalty to a constrain
   2. average KL
      • max operation involving traversal the whole state space
      • use average KL instead
        ${\bar{D}}_{\mathrm{K}\mathrm{L}}^{{\rho }_{old}}\left({\theta }_1,{\theta }_2\right):={\mathbb{E}}_{s\sim {\rho }_{old}}\left[D_{\mathrm{K}\mathrm{L}}\left({\pi }_{{\theta }_1}(\cdot \mid s)\Vert {\pi }_{{\theta }_2}(\cdot \mid s)\right)\right]$
2. optimization problem up to now
   $$\begin{array}{c}\underset{\theta }{\mathrm{maximize}}{\sum }_s{\rho }_{{\theta }_{\text{old }}}(s){\sum }_a{\pi }_{\theta }(a\mid s)A_{{\theta }_{\text{old }}}(s,a)\\ \text{ subject to }{\bar{D}}_{\mathrm{K}\mathrm{L}}^{{\rho }_{\text{old }}}\left({\theta }_{\text{old }},\theta \right)\le \delta \end{array}$$

5 Sample-Based Estimation of the Objective and Constraint

$$\begin{array}{l}\underset{\theta }{\mathrm{maximize}}{\mathbb{E}}_{s\sim {\rho }_{\theta \mathrm{o}\mathrm{l}\mathrm{d}},a\sim q}\left[\frac{{\pi }_{\theta }(a\mid s)}{q(a\mid s)}{Q}_{{\theta }_{\mathrm{o}\mathrm{l}\mathrm{d}}}(s,a)\right]\\ \text{ subject to }{\mathbb{E}}_{s\sim {\rho }_{\theta \mathrm{o}\mathrm{l}\mathrm{d}}}\left[D_{\mathrm{K}\mathrm{L}}\left({\pi }_{{\theta }_{\mathrm{o}\mathrm{l}\mathrm{d}}}(\cdot \mid s)\Vert {\pi }_{\theta }(\cdot \mid s)\right)\right]\le \delta \end{array}$$

1. $s\sim {\rho }_{\theta \mathrm{o}\mathrm{l}\mathrm{d}}$
   • by defination of expectation
2. $a\sim q$
   • by important sampling
3. $Q_{{\theta }_{\mathrm{o}\mathrm{l}\mathrm{d}}}(s,a)$
   • by the fact that
     1. $A(s,a)=Q(s,a)-V(s)$
     2. ${\sum }_s{V}^{{\theta }_{old}}(s)=constant$
   • so that MC can be used to estimate Q (mabey there is no proper MC estimation for advantage)
   • estimation methods
     1. single path
     2. Vine
        • better estimation
        • but need more simulator
        • limit to the system where states can be recover

6 Practical Algorithm

1. Update Q by collected trajectoriess
2. Calulate objective and its constrain
3. Solve the constrain problem by Conjugate gradient alg TODO
   • references
     1. paper Appendix C
     2. post with thorem and implement https://www.telesens.co/2018/06/09/efficiently-computing-the-fisher-vector-product-in-trpo/

8 Experiments

1. TRPO is related to prior methods (e.g. natural policy
   gradient) but makes several changes, most notably by using a fixed KL divergence rather than a fixed penalty coefficient.
   • These results provide empirical evidence that constraining the KL divergence is a more robust way to choose step sizes and make fast, consistent progress, compared to using a fixed penalty.
2. can obtain high-quality locomotion controllers from scratch, which is considered to be a hard problem.
3. the method we proposed is scalable and has strong theoretical foundations.