[RL 6] Deterministic Policy Gradient Algorithms (ICML, 2014)

Deterministic Policy Gradient Algorithms (ICML, 2014)

Stochastic PGT (SPGT)

1. Theorem
   $\begin{array}{rl}{\nabla }_{\theta }J\left({\pi }_{\theta }\right)& ={\int }_{\mathcal{S}}{\rho }^{\pi }(s){\int }_{\mathcal{A}}{\nabla }_{\theta }{\pi }_{\theta }(a\mid s)Q^{\pi }(s,a)\mathrm{d}a\mathrm{d}s\\ & ={\mathbb{E}}_{s\sim {\rho }^{\pi },a\sim {\pi }_{\theta }}\left[{\nabla }_{\theta }\log {\pi }_{\theta }(a\mid s)Q^{\pi }(s,a)\right]\end{array}$
   • Proof see: https://web.stanford.edu/class/cme241/lecture_slides/PolicyGradient.pdf
2. PGT derived algorithms
   1. on-policy AC
      1. actor update: PGT
      2. critic update: any TD leanring
   2. off-policy AC
      1. actor update: off-PGT (TODO proof see Degris 2012)
      2. critic: any TD (or TODO more general GAE)

Intuition DGPT

1. Greedy policy improvement in GPI
   • argmax Q is not suitable for continues action space
2. DGPT
   • move the policy in the direction of the gradient of Q, rather than globally maximising Q.
   • 思想同argmax, 改变policy选择Q value较大的action

Formal DPGT

1. Settings
   • episode
   • with discount factor
   • for continous task, set $\gamma =1$, and use state distribution ${\mu }_{\theta }(S)$ in RL charpt 9
2. on-policy
   1. Objective
      $$J\left({\mu }_{\theta }\right)={\int }_{\mathcal{S}}{p}_1(s)V^{{\mu }_{\theta }}(s)\mathrm{d}s$$
   2. Theorem
      1. on-policy DPG
         $$\begin{gathered} \begin{array}{rl}{\nabla }_{\theta }J\left({\mu }_{\theta }\right)& ={{\int }_{\mathcal{S}}{\rho }^{\mu }(s){\nabla }_{\theta }{\mu }_{\theta }(s){\nabla }_a{Q}^{\mu }(s,a)|}_{a={\mu }_{\theta }(s)}\mathrm{d}s\\ & ={\mathbb{E}}_{s\sim {\rho }^{\mu }}\left[{{\nabla }_{\theta }{\mu }_{\theta }(s){\nabla }_a{Q}^{\mu }(s,a)|}_{a={\mu }_{\theta }(s)}\right]\end{array} \\ where,{\rho }^{\mu }(s)={\int }_{\mathcal{S}}\sum _{t=1}^{\infty }{\gamma }^{t-1}{p}_1(s)p\left(s\to s^{\prime },t,\pi \right)\mathrm{d}s \end{gathered}$$
         1. discount state distribution ${\rho }^{\mu }(s)$:
            • 定义: 状态分布, 可以理解为:依据policy ${\mu }_{\theta }$, 遇到这个状态的概率
            • 计算: 求和在所有time step t遇到s的概率, 通过gamma加权
            • 从该分布采样: 直接用policy ${\mu }_{\theta }$与环境交互即可, 因为policy考虑最大化accumulative reward, 越靠后的reward(以及相应的state)权重越小
   3. Regularity Conditions
      $$\begin{array}{rl}& \text{ Regularity conditions A.1: }p\left(s^{\prime }\mid s,a\right),{\nabla }_{a}p\left(s^{\prime }\mid s,a\right),{\mu }_{\theta }(s),{\nabla }_{\theta }{\mu }_{\theta }(s),r(s,a),{\nabla }_{a}r(s,a),p_1(s)\text{ are continuous in all }\\ & \text{ parameters and variables }s,a,s^{\prime }\text{ and }x\text{ . }\\ & \text{ Regularity conditions A.2: there exists a }b\text{ and }L\text{ such that }\underset{s}{\sup }{p}_1(s)<b,\underset{a,s,s^{\prime }}{\sup }p\left(s^{\prime }\mid s,a\right)<b,\underset{a,s}{\sup }r(s,a)<b\\ & \underset{a,s,s^{\prime }}{\sup }\Vert {\nabla }_{a}p\left(s^{\prime }\mid s,a\right)\Vert <L,\text{ and }\underset{a,s}{\sup }\Vert {\nabla }_{a}r(s,a)\Vert <L\end{array}$$
      • A.1 保证了V可对$\theta$求导, 同时使得推导过程可以使用
        1. Leibniz积分公式, 改变微分和积分次序: $\nabla \int$-> $\int \nabla$
        2. Fubini定理, 改变积分次序
      • A.2 保证了梯度有界
   4. Part of Proof
      $$\begin{array}{rl}{\nabla }_{\theta }{V}^{{\mu }_{\theta }}(s)=& {\nabla }_{\theta }{Q}^{{\mu }_{\theta }}\left(s,{\mu }_{\theta }(s)\right)\\ =& {\nabla }_{\theta }\left(r\left(s,{\mu }_{\theta }(s)\right)+{\int }_{\mathcal{S}}\gamma p\left(s^{\prime }\mid s,{\mu }_{\theta }(s)\right)V^{{\mu }_{\theta }}\left(s^{\prime }\right)\mathrm{d}{s}^{\prime }\right)\\ =& {{\nabla }_{\theta }{\mu }_{\theta }(s){\nabla }_{a}r(s,a)|}_{a={\mu }_{\theta }(s)}+{\nabla }_{\theta }{\int }_{\mathcal{S}}\gamma p\left(s^{\prime }\mid s,{\mu }_{\theta }(s)\right)V^{{\mu }_{\theta }}\left(s^{\prime }\right)\mathrm{d}{s}^{\prime }\\ =& {{\nabla }_{\theta }{\mu }_{\theta }(s){\nabla }_{a}r(s,a)|}_{a={\mu }_{\theta }(s)}\\ & +{\int }_{\mathcal{S}}\gamma \left(p\left(s^{\prime }\mid s,{\mu }_{\theta }(s)\right){\nabla }_{\theta }{V}^{{\mu }_{\theta }}\left(s^{\prime }\right)+{{\nabla }_{\theta }{\mu }_{\theta }(s){\nabla }_{a}p\left(s^{\prime }\mid s,a\right)|}_{a={\mu }_{\theta }(s)}{V}^{{\mu }_{\theta }}\left(s^{\prime }\right)\right)\mathrm{d}{s}^{\prime }\\ =& {{\nabla }_{\theta }{\mu }_{\theta }(s){\nabla }_a\left(r(s,a)+{\int }_{\mathcal{S}}\gamma p\left(s^{\prime }\mid s,a\right)V^{{\mu }_{\theta }}\left(s^{\prime }\right)\mathrm{d}{s}^{\prime }\right)|}_{a={\mu }_{\theta }(s)}\\ & +{\int }_{\mathcal{S}}\gamma p\left(s^{\prime }\mid s,{\mu }_{\theta }(s)\right){\nabla }_{\theta }{V}^{{\mu }_{\theta }}\left(s^{\prime }\right)\mathrm{d}{s}^{\prime }\\ =& {{\nabla }_{\theta }{\mu }_{\theta }(s){\nabla }_a{Q}^{{\mu }_{\theta }}(s,a)|}_{a={\mu }_{\theta }(s)}+{\int }_{\mathcal{S}}\gamma p\left(s\to s^{\prime },1,{\mu }_{\theta }\right){\nabla }_{\theta }{V}^{{\mu }_{\theta }}\left(s^{\prime }\right)\mathrm{d}{s}^{\prime }\end{array}$$
      从标红的地方,可以知道:
      1. Action空间必须是连续的: 要求函数${\mu }_{\theta }$ (S->A的映射) 连续, 即要求action space A是连续的. 同时, 由于不涉及输出为state的函数, 所以实际建模的MDP中state space不一定需要连续, 只要以state作为输入的函数, 对state有连续的定义域
      2. reward function 连续
      3. transition function (概率函数) 连续
      4. TODO 试证明discrete state space & continue action space 的 PGT & DPG
3. off-policy DPG
   1. Objective
      $\begin{array}{rl}{J}_{\beta }\left({\mu }_{\theta }\right)& ={\int }_{\mathcal{S}}{\rho }^{\beta }(s)V^{\mu }(s)\mathrm{d}s\\ & ={\int }_{\mathcal{S}}{\rho }^{\beta }(s)Q^{\mu }\left(s,{\mu }_{\theta }(s)\right)\mathrm{d}s\end{array}$
   2. Theorem
      $\begin{array}{rl}{\nabla }_{\theta }{J}_{\beta }\left({\mu }_{\theta }\right)& \approx {\int }_{\mathcal{S}}{\rho }^{\beta }(s){\nabla }_{\theta }{\mu }_{\theta }(a\mid s)Q^{\mu }(s,a)\mathrm{d}s\\ & ={\mathbb{E}}_{s\sim {\rho }^{\beta }}\left[{{\nabla }_{\theta }{\mu }_{\theta }(s){\nabla }_a{Q}^{\mu }(s,a)|}_{a={\mu }_{\theta }(s)}\right]\end{array}$
      1. 相比SPG的期望, DPG中没有important sampling ratio, 这是因为DPG中不涉及Action space的遍历, 因此也就不涉及如何将action space的遍历转化为期望以便对梯度进行采样估计的问题
      2. 此式给出的off-policy DPG actor的更新式, 但要实现off-policy control, critic也要能从off-policy data中学习
   3. Proof
      1. may be partly supported by Degris 2012 TODO

DPGT Derived AC Algorithms

1. on-policy AC
   1. actor update: DPGT
   2. critic update: SARSR TD learning
2. off-policy AC
   1. actor update: off-DPGT
   2. critic update: Q-learning TD learning

7. Discussion and Related Work

1. problems of SPG
   1. policy gradient ${\nabla }_{\theta }{\pi }_{\theta }(a\mid s)$ changes more rapidly near the mean
   2. The inner integral is computed by sampling a high dimensional action space.