【5分钟 Paper】Deterministic Policy Gradient Algorithms

• 论文题目：Deterministic Policy Gradient Algorithms

所解决的问题？

  stochastic policy的方法由于含有部分随机，所以效率不高，方差大，采用deterministic policy方法比stochastic policy的采样效率高，但是没有办法探索环境，因此只能采用off-policy的方法来进行了。

背景

  以往的action是一个动作分布${\pi }_{\theta }(a|s)$，作者所提出的是输出一个确定性的策略(deterministic policy) $a={\mu }_{\theta }(s)$。

   In the stochastic case，the policy gradient integrates over both state and action spaces, whereas in the deterministic case it only integrates over the state space.

• Stochastic Policy Gradient

   前人采用off-policy的随机策略方法， behaviour policy $\beta (a|s)\ne {\pi }_{\theta }(a|s)$：

$$\begin{array}{rl}{J}_{\beta }\left({\pi }_{\theta }\right)& ={\int }_{\mathcal{S}}{\rho }^{\beta }(s)V^{\pi }(s)\mathrm{d}s\\ & ={\int }_{\mathcal{S}}{\int }_{\mathcal{A}}{\rho }^{\beta }(s){\pi }_{\theta }(a|s)Q^{\pi }(s,a)\mathrm{d}a\mathrm{d}s\end{array}$$

   Differentiating the performance objective and applying an approximation gives the off-policy policy-gradient (Degris et al., 2012b)

$$\begin{array}{rl}{\nabla }_{\theta }{J}_{\beta }\left({\pi }_{\theta }\right)& \approx {\int }_{\mathcal{S}}{\int }_{\mathcal{A}}{\rho }^{\beta }(s){\nabla }_{\theta }{\pi }_{\theta }(a|s)Q^{\pi }(s,a)\mathrm{d}a\mathrm{d}s\\ & ={\mathbb{E}}_{s\sim {\rho }^{\beta },a\sim \beta }\left[\frac{{\pi }_{\theta }(a|s)}{{\beta }_{\theta }(a|s)}{\nabla }_{\theta }\log {\pi }_{\theta }(a|s)Q^{\pi }(s,a)\right]\end{array}$$

  This approximation drops a term that depends on the action-value gradient ${\nabla }_{\theta }{Q}^{\pi }(s,a)$; (Degris et al., 2012b)

   ${\mu }_{\theta }(s)$ 更新公式：

$${\theta }^{k+1}={\theta }^k+\alpha {\mathbb{E}}_{s\sim {\rho }^{{\mu }^k}}\left[{\nabla }_{\theta }{Q}^{{\mu }^k}\left(s,{\mu }_{\theta }(s)\right)\right]$$

  引入链导法则：

$${\theta }^{k+1}={\theta }^k+\alpha {\mathbb{E}}_{s\sim {\rho }^{{\mu }^k}}\left[{\nabla }_{\theta }{\mu }_{\theta }(s){\nabla }_a{Q}^{{\mu }^k}\left(s,a\right){|}_{a={\mu }_{\theta }(s)}\right]$$

所采用的方法？

• On-Policy Deterministic Actor-Critic

  如果环境有大量噪声帮助智能体做exploration的话，这个算法还是可以的，使用sarsa更新critic，使用 $Q^w(s,a)$ 近似true action-value $Q^{\mu }$：

$$\begin{array}{rl}{\delta }_t& =r_t+\gamma Q^w\left(s_{t+1},a_{t+1}\right)-Q^w\left(s_t,a_t\right)\\ w_{t+1}& =w_t+{\alpha }_w{\delta }_t{\nabla }_w{Q}^w\left(s_t,a_t\right)\\ {\theta }_{t+1}& ={\theta }_t+{{\alpha }_{\theta }{\nabla }_{\theta }{\mu }_{\theta }\left(s_t\right){\nabla }_a{Q}^w\left(s_t,a_t\right)|}_{a={\mu }_{\theta }(s)}\end{array}$$

• Off-Policy Deterministic Actor-Critic

  we modify the performance objective to be the value function of the target policy, averaged over the state distribution of the behaviour policy

$$\begin{array}{rl}{J}_{\beta }\left({\mu }_{\theta }\right)& ={\int }_{\mathcal{S}}{\rho }^{\beta }(s)V^{\mu }(s)\mathrm{d}s\\ & ={\int }_{\mathcal{S}}{\rho }^{\beta }(s)Q^{\mu }\left(s,{\mu }_{\theta }(s)\right)\mathrm{d}s\end{array}$$

$$\begin{array}{rl}{\nabla }_{\theta }{J}_{\beta }\left({\mu }_{\theta }\right)& \approx {\int }_{\mathcal{S}}{\rho }^{\beta }(s){\nabla }_{\theta }{\mu }_{\theta }(a|s)Q^{\mu }(s,a)\mathrm{d}s\\ & ={\mathbb{E}}_{s\sim {\rho }^{\beta }}[{\nabla }_{\theta }{\mu }_{\theta }(s){\nabla }_a{Q}^{\mu }(s,a){|}_{a={\mu }_{\theta }(s)}]\end{array}$$

  得到off-policy deterministic actorcritic (OPDAC) 算法：

$$\begin{array}{rl}{\delta }_t& =r_t+\gamma Q^w\left(s_{t+1},{\mu }_{\theta }\left(s_{t+1}\right)\right)-Q^w\left(s_t,a_t\right)\\ w_{t+1}& =w_t+{\alpha }_w{\delta }_t{\nabla }_w{Q}^w\left(s_t,a_t\right)\\ {\theta }_{t+1}& ={\theta }_t+{{\alpha }_{\theta }{\nabla }_{\theta }{\mu }_{\theta }\left(s_t\right){\nabla }_a{Q}^w\left(s_t,a_t\right)|}_{a={\mu }_{\theta }(s)}\end{array}$$

  与stochastic off policy算法不同的是由于这里是deterministic policy，所以不需要用重要性采样(importance sampling)。

取得的效果？

所出版信息？作者信息？

  这篇文章是ICML2014上面的一篇文章。第一作者David Silver是Google DeepMind的research Scientist，本科和研究生就读于剑桥大学，博士于加拿大阿尔伯特大学就读，2013年加入DeepMind公司，AlphaGo创始人之一，项目领导者。

参考链接

• 参考文献：Degris, T., White, M., and Sutton, R. S. (2012b). Linear off-policy actor-critic. In 29th International Conference on Machine Learning.

扩展阅读

  假定真实的action-value function为 $Q^{\pi }(s,a)$，用一个function近似它 $Q^w(s,a)\approx Q^{\pi }(s,a)$。However, if the function approximator is compatible such that 1. $Q^w(s,a)={\nabla }_{\theta }\log {\pi }_{\theta }(a|s{)}^{\top }w$ (linear in “fearure”) 2. the parameters $w$ are chosen to minimise the mean-squared error ${\epsilon }^2(w)={\mathbb{E}}_{s\sim {\rho }^{\pi },a\sim {\pi }_{\theta }}[(Q^w(s,a)-Q^{\pi }(s,a){)}^2]$ (linear regression problem form these feature )，then there is no bias (Sutton et al., 1999),

$${\nabla }_{\theta }J\left({\pi }_{\theta }\right)={\mathbb{E}}_{s\sim {\rho }^{\pi },a\sim {\pi }_{\theta }}\left[{\nabla }_{\theta }\log {\pi }_{\theta }(a|s)Q^w(s,a)\right]$$

  最后，论文给出了DPG的采用线性函数逼近定理，以及一些理论证明基础。

• 参考文献：Sutton, R.S., McAllester D. A., Singh, S. P., and Mansour, Y. (1999). Policy gradient methods for reinforcement learning with function approximation. In Neural Information Processing Systems 12, pages 1057–1063.

  这篇文章以后有时间再读一遍吧，里面还是有些证明需要仔细推敲一下。

我的微信公众号名称：小小何先生
公众号介绍：主要研究分享深度学习、机器博弈、强化学习等相关内容！期待您的关注，欢迎一起学习交流进步！