【强化学习】SPG DPG DDPG（DPG3）

数学公式警告

Policy Gradient

$$\begin{array}{rl}J({\pi }_{\theta })=& {\int }_S{\rho }^{\pi }(s){\int }_A{\pi }_{\theta }(s,a)r(s,a)dads\\ =& E_{s\sim {\rho }^{\pi },a\sim {\pi }_{\theta }}[r(s,a)]\end{array}$$

${\rho }^{\pi }(s^{\prime })={\int }_S{\sum }_{t=1}^{\infty }{\gamma }^{t-1}{p}_1(s)p(s\to s^{\prime },t,\pi )ds$

其中$p_1(s)$表示初始状态为s的概率

$p(s->s^{\prime },t,\pi )$表示在策略$\pi$下状态s经过t时间到达s’

SPG

stochastic policy gradient

随机指随即策略${\pi }_{\theta }(a|s)=P[a|s,;\theta ]$，
$$\begin{array}{rl}{\nabla }_{\theta }J({\pi }_{\theta })=& {\int }_S{\rho }^{\pi }(s){\int }_A{\nabla }_{\theta }{\pi }_{\theta }(s,a)Q^{\pi }(s,a)dads\\ =& E_{s\sim {\rho }^{\pi },a\sim {\pi }_{\theta }}[{\nabla }_{\theta }log{\pi }_{\theta }(s,a)Q^{\pi }(s,a)]\end{array}$$

DPG

deterministic policy gradient

得出的Policy对于一个state的action是确定的
$$\begin{array}{rl}J({\mu }_{\theta })=& {\int }_S{\rho }^{\mu }(s)r(s,{\mu }_{\theta }(s))ds\\ =& E_{s\sim {\rho }^{\mu }}[r(s,{\mu }_{\theta }(s))]\end{array}$$

$$\begin{array}{rl}{\nabla }_{\theta }J({\mu }_{\theta })=& {\int }_S{\rho }^{\mu }(s){\nabla }_{\theta }{\mu }_{\theta }(s){\nabla }_a{Q}^{\mu }(s,a){|}_{a={\mu }_{\theta }(s)}ds\\ =& E_{s\sim {\rho }^{\mu }}[{\nabla }_{\theta }{\mu }_{\theta }(s){\nabla }_a{Q}^{\mu }(s,a){|}_{a={\mu }_{\theta }(s)}]\end{array}$$

DDPG