ddpg/Continuous control with deep reinforcement learning

总结

融合dqn buffer+actor/critic分别2个网络，连续的action（更新时不是${\nabla }_{\theta }\pi (a|s)$了，而是${\nabla }_{\theta }\mu (s)$，$\mu$是actor网络）、连续的states

细节

之前的方法action都是离散的，这里可以解决连续action的问题

对action和critic分别建了2个网络，actor网络$\mu ,{\mu }^{\prime }$，critic网络$Q,Q^{\prime }$，更新步骤为：

1. 根据当前actor $\mu$及噪音${\mathcal{N}}_t$选动作：$a_t=\mu (s_t)+{\mathcal{N}}_t$
2. 执行动作$a_t$获得reward $r_t$以及新的state $s_{t+1}$
3. buffer
4. 从buffer中sample N个trajectory
5. 计算$y_i=r_i+\gamma Q^{\prime }(s_{i+1},{\mu }^{\prime }(s_{i+1}))$
6. 更新critic网络：$L=\frac{1}{N}{\sum }_i(y_i-Q(s_i,a_i){)}^2$
7. 更新actor网络：${\nabla }_{{\theta }_{\mu }}\approx \frac{1}{N}{\sum }_i{\nabla }_{a}Q(s,a){\nabla }_{{\theta }_{\mu }}\mu (s)$，作为对比，之前的policy gradient更新时是${\nabla }_{{\theta }_{\mu }}\approx \frac{1}{N}{\sum }_i{\nabla }_{a}Q(s,a){\nabla }_{\theta }\pi (a|s)$
8. 更新critic, actor的target网络：
   $$\begin{gathered} {\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^Q+(1-\tau ){\theta }^{Q^{\prime }} \\ {\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }} \end{gathered}$$