强化学习笔记_8_连续控制

1. 离散控制与连续控制 Discrete VS Continuous Control

• Discrete Action Space

• Continuous Action Space

• DQN算法、Policy Network等可以解决离散控制问题，输出为一个确定维度的向量

• Discretization，离散化，将动作空间变为有限的离散空间；适用于维度比较小的问题。

  设控制问题的自由度为$d$，则动作空间为$d$维的，离散化时，离散空间内点的数量随着$d$指数增加，导致维数灾难、训练困难。

• 其他方法：Deterministic policy network; Stochastic policy network.

2. Deterministic Policy Gradient (DPG, 确定策略梯度)

2.1. Deterministic Actor-Critic

• deterministic policy network (actor): $a=\pi (s;\theta )$，输出不是一个概率，而是一个具体的动作$a$，输出维度为动作空间的维数；
• value network (critic): $q(s,a;w)$；

2.2. Updating Value Network by TD

• Transition: $(s_t,a_t,r_t,s_{t+1})$

• Value network:
  $$\begin{array}{rl}{q}_t& =q(s,a;w)\\ q_{t+1}& =q(s_{t+1},a_{t+1};w),where{a}_{t+1}^{\prime }=\pi (s_{t+1};\theta )\end{array}$$

• TD error: ${\delta }_t=q_t-(r_t+\gamma \cdot q_{t+1})$

  TD target: $y_t=r_t+\gamma \cdot q_{t+1}$

• Update: $w\leftarrow w-\alpha \cdot {\delta }_t\cdot \frac{\partial q(s_t,a_t;w)}{\partial w}$

2.3. Updating Policy Network by DPG

Goal: Increasing $q(s,a;w)$, where $a=\pi (s;\theta )$

DPG:
$$g=\frac{\partial q(s,\pi (s;\theta );w)}{\partial \theta }=\frac{\partial a}{\partial \theta }\cdot \frac{\partial q(s,a;w)}{\partial a}g=\frac{\partial q(s,\pi (s;\theta );w)}{\partial \theta }=\frac{\partial a}{\partial \theta }\cdot \frac{\partial q(s,a;w)}{\partial a}$$
Gradient ascent: $\theta \leftarrow \theta +\beta \cdot g$

2.4. Improvement: Using Target Network

• Bootstrapping

  TD error ${\delta }_t=q_t-(r_t+\gamma \cdot q_{t+1})$导致Bootstrapping，如果初始产生了高估（低估），则会导致后续的高估（低谷）。

  解决方案：使用不同的网络计算TD target——target networks

• target networks

  • Value network: $q_t=q(s_t,a_t;w)$

  • Value network: $q_{t+1}=q(s_{t+1},a_{t+1}^{\prime };w^{-}),where{a}_{t+1}^{\prime }=\pi (s_{t+1};{\theta }^{-})$

    Target value network: $q(s_t,a_t^{\prime };w^{-})$

    Target policy network: $\pi (s_t;{\theta }^{-})$

• 用到的算法概括如下：

• Updating target network

  hyper-parameter $\tau \in (0,1)$，使用加权平均 (weighted averaging) 更新参数：
  $$\begin{array}{rl}{w}^{-}& \leftarrow \tau \cdot w+(1-\tau )\cdot w^{-}\\ {\theta }^{-}& \leftarrow \tau \cdot \theta +(1-\tau )\cdot {\theta }^{-}\end{array}$$
  target network中的参数依然与原网络相关，故无法完全解决bootstrapping

2.5. Improvements

• Target network
• Experience relay
• Multi-step TD target

2.6. Stochastic Policy VS Deterministic Policy

3. Stochastic Policy for Continuous Control (离散策略)

3.1. Policy Network

• Univariate Normal Distribution (单变量正态分布)

  考虑单自由度情况，自由度$d=1$，均值(mean) $\mu$和标准差(std) $\sigma$是状态$s$的函数；

  使用正态分布的概率密度函数作为策略函数：
  $$\pi (a|s)=\frac{1}{\sqrt{2\pi }\sigma }\cdot \exp (-\frac{(a-\mu {)}^2}{2{\sigma }^2})$$

• Multivariate Normal Distribution (多变量正态分布)

  自由度为$d$，动作空间action $a$为$d$维，均值和标准差分别为$\mu ,\sigma :\mathcal{S}\to {\mathbb{R}}^d$，输入为状态$s$，输出为$d$维向量。

  使用${\mu }_i,{\sigma }_i$表示$\mu (s),\sigma (s)$的第$i$个分量。假设动作空间内各个维度都是独立的，则PDF：
  $$\pi (a|s)={\Pi }_{i=1}^d\frac{1}{\sqrt{2\pi }{\sigma }_i}\cdot \exp (-\frac{(a_i-{\mu }_i{)}^2}{2{\sigma }_i^2})$$

• Function Approximation

  • 使用神经网络$\mu (s;{\theta }^{\mu })$对均值$\mu (s)$进行近似；

  • 使用神经网络$\sigma (s;{\theta }^{\sigma })$对均值$\sigma (s)$进行近似（效果不好）；

  • 对方差的对数进行近似，使用神经网络$\rho (s;{\theta }^{\rho })$对$\rho$近似
    $${\rho }_i=\ln {\sigma }_i^2,i=1,\cdot \cdot \cdot ,d$$

  

• Continuous Control

  • 观测得到当前状态$s_t$；

  • 计算均值和方差$\hat{\mu }=\mu (s;{\theta }^{\mu })$，方差$\hat{\rho }=\mu (s;{\theta }^{\rho }),{\hat{\sigma }}_i^2=\exp ({\hat{\rho }}_i)$；

  • 随机采样得到动作
    $$a\sim \mathcal{N}(\hat{\mu },{\hat{\sigma }}_i),i=1,\cdot \cdot \cdot ,d$$

• Training Policy Network

  • Auxiliary network，辅助神经网络

  • Policy gradient methods

    • option 1: REINFORCE
    • option 2: Actor-Critic

3.2. Training (1/2): Auxiliary Network

• Stochastic policy gradient:
  $$g(a)=\frac{\partial \ln \pi (a|s;\theta )}{\partial \theta }\cdot Q_{\pi }(s,a)$$

• Policy network:
  $$\pi (a|s;{\theta }^{\mu },{\theta }^{\rho })={\Pi }_{i=1}^d\frac{1}{\sqrt{2\pi }{\sigma }_i}\cdot \exp (-\frac{(a_i-{\mu }_i{)}^2}{2{\sigma }_i^2})$$
  Log of policy network:
  $$\begin{array}{rl}\ln \pi (a|s;{\theta }^{\mu },{\theta }^{\rho })& =\sum _{i=1}^d[-\ln {\sigma }_i-\frac{(a_i-{\mu }_i{)}^2}{2{\sigma }_i^2}]+const\\ & =\sum _{i=1}^d[-\frac{{\rho }_i}{2}-\frac{(a_i-{\mu }_i{)}^2}{2\cdot \exp ({\rho }_i)}]+const\end{array}$$

• Auxiliary Network:
  $$f(s,a;\theta )=\sum _{i=1}^d[-\frac{{\rho }_i}{2}-\frac{(a_i-{\mu }_i{)}^2}{2\cdot \exp ({\rho }_i)}],\theta =({\theta }^{\mu },{\theta }^{\rho })$$
  

  

3.2. Training (2/2): Policy gradient methods

• Stochastic policy gradient:
  $$\begin{array}{rl}f(s,a;\theta )& =\ln \pi (a|s;\theta )+const\\ g(a)& =\frac{\partial \ln \pi (a|s;\theta )}{\partial \theta }\cdot Q_{\pi }(s,a)\end{array}$$
  得到：
  $$g(a)=\frac{\partial f(s,a;\theta )}{\partial \theta }\cdot Q_{\pi }(s,a)$$
  接下来对$Q_{\pi }(s,a)$进行近似。

• option 1: REINFORCE

  蒙特卡洛近似，使用观测值$u_t$进行近似$Q_{\pi }(s,a)$，参数更新：
  $$\theta \leftarrow \theta +\beta \cdot \frac{\partial f(s,a;\theta )}{\partial \theta }\cdot u_t$$

• option 2: Actor-Critic

  使用价值网络$q(s,a;w)$进行近似$Q_{\pi }(s,a)$，参数更新：
  $$\theta \leftarrow \theta +\beta \cdot \frac{\partial f(s,a;\theta )}{\partial \theta }\cdot q(s,a;w)$$
  使用TD learning训练价值网络$q(s,a;w)$

3.3. Improvement: Policy gradient with baseline

• Reinforce: Reinforce with baseline.
• Actor-Critic: Advantage Actor-Critic (A2C).