强化学习笔记_3_策略学习_Policy-Based Reinforcement Learning

1.Policy Function Approximation

• Policy Network $\pi (a|s;\theta )$

  使用$\pi (a|s;\theta )$对策略函数$\pi (a|s)$进行近似

  [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-b9eBdiip-1672473083781)(null)]

  使用Softmax可以满足${\sum }_{a\in \mathcal{A}}\pi (a|s;\theta )=1$

2.State-Value Function Approximation

Actioni-Value function: $Q_{\pi }(s_t,a_t)=E[U_t|S_t=s_t,A_t=a_t]$

State-Value function: $V_{\pi }(s_t)=E_A[Q_{\pi }(s_t,A)]$

• Policy-Based Reinforcement Learning
  $$V_{\pi }(s_t)=E_A[Q_{\pi }(s_t,A)]=\sum _a\pi (a|s_t)\cdot Q_{\pi }(s_t,a)$$
  将策略函数$\pi (a_t|s_t)$使用Policy Network进行近似后，状态价值函数可近似为
  $$V_{\pi }(s_t;\theta )=\sum _a\pi (a|s_t;\theta )\cdot Q_{\pi }(s_t,a)$$
  学习目标：改进$\theta$，使得$V_{\pi }(s;\theta )$更大，可将目标函数定义为
  $$maximizesJ(\theta )=E_S[V(S;\theta )]$$
  参数更新：Policy gradient ascent 策略梯度上升

  • 观测状态$s$
  • 更新参数$\theta$：$\theta \leftarrow \theta +\beta \cdot \frac{\partial V(s;\theta )}{\partial \theta }$

3.Policy Gradient

$$V(s;\theta )=\sum _a\pi (a|s;\theta )\cdot Q_{\pi }(s,a)$$

$$\begin{array}{rl}\frac{\partial V(s;\theta )}{\partial \theta }& =\frac{\partial {\sum }_a\pi (a|s;\theta )\cdot Q_{\pi }(s,a)}{\partial \theta }\\ & =\sum _a\frac{\partial \pi (a|s;\theta )}{\partial \theta }\cdot Q_{\pi }(s,a)\\ & =\sum _a\pi (a|s;\theta )\frac{\partial \log (\pi (a|s;\theta ))}{\partial \theta }\cdot Q_{\pi }(s,a)\\ & =E_A[\frac{\partial \log (\pi (a|s;\theta ))}{\partial \theta }\cdot Q_{\pi }(s,a)]\end{array}$$

（以上推导并不严谨，认为$Q_{\pi }(s,a)$与$\theta$是无关的，但由于$\pi$与$\theta$有关，所以假设实际上不成立。但考虑与否的推导结果相同。）

得到Policy Gradient的两种计算方法

• 方法1：
  $$\frac{\partial V(s;\theta )}{\partial \theta }=\sum _a\frac{\partial \pi (a|s;\theta )}{\partial \theta }\cdot Q_{\pi }(s,a)$$
  对于离散动作，对所有动作计算$f(a,\theta )=\frac{\partial \pi (a|s;\theta )}{\partial \theta }\cdot Q_{\pi }(s,a)$，然后累加

• 方法2：
  $$\frac{\partial V(s;\theta )}{\partial \theta }=E_A[\frac{\partial \log (\pi (a|s;\theta ))}{\partial \theta }\cdot Q_{\pi }(s,a)]$$
  可以用于连续或离散动作，通过积分的方法计算期望，但是由于$\pi$是通过神经网络计算的，无法直接计算积分，故通过蒙特卡洛近似的方法计算：

  • 根据当前预测策略$\pi (\cdot |s;\theta )$，在动作空间内随机采样，得到动作$\hat{a}$
  • 计算$g(\hat{a},\theta )=\frac{\partial \log (\pi (\hat{a}|s;\theta ))}{\partial \theta }\cdot Q_{\pi }(s,\hat{a})$
  • $g(\hat{a},\theta )$是对$\frac{\partial V(s;\theta )}{\partial \theta }$的无偏估计，将其作为策略梯度的近似值

4.Update policy network using policy gradient

• Observe the state $s_t$
• Randomly sample action $a_t$ according to $\pi (\cdot |s_t;{\theta }_t)$
• Computer $q_t\approx Q_{\pi }(s_t,a_t)$
• Differentiate policy network: $d_{\theta ,t}=\frac{\partial \log (\pi (a|s;\theta ))}{\partial \theta }{|}_{\theta ={\theta }_t}$
• (Approximate) policy gradient: $g(a_t,{\theta }_t)=q_t\cdot d_{\theta ,t}$
• Update policy network: $\theta \leftarrow \theta +\beta \cdot g(a_t,{\theta }_t)$

5.$q_t=Q_{\pi }(s_t,a_t)$的计算

• 方法1：REINFORCE

  完成一个完整过程，得到序列
  $$s_1,a_1,r_1,\cdot \cdot \cdot ,s_T,a_T,r_T$$
  计算Return $u_t={\sum }_{k=t}^T{\gamma }^{k-t}{r}_k$，由于$Q_{\pi }(s_t,a_t)=E[U_t]$，故可以使用$u_t$近似$Q_{\pi }(s_t,a_t)$，即
  $$q_t=u_t$$

• 方法2：使用神经网络计算，actor-critic method

故可以使用$u_t$近似$Q_{\pi }(s_t,a_t)$，即
$$q_t=u_t$$

• 方法2：使用神经网络计算，actor-critic method