【RL】4.Policy Gradient

RL-Ch4-Policy Gradient

策略梯度(Policy Gradient)

强化学习的例子

Scene	Agent	Env	Reward Function
Video	游戏手柄	主机	杀1怪得20分
Go	AlphaGo	李世石	the Rule of Go

在上述例子中，策略(policy)$\pi$的具体表现形式可认为是神经网络从输入层到输出层之间的参数矩阵$\theta$。

下图为一个加入了action的马尔可夫链，

记Trajectory τ = { s 1 , a 1 , . . . , s T , a T } \tau=\{s_1,a_1,...,s_T,a_T\} τ={ s1​,a1​,...,sT​,aT​}，则

$$\begin{gathered} p_{\theta }(\tau )=p(s_1)p_{\theta }(a_1|s_1)p(s_2|s_1,a_1)p_{\theta }(a_2|s_2)p(s_3|s_2,a_2)... \\ =p(s_1)\prod _{t=1}^T{p}_{\theta }(a_t|s_t)p(s_{t+1}|s_t,a_t) \end{gathered}$$

同时决策需要考虑到收益，我们在上图中加入reward。
$$R(\tau )=\sum _{t=1}^T{r}_t$$

则期望收益为
$$\overline{R_{\theta }}=\sum _{\tau }R(\tau )p_{\theta }(\tau )={\mathbb{E}}_{\tau \sim p_{\theta }(\tau )}[R(\tau )]$$
求期望收益的梯度有
$$\begin{gathered} \nabla \overline{R_{\theta }}=\sum _{\tau }R(\tau )\nabla p_{\theta }(\tau )=\sum _{\tau }R(\tau )p_{\theta }(\tau )\frac{\nabla p_{\theta }(\tau )}{p_{\theta }(\tau )} \\ =\sum _{\tau }R(\tau )p_{\theta }(\tau )\nabla \log p_{\theta }(\tau ) \\ ={\mathbb{E}}_{\tau \sim p_{\theta }(\tau )}[R(\tau )\nabla \log p_{\theta }(\tau )] \end{gathered}$$

策略梯度的计算

有两种计算方法：

1. 蒙特卡洛采样，式(4)可以改写为如下式(5)

$$\nabla \overline{R_{\theta }}\approx \frac{1}{N}\sum _{n=1}^{N}R({\tau }^n)\nabla \log p_{\theta }({\tau }^n)\stackrel{\text{式(1)}}{=}\frac{1}{N}\sum _{n=1}^N\sum _{t=1}^{T_n}R({\tau }^n)\nabla \log p_{\theta }(a_t^n|s_t^n)$$

2. 时序差分更新，式(4)可以改写为如下式(6)

∇ R θ ˉ ≈ 1 N ∑ n = 1 N ∑ t = 1 T n Q n ( s t n , a t n ) ∇ log