深度强化学习CS285 lec13-lec15 （中）

概述

• 目标理解：

  • 给定Objective，找出Policy；相似说法为给定Intent，找出一个Optimal behavior
  • 给定Stochastic Optimal Behavior，推断其Intent；相似说法为给定expert demonstrations，推断其Objective

• Intuition：

  • Soft Optimality也称为Stochastic Optimal Control ，是对Stochastic Optimal Behavior进行建模，引入二值Optimality Variable
  • 一些mistakes更重要，但没必要一直Optimal，在Intent可Reach的情况下，允许做一些Sub-Optimal的事，去explore一下
  • 使用PGM for decision making会解释一下，为什么good behavior确实会好过bad behavior以及同为good behavior的时候，有同样的概率reach goal。（slightly random slightly different）

• 问题描述：

  • 如何在模型中加入Intent？（Optimality Variable）
  • $O_t=1$ 是trying to be optimal的变量，表达了一种intent的意图而不是实际的最优，只是希望最优的意图intent，$O_t=0$可以理解成做随机策略的意思，所以有$p(\tau |O_{1:T})$表示Generative Model Of Optimal Behavior即每时每刻都维持这个最优的intent，会产生什么样的轨迹。
  • 选择$p(O_t|s_t,a_t)=exp(r(s_t,a_t))$，这个intent应该是跟reward有关的，reward大的地方intent就更强烈～
  • 这个Stochastic Behavior model，可以对Sub-Optimal的动作建模，但并不具备推理的能力

一、概率图基础知识

在概率图PGM中，Inference有如下几类，已知n维观测对象 { x 1 , . . . , x n } \{x_1,...,x_n\} {x1​,...,xn​}，k维状态对象 { z 1 , . . . , z k } \{z_1,...,z_k\} {z1​,...,zk​}，引入时间维度后，有 X = { X 1 , . . , X T } , { Z 1 , . . . , Z T } X=\{X_1,..,X_T\},\{Z_1,...,Z_T\} X={X1​,..,XT​},{Z1​,...,ZT​}.

• Learning：学习模型的参数$\lambda$
  $${\lambda }^{\ast }=\underset{\lambda }{\mathrm{argmax}}P(X|\lambda )$$
• Inference：已知模型的参数$\lambda$，去Inference一些变量
  • Decoding：给定观测序列$X$，预测隐状态序列$Z$
    $$\begin{gathered} Z=\underset{Z}{\mathrm{argmax}}{P}_{\lambda }(Z|X) \\ {Z}_1,..,Z_t=\underset{Z}{\mathrm{argmax}}{P}_{\lambda }(Z_1,...,Z_t|X_1,...,X_t) \end{gathered}$$
  • Evidence：出现观测序列的概率
    $$p_{\lambda }(X)=p_{\lambda }(X_1,...,X_T)$$
  • Filtering ：给定当前时刻前的历史信息，预测当前时刻的隐状态
    $$P(Z_t|X_1,...,X_t)$$
  • Smoothing：给定所有信息，复盘一下某时刻的隐状态
    $$P(Z_t|X_1,...,X_T)$$
  • Prediction：给定当前时刻前的历史信息，预测下一时刻的观测变量或者隐状态
    $$\begin{gathered} P(Z_{t+1}|X_1,X_2,...,X_t) \\ P(X_{t+1}|X_1,X_2,...,X_t) \end{gathered}$$

二、Soft Optimality Framework

2.1 Recap

在lec13-lec15 (上)中有

• Soft Optimality

1. Forward
   $${\alpha }_t(s_t)=p(s_t|O_{1:t-1})$$
2. Backward
   $${\beta }_t(s_t,a_t)=p(O_{t:T}|s_t,a_t)$$
3. Policy
   $$\begin{gathered} \pi (a_t|s_t)=p(a_t|s_t,O_{1:T})=\frac{{\beta }_t(s_t,a_t)}{{\beta }_t(s_t)} \\ p(s_t|O_{1:T})\propto {\beta }_t(s_t){\alpha }_t(s_t) \end{gathered}$$
4. Optimality Variable$$p(O_t=1|s_t,a_t)=exp(r(s_t,a_t))$$

解释一下：

• ${\beta }_t(s_t,a_t)=p(O_{t:T}|s_t,a_t)$：如果在该state和action下，我需要一直optimal才能维持intent可reach的概率
• $p(a_t|s_t,O_{1:T})$：给定我一直要维持intent即$O_{1:T}=1$，在某个state下，可以做哪些action的概率
• ${\alpha }_t(s_t)=p(s_t|O_{1:t-1})$：如果我之前一直在optimal，那我会reach到哪个state的概率
• $p(O_t=1|s_t,a_t)\propto exp(r(s_t,a_t))$：这个Optimality Variable是对Intent进行建模，在当前$s_t,a_t$条件下，是否需要维持Optimal这个Intent正比于$r(s_t,a_t)$
• $p(s_t|O_{1:T})\propto {\beta }_t(s_t){\alpha }_t(s_t)$：给定一直维持Optimal Intent的情况下，到达某时刻state的概率正比于$p(s_t|O_{1:t-1})p(O_{t:T}|s_t)$

2.2 Soft Optimality与Value Iteration的联系

1. Backward Process：
   $$\begin{gathered} {\beta }_t(s_t,a_t)=p(O_t|s_t,a_t)E_{s_{t+1}\sim p(s_{t+1}|s_t,a_t)}[{\beta }_{t+1}(s_{t+1})] \\ {\beta }_t(s_t)=E_{a_t\sim p(a_t|s_t)}[{\beta }_t(s_t,a_t)] \end{gathered}$$

2. Soft Optimality的Backward Process通过变换会与Model-free中熟悉的Value Iteration建立联系（Soft Value Iteration）
   $$Let{\beta }_t(s_t)=exp(V_t(s_t)),Let{\beta }_t(s_t,a_t)=exp(Q_t(s_t,a_t))$$
   $$\begin{array}{rl}& 1.Q_t(s_t,a_t)=r(s_t,a_t)+log{E}_{s_{t+1}\sim p(s_{t+1}|s_t,a_t)}[exp(V_t(s_{t+1}))]\\ & 2.V_t(s_t)=log{E}_{a_t\sim p(a_t|s_t)}\left[\underset{actionprior}{\underbrace{exp(Q_t(s_t,a_t)+logp(a_t|s_t))}}\right]\end{array}$$

   按照如下policy的公式做出决策：
   $$\pi (a_t|s_t)=\frac{{\beta }_t(s_t,a_t)}{{\beta }_t(s_t)}=\frac{exp(Q_t(s_t,a_t))}{exp(V_t(s_t))}=exp(Q_t(s_t,a_t)-V_t(s_t))=exp(A_t(s_t,a_t))$$

3. 传统的Value Iteration

   1. $Q(s,a)=r(s,a)+E_{s^{\prime }\sim p(s^{\prime }|s,a)}[V(s^{\prime })]$
   2. $V(s)={\max }_{a}Q(s,a)$
      
      按照如下policy的公式做出决策：
      $\pi (a_t|s_t)=\delta (a=\underset{a_t}{\mathrm{argmax}}Q(s_t,a_t))，\delta$可为greedy policy的一个函数。

2.3 Optimism Problem(关键哦！）

2.3.1 Optimism问题的引出

action prior一般假设是均匀的，于是Soft Value Iteration变成如下：
$$\begin{array}{rl}& 1.Q_t(s_t,a_t)=r(s_t,a_t)+log{E}_{s_{t+1}\sim p(s_{t+1}|s_t,a_t)}[exp(V_t(s_{t+1}))]\\ & 2.V_t(s_t)=log\int \left[exp\right(Q_t(s_t,a_t)\left)\right]d{a}_t\end{array}$$

对于第二步，$V_t(s_t)=log\int \left[exp\right(Q_t(s_t,a_t)\left)\right]d{a}_t$，当某个$a_t$的值较大时，根据exp平滑的特性，有$log\int \left[exp\right(Q_t(s_t,a_t)\left)\right]d{a}_t\to ma{x}_{a}Q(s_t,a_t)$，与传统VI第二步差不多

对于第一步，$Q_t(s_t,a_t)=r(s_t,a_t)+log{E}_{s_{t+1}\sim p(s_{t+1}|s_t,a_t)}[exp(V_t(s_{t+1}))]$当下一状态的value大时，根据exp平滑拉大差距的特性，就与传统VI第一步相差远了$Q(s,a)=r(s,a)+E_{s^{\prime }\sim p(s^{\prime }|s,a)}[V(s^{\prime })]$。

==这样会出现一个Optimism的问题，当下一状态是deterministic的时候无所谓，毕竟下一状态是确定的，有

$$Q_t(s_t,a_t)=r(s_t,a_t)+log{E}_{s_{t+1}\sim p(s_{t+1}|s_t,a_t)}[exp(V_t(s_{t+1}))]=r(s_t,a_t)+V_{t+1}(s_{t+1})$$

但当下一状态是stochastic的时候会出现Optimism问题，即若$E[V(s^{\prime })]$的expected Value是中间的值，但下一状态Value的分布是一高一低，则$E[exp(V(s^{\prime }))]$则会向高的值，导致risk-seeking behavior，这个就是Optimism的问题，那为什么会这样呢？

2.3.2 Soft与Standard RL的区别

因为这个Soft Framework的出发角度与之前standard RL是不一样的！

在Standard RL中，是给定一个Objective：
$$\begin{array}{rl}{\theta }^{\ast }& =\underset{\theta }{\mathrm{argmax}}{E}_{\tau \sim p_{\theta }(\tau )}[r(\tau )]\\ & =\underset{\pi }{\mathrm{argmax}}{E}_{s_t\sim p(s_t|s_{t-1},a_{t-1}),a_t\sim {\pi }_{\theta }(a_t|s_t)}[\sum _{t}r(s_t,a_t)]\\ p(\tau )& =p(s_1)\prod _{t=1}^{T}p(s_{t+1}|s_t,a_t){\pi }_{\theta }(a_t|s_t)\end{array}$$

在dynamics环境$p(s^{\prime }|s,a)$与我们的目标policy即$\pi (a_t|s_t)$的共同作用下，产生一些trajectory即$\tau$，根据最大化trajectory expected reward的目标约束下调整policy，使其最优

但是在Soft Framework里的出发点完全不一样，一没有Objective，二是从概率图角度构建问题的，因此对于一个带有optimal这种intent的trajectory的产生有：
$$\begin{array}{rl}p(\tau |O_{1:T})& =p(s_{1:T},a_{1:T}|O_{1:T})\\ & \propto p(s_{1:T},a_{1:T},O_{1:T})\\ & =p(s_1)\prod _{t=1}^{T}p(s_{t+1}|s_t,a_t)p(O_t|s_t,a_t)\\ & =[p(s_1)\prod _{t=1}^{T}p(s_{t+1}|s_t,a_t)\left]exp\right(\sum _{t=1}^{T}r(s_t,a_t))\end{array}$$

通过归一化之类的方式，有
$$p(\tau |O_{1:T})=[\underset{FeasibleTrajectory}{\underbrace{p(s_1)\prod _{t=1}^{T}p(s_{t+1}|s_t,a_t)]}}exp(\sum _{t=1}^{T}r(s_t,a_t))$$

最优轨迹的分布除了不可控的环境dynamics以外，就剩下由reward控制了。（如果有expert demonstration的话，实际上$p(\tau |O_{1:T})$就是专家数据的建模表示）

而实际上我们要的是一个policy即$p(a_t|s_t,O_{1:T})$，于是希望通过Learning使policy产生的trajectory distribution与上述引入Optimal Variable的专家轨迹分布接近，即
$$\hat{p}(\tau )=p(s_1|O_{1:T})\prod _{t=1}^{T}p(s_{t+1}|s_t,a_t,O_{1:T})p(a_t|s_t,O_{1:T})$$

2.3.3 Deterministic dynamics的Objective

在Deterministic dynamics的情况下，做一些简化

1. 因为只有一个transition的状态，所以有$p(s_{t+1}|s_t,a_t,O_{1:T})=p(s_{t+1}|s_t,a_t)$
2. 初始状态可人为设定，不太重要即可令$p(s_1|O_{1:T})=p(s_1)$
3. 我们想学习的策略${\pi }_{\theta }(a_t|s_t)=p(a_t|s_t,O_{1:T})$

所以目前有Soft Optimality概率图reward最优的轨迹分布：
$$p(\tau |O_{1:T})=[\underset{FeasibleTrajectory}{\underbrace{p(s_1)\prod _{t=1}^{T}p(s_{t+1}|s_t,a_t)]}}exp(\sum _{t=1}^{T}r(s_t,a_t))$$

通过我们的目标policy构造的轨迹分布：
$$\begin{array}{rl}\hat{p}(\tau |O_{1:T})& =p(s_1|O_{1:T})\prod _{t=1}^{T}p(s_{t+1}|s_t,a_t,O_{1:T})p(a_t|s_t,O_{1:T})\\ & =p(s_1)\prod _{t=1}^{T}p(s_{t+1}|s_t,a_t){\pi }_{\theta }(a_t|s_t)\end{array}$$

我们拉近这两个分布
$$\begin{array}{rl}\underset{\theta }{\min }{D}_{KL}(\hat{p}(\tau |O_{1:T})||p(\tau |O_{1:T}))& =\underset{\theta }{\min }{E}_{\tau \sim \hat{p}(\tau )}[log\hat{p}(\tau )-logp(\tau )]\\ & =\underset{\theta }{\max }{E}_{\tau \sim \hat{p}(\tau )}[logp(\tau )-log\hat{p}(\tau )]\\ & =\underset{\theta }{\max }{E}_{\tau \sim \hat{p}(\tau )}[logp(s_1)\prod _{t=1}^{T}p(s_{t+1}|s_t,a_t)exp(\sum _{t=1}^{T}r(s_t,a_t))-logp(s_1)\prod _{t=1}^{T}p(s_{t+1}|s_t,a_t)]{\pi }_{\theta }(a_t|s_t)]\\ & =\underset{\theta }{\max }{E}_{\tau \sim \hat{p}(\tau )}[\sum _{t=1}^T[r(s_t,a_t)-log{\pi }_{\theta }(a_t|s_t)]]\\ & =\underset{\theta }{\max }\sum _{t=1}^T{E}_{(s_t,a_t)\sim \hat{p}(s_t,a_t)}[r(s_t,a_t)]+\sum _{t=1}^T{E}_{s_t\sim \hat{p}(s_t),a_t\sim {\pi }_{\theta }(a_t|s_t)}[-log{\pi }_{\theta }(a_t|s_t)]\\ & =\underset{\theta }{\max }\sum _{t=1}^T{E}_{(s_t,a_t)\sim \hat{p}(s_t,a_t)}[r(s_t,a_t)]+E_{s_t\sim \hat{p}(s_t)}[H[{\pi }_{\theta }(a_t|s_t)]]\end{array}$$

于是发现通过概率图对stochastic optimal behavior 建模后，在deterministic dynamics的情况下，希望我们想要的policy与专家数据中的policy接近$min{D}_{KL}$，其相当于优化目标Maximum Entropy Objective！

2.3.4 Stochastic Dynamics的Objective

当环境是Stochastic的情况下，
$$\hat{p}(\tau |O_{1:T})=p(s_1|O_{1:T})\prod _{t=1}^{T}p(s_{t+1}|s_t,a_t,O_{1:T})p(a_t|s_t,O_{1:T})$$

$p(s_{t+1}|s_t,a_t,O_{1:T})\ne p(s_{t+1}|s_t,a_t)$，因为不只有一个transition状态了，但初始化状态还是可以设成一样的即$p(s_1|O_{1:T})=p(s_1)$。这说明，在Soft Optimality Framework的情况下，Stochastic的时候Agent具备操纵dynamics的能力，我们想Agent只能操纵Policy才对，那怎么办呢？
Variational Inference进行分布近似

构建一个新的轨迹分布$q(\tau )=q(s_{1:T},a_{1:T})$来近似$\hat{p}(\tau )=p(s_{1:T},a_{1:T}|O_{1:T})$，这个过程就如基础知识中提到的变分推断过程。

令$x=O_{1:T},z=(s_{1:T},a_{1:T}),q(z)=q(\tau )=p(s_1){\prod }_{t}p(s_{t+1}|s_t,a_t)q(a_t|s_t)$，由Variational Inference有：
$$\begin{array}{rl}logp(O_{1:T})=logp(x)& \ge E_{z\sim q(z)}[logp(x,z)-logq(z)]\\ & =E_{(s_{1:T},a_{1:T})\sim q(s_{1:T},a_{1:T})}[logp(s_{1:T},a_{1:T},O_{1:T})-logq(s_{1:T},a_{1:T})]\\ & =E_{(s_{1:T},a_{1:T})\sim q(s_{1:T},a_{1:T})}[logp(s_1)\prod _{t}p(s_{t+1}|s_t,a_t)p(O_t|s_t,a_t)-logp(s_1)\prod _{t}p(s_{t+1}|s_t,a_t)q(a_t|s_t)]\\ & =E_{(s_{1:T},a_{1:T})\sim q(s_{1:T},a_{1:T})}[\sum _{t}r(s_t,a_t)-logq(a_t|s_t)]\\ & =\sum _t{E}_{(s_t,a_t)\sim q(s_t,a_t)}[r(s_t,a_t)]+E_{s_t\sim q(s_t)}[H[q(a_t|s_t)]]\end{array}$$

最大化这个ELBO，从末尾时刻T开始：
$$\begin{array}{rl}q\left({\mathbf{a}}_T|{\mathbf{s}}_T\right)& =\arg \max E_{{\mathbf{s}}_T\sim q\left({\mathbf{s}}_T\right)}\left[E_{{\mathbf{a}}_T\sim q\left({\mathbf{a}}_T|{\mathbf{s}}_T\right)}\left[r\left({\mathbf{s}}_T,{\mathbf{a}}_T\right)\right]+\mathcal{H}\left(q\left({\mathbf{a}}_T|{\mathbf{s}}_T\right)\right)\right]\\ & =\arg \max E_{{\mathbf{s}}_T\sim q\left({\mathbf{s}}_T\right)}\left[E_{{\mathbf{a}}_T\sim q\left({\mathbf{a}}_T|{\mathbf{s}}_T\right)}\left[r\left({\mathbf{s}}_T,{\mathbf{a}}_T\right)-\log q\left({\mathbf{a}}_T|{\mathbf{s}}_T\right)\right]\right]\end{array}$$

如何选择这个$a_T$呢？末尾时刻T肯定跟着reward的大小选择呀！
$$q(a_T|s_T)=\frac{exp(r(s_T,a_T))}{\int exp(r(s_T,a_T))da}=exp(Q(s_T,a_T)-V(s_T))$$

最后时刻有$r(s_T,a_T)=Q(s_T,a_T)$，且
$$logq(a_T|s_T)=Q(s_T,a_T)-V(s_T)$$

所以：

$$q(a_T|s_T)=E_{s_T\sim q(s_T)}[E_{a_T\sim q(a_T|s_T)}[V(s_T)]]$$

我们解了个啥？选择policy即$q(a_t|s_t)$来最大化ELBO：
$$logp(O_{1:T})\ge \sum _{t=1}^T{E}_{(s_t,a_t)\sim q(s_t,a_t)}[r(s_t,a_t)]+E_{s_t\sim q(s_t)}[H[q(a_t|s_t)]]$$

选了末尾时刻T开始考虑，得出一项：$$q(a_T|s_T)=E_{s_T\sim q(s_T)}[E_{a_T\sim q(a_T|s_T)}[V(s_T)]]$$

于是普通的情况$q(a_t|s_t)$有：

$$\begin{array}{rl}q\left({\mathbf{a}}_t|{\mathbf{s}}_t\right)& =\arg \max E_{{\mathbf{s}}_t\sim q\left({\mathbf{s}}_t\right)}\left[E_{{\mathbf{a}}_t\sim q\left({\mathbf{a}}_t|{\mathbf{s}}_t\right)}\left[r\left({\mathbf{s}}_t,{\mathbf{a}}_t\right)+E_{{\mathbf{s}}_{t+1}\sim p\left({\mathbf{s}}_{t+1}|{\mathbf{s}}_t,{\mathbf{a}}_t\right)}\left[V\left({\mathbf{s}}_{t+1}\right)\right]\right]+\mathcal{H}\left(q\left({\mathbf{a}}_t|{\mathbf{s}}_t\right)\right)\right]\\ & =\arg \max E_{{\mathbf{s}}_t\sim q\left({\mathbf{s}}_t\right)}\left[E_{{\mathbf{a}}_t\sim q\left({\mathbf{a}}_t|{\mathbf{s}}_t\right)}\left[Q\left({\mathbf{s}}_t,{\mathbf{a}}_t\right)\right]+\mathcal{H}\left(q\left({\mathbf{a}}_t|{\mathbf{s}}_t\right)\right)\right]\\ & =\arg \max E_{{\mathbf{s}}_t\sim q\left({\mathbf{s}}_t\right)}\left[E_{{\mathbf{a}}_t\sim q\left({\mathbf{a}}_t|{\mathbf{s}}_t\right)}\left[Q\left({\mathbf{s}}_t,{\mathbf{a}}_t\right)-\log q\left({\mathbf{a}}_t|{\mathbf{s}}_t\right)\right]\right]\end{array}$$

最终有，

$$\begin{gathered} Q_t(s_t,a_t)=r(s_t,a_t)+E[V_{t+1}(s_{t+1})] \\ {V}_t(s_t)=log\int exp(Q_t(s_t,a_t))d{a}_t \\ q(a_t|s_t)=exp(Q(s_t,a_t)-V(s_t)) \end{gathered}$$

2.4 Soft Value Iteration

经过Soft Optimality在Deterministic与Stochastic的dynamics下，才知道原来我们在优化的Objective不过是加了个Entropy term，因为PGM建模方式有个Optimism Problem的问题，于是通过Variational Inference解决Stochastic Dynamics后终于发现了与Value Iteration关键的联系！！！

为大家呈现Soft Value Iteration：

它们之间的对比：

2.5 Soft Q-leanring

知道了Softmax的含义后，即
$$softma{x}_{a}Q(s,a)=log\int exp(V(s))da$$

将原来的Q-learning中Q值换一个Softemax就变成Soft Q-leanring了！

只需要注意一下policy的获取$\pi (a|s)=exp(Q(s,a)-V(s))=exp(A(s,a))$，因为只有这种形式的policy才能Optimize刚才辛苦推断出来的Objective，具体的参数更新为：
$$\begin{gathered} \varphi \leftarrow \varphi +\alpha {\nabla }_{\varphi }{Q}_{\varphi }(s,a)(r(s,a)+\gamma V(s^{\prime })-Q_{\varphi }(s,a)) \\ V(s^{\prime })=log\int \exp (Q_{\varphi }(s^{\prime },a^{\prime }))d{a}^{\prime } \end{gathered}$$

于是Q-learning所有tricks都可以往上蹭了，如Double Q-Network、Dueling Structure、HER等等，具体可以参见Rainbow AAAI 2018这篇集齐所有tricks的Paper～

2.6 Policy Gradient with Soft Optimality

我们从上面，推断出Soft Optimality的Objective：
$$\begin{array}{rl}\underset{\theta }{\min }J(\theta )& =\underset{\theta }{\min }{D}_{KL}(\hat{p}(\tau )||p(\tau ))\\ & =\underset{\theta }{\max }\sum _{t=1}^T{E}_{(s_t,a_t)\sim \hat{p}(s_t,a_t)}[r(s_t,a_t)]+E_{s_t\sim \hat{p}(s_t)}[H[{\pi }_{\theta }(a_t|s_t)]]\\ & =\underset{\theta }{\max }\sum _{t=1}^T{E}_{(s_t,a_t)\sim \hat{p}(s_t,a_t)}[r(s_t,a_t)-log{\pi }_{\theta }(a_t|s_t)]\\ & =\underset{\theta }{\max }\sum _{t=1}^T{E}_{s_t\sim \hat{p}(s_t),a_t\sim {\pi }_{\theta }(a_t|s_t)}[r(s_t,a_t)-log{\pi }_{\theta }(a_t|s_t)]\end{array}$$

求梯度：$$\nabla J(\theta )={\nabla }_{\theta }\sum _{t=1}^T{E}_{(s_t\sim \hat{p}(s_t),a_t\sim {\pi }_{\theta }(a_t|s_t)})[r(s_t,a_t)]+{\nabla }_{\theta }\sum _{t=1}^T{E}_{(s_t\sim \hat{p}(s_t),a_t\sim {\pi }_{\theta }(a_t|s_t)})[-log{\pi }_{\theta }(a_t|s_t)]$$

第一项非常熟悉，先求：

$$\begin{array}{rl}& {\nabla }_{\theta }\sum _{t=1}^T{E}_{s_t\sim \hat{p}(s_t),a_t\sim {\pi }_{\theta }(a_t|s_t)}[r(s_t,a_t)]\\ & ={\nabla }_{\theta }{E}_{\tau \sim {\pi }_{\theta }(\tau )}[r(\tau )]\\ & ={\nabla }_{\theta }\int {\pi }_{\theta }(\tau )r(\tau )d\tau \\ & =\int {\pi }_{\theta }(\tau ){\nabla }_{\theta }log{\pi }_{\theta }(\tau )r(\tau )d\tau \\ & =E_{\tau \sim {\pi }_{\theta }(\tau )}[{\nabla }_{\theta }log{\pi }_{\theta }(\tau )r(\tau )](1)\\ & =E_{(s_t,a_t)\sim \hat{p}(s_t,a_t)}[\sum _{t=1}^T{\nabla }_{\theta }log\pi (a_t|s_t)\left]\right[\sum _{t=1}^{T}r(s_t,a_t)]\\ & {\to }^{baseline}{E}_{(s_t,a_t)\sim \hat{p}(s_t,a_t)}[\sum _{t=1}^T{\nabla }_{\theta }log\pi (a_t|s_t)\left]\right[\sum _{t^{\prime }=t}^{T}r(s_{t^{\prime }},a_{t^{\prime }})]\end{array}$$

再求第二项，省略负号，后续加上：

$$\begin{array}{rl}& {\nabla }_{\theta }\sum _{t=1}^T{E}_{s_t\sim \hat{p}(s_t),a_t\sim {\pi }_{\theta }(a_t|s_t)}[log{\pi }_{\theta }(a_t|s_t)]\\ & ={\nabla }_{\theta }{E}_{\tau \sim {\pi }_{\theta }(\tau )}[\sum _{t=1}^{T}log{\pi }_{\theta }(a_t|s_t)]\\ & ={\nabla }_{\theta }\int {\pi }_{\theta }(\tau )\sum _{t=1}^{T}log{\pi }_{\theta }(a_t|s_t)d\tau \\ & ={\nabla }_{\theta }\int {\pi }_{\theta }(\tau )log{\pi }_{\theta }(\tau )d\tau \\ & =\int {\nabla }_{\theta }{\pi }_{\theta }(\tau )log{\pi }_{\theta }(\tau )d\tau +\int {\pi }_{\theta }(\tau ){\nabla }_{\theta }log{\pi }_{\theta }(\tau )d\tau \\ & =\int {\pi }_{\theta }(\tau )[{\nabla }_{\theta }log{\pi }_{\theta }(\tau )]log{\pi }_{\theta }(\tau )+\int {\pi }_{\theta }(\tau ){\nabla }_{\theta }log{\pi }_{\theta }(\tau )d\tau \\ & =\int {\pi }_{\theta }(\tau ){\nabla }_{\theta }log{\pi }_{\theta }(\tau )[log{\pi }_{\theta }(\tau )+1]d\tau \\ & =E_{\tau \sim {\pi }_{\theta }(\tau )}[{\nabla }_{\theta }log{\pi }_{\theta }(\tau )[log{\pi }_{\theta }(\tau )+1\left]\right](2)\end{array}$$

利用$(1)-(2)$式复原下梯度：
$$\begin{array}{rl}\nabla J(\theta )& =E_{\tau \sim {\pi }_{\theta }(\tau )}[{\nabla }_{\theta }log{\pi }_{\theta }(\tau )r(\tau )]-E_{\tau \sim {\pi }_{\theta }(\tau )}[{\nabla }_{\theta }log{\pi }_{\theta }(\tau )[log{\pi }_{\theta }(\tau )+1\left]\right]\\ & =E_{\tau \sim {\pi }_{\theta }(\tau )}[{\nabla }_{\theta }log{\pi }_{\theta }(\tau )[r(\tau )-log{\pi }_{\theta }(\tau )-1\left]\right]\\ & =\sum _{t=1}^T{E}_{(s_t,a_t)\sim \hat{p}(s_t,a_t)}[{\nabla }_{\theta }log{\pi }_{\theta }(a_t|s_t)[\sum _{t^{\prime }=t}^{T}r(s_t,a_t)-\sum _{t^{\prime }=t}^{T}log{\pi }_{\theta }(a_t|s_t)-1\left]\right]\\ & {\approx }^{baseline}\sum _{t=1}^T{E}_{(s_t,a_t)\sim \hat{p}(s_t,a_t)}[{\nabla }_{\theta }log{\pi }_{\theta }(a_t|s_t)[r(s_t,a_t)+\sum _{t^{\prime }=t+1}^{T}r(s_{t^{\prime }},a_{t^{\prime }})-\sum _{t^{\prime }=t+1}^{T}log{\pi }_{\theta }(a_{t^{\prime }}|s_{t^{\prime }})-log{\pi }_{\theta }(a_t|s_t)-1]]\end{array}$$

因为policy的选择是${\pi }_{\theta }(a_t|s_t)=exp(Q(s_t,a_t)-V(s_t))$且$V(s_t)=log\int exp(Q(s_t,a_t))d{a}_t$，代入上面有

$$\begin{array}{rl}& \sum _{t=1}^T{E}_{(s_t,a_t)\sim \hat{p}(s_t,a_t)}[{\nabla }_{\theta }log{\pi }_{\theta }(a_t|s_t)[r(s_t,a_t)+\sum _{t^{\prime }=t+1}^{T}r(s_{t^{\prime }},a_{t^{\prime }})-\sum _{t^{\prime }=t+1}^{T}log{\pi }_{\theta }(a_{t^{\prime }}|s_{t^{\prime }})-log{\pi }_{\theta }(a_t|s_t)-1]]\\ & =\sum _{t=1}^T{E}_{(s_t,a_t)\sim \hat{p}(s_t,a_t)}\left[\right[{\nabla }_{\theta }{Q}_{\theta }(s_t,a_t)-{\nabla }_{\theta }{V}_{\theta }(s_t)\left]\right[r(s_t,a_t)+\underset{Q_{\theta }(s_{t^{\prime }},a_{t^{\prime }})}{\underbrace{\sum _{t^{\prime }=t+1}^{T}r(s_{t^{\prime }},a_{t^{\prime }})-\sum _{t^{\prime }=t+1}^{T}log{\pi }_{\theta }(a_{t^{\prime }}|s_{t^{\prime }})}}-\underset{V_{\theta }(s_t)-Q_{\theta }(s_t,a_t)}{\underbrace{log{\pi }_{\theta }(a_t|s_t)}}-1\left]\right]\\ & \approx \sum _{t=1}^T{E}_{(s_t,a_t)\sim \hat{p}(s_t,a_t)}[{\nabla }_{\theta }{Q}_{\theta }(s_t,a_t)[r(s_t,a_t)+softma{x}_{a_{t^{\prime }}}{Q}_{\theta }(s_{t^{\prime }},a_{t^{\prime }})-Q_{\theta }(s_t,a_t)]]\end{array}$$

这里进一步揭示了Policy Gradient with Soft Optimality 与Soft Q-leanring的联系，在Schulman 17年的Paper即Equivalence Between Policy Gradients and Soft Q-Learning

2.7 Soft Actor-Critic

从OpenAI盗一张图：

三、Summary

• Soft Policy Gradient是优化带Entropy regularized的Objective
• Soft Q-leanring仅仅是用softmax计算Q值即log-int-exp
• Soft的共同点是Policy为$\pi (a_t|s_t)=exp(Q(s_t,a_t)-V(s_t))$
• Softmax体现在$V(s_t)=log\int exp(Q(s_t,a_t))d{a}_t$
• Soft Optimality 从PGM角度开始推导，在Objective的角度来看，无非是多了一个entropy，所以有以下类似的说法，Soft Optimality Framework、Maximum Entropy RL、Entropy-regularized、Stochastic Energy Based Policy

Soft的好处：

后记

• Soft Actor Critic的Paper有点长，还没来得及消化，过一会再补上
• 整个Soft Optimality Framework逻辑还没有理得特别清楚，找一天推一遍，再补上（好烦，要去当助教了=。=
• 后续跑实验的时候，会顺便补点实验文章
• 接下来终于可以轻松点了，这Lecture涉及到PGM，彻底搞明白，花了不少时间= =，疯狂的理论推导，工程实践却如此简单，一个softmax，一个Energy-based policy，我的天！感觉与WGAN那两篇推导如出一辙，十分漂亮～

补一些参考资料

PGM Course Note 20
PGM Course Note 21
CS285 lec14
Soft Actor-Critic
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
（推荐）
Equivalence Between Policy Gradients and Soft Q-Learning
Reinforcement Learning with Deep Energy-Based Policies