Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review

• 基本概念

  • 轨迹概率 (轨迹$\tau$出现的概率)

  $$p(\tau )=p(s_1){\Pi }_{t}p(a_t|s_t)p(s_{t+1}|s_t,a_t)$$

  • 随机变量$O$ (是因为最优决策${}_{而非不小心}$选到动作a的概率)

  $$p(O_t|s_t,a_t)=\exp (r(s_t,a_t))$$

  问题一: r是正的概率会大于一。可以减去最大的r

  问题二: r的绝对值大小是否有影响。只有r的相对关系对结果有影响，因为我们考虑的是$P(\cdot |O)$

  • backward message (这步走完以后后续轨迹都会遵守最优决策的概率)

  $$\beta (s_t,a_t)=p(O_{t:T}|s_t,a_t)$$

\\

• 最优策略与随机变量$O$

  • 在最优策略下任意轨迹的出现概率

  $$\begin{gathered} p(\tau |O_{1:T})\propto p(\tau ,O_{1:T})=p(s_1){\Pi }_{t}p(O_t|s_t,a_t)p(s_{t+1}|s_t,a_t) \\ =p(s_1){\Pi }_{t}p(s_{t+1}|s_t,a_t)\exp (\sum _{t}r(s_t,a_t)) \end{gathered}$$

  在此文中我们认为只有$P(\cdot |O)$是可控的，因此除此之外的变量都能视为常数，如上面的正比就是因为分母上的$P(O_{1:T})$不可控是常数

  • 物理意义: 把最优策略定义为 : 在确定性环境中，相同累计回报的轨迹有相同的概率被选取，累计回报较低的轨迹有较低的概率被选取，且以指数方式递减。

\\

• Backward Message

  • 后向信息迭代关系式

  $$\begin{gathered} \beta (s_t,a_t)=p(O_{t:T}|s_t,a_t)={\int }_{s_{t+1}}p(O_{t:T},s_{t+1}|s_t,a_t) \\ ={\int }_{s_{t+1}}p(O_{t+1:T}|s_{t+1})p(s_{t+1}|s_t,a_t)p(O_t|s_t,a_t) \end{gathered}$$

  第一项是$\beta (s_{t+1})$定义如下行，第二项是环境给出，第三项正比于$\exp (r)$

  • 后向信息价值函数

  $$\beta (s_t)={\int }_{a_t}p(a_t|s_t)p(O_{t:T}|s_t,a_t)=p(O_{t:T}|s_t)$$

  第一项$p(a_t|s_t)$与最佳策略无关，因此我们不关心可以当常数(均匀分布)，第二项是$\beta (s_t,a_t)$

  • 求解方法(迭代) : 从T开始由后往前，计算$\beta (s_t,a_t)$、$\beta (s_t)$

\\

• 后验概率与决策

  • 决策

  $$\begin{gathered} p(a_t|s_t,O_{t:T})=\frac{p(a_t,s_t|O_{t:T})}{p(s_t|O_{t:T})}=\frac{p(a_t,s_t,O_{t:T})}{p(s_t,O_{t:T})} \\ =\frac{p(O_{t:T}|a_t,s_t)p(a_t|s_t)p(s_t)}{p(O_{t:T}|s_t)p(s_t)}\propto \frac{p(O_{t:T}|a_t,s_t)}{p(O_{t:T}|s_t)}=\frac{\beta (a_t,s_t)}{\beta (s_t)} \end{gathered}$$

  表示求取$\beta$等价于求取策略、而$\beta$求解法已经由上面给出

  • 价值函数

  $$\begin{gathered} Q(s_t,a_t)=\log \beta (s_t,a_t)=r(s_t,a_t)+\log E_{s_{t+1}}[\exp (V(s_{t+1}))] \\ V(s_t)=\log \beta (s_t)=\log E_A[\exp (Q(s_t,a_t))] \end{gathered}$$

  看到上面决策的方法后，很容易发现与SAC相同，因此用SAC的方式定义价值函数的关系

  当Q很大时$V\approx \max Q$，因此称为softmax

\\

• 优化目标 (与最大熵的联系)

  • 确定性环境

  $$p(\tau |O_{1:T})\propto p(s_1){\Pi }_{t}p(s_{t+1}|s_t,a_t)\exp (\sum _{t}r(s_t,a_t))=\exp (\sum _{t}r(s_t,a_t))$$

  • 函数逼近 (用$\hat{p}(\tau )$逼近$p(\tau |O_{1:T})$)

  $$\begin{gathered} \underset{\hat{p}(\tau )}{\min }{D}_{KL}(\hat{p}(\tau )||p(\tau |O_{1:T})) \\ =\underset{\hat{p}(\tau )}{\max }{E}_{s,a\sim \hat{p}(\tau )}[\log p(\tau |O_{1:T})-\log \hat{p}(\tau )] \\ =\underset{\hat{p}(\tau )}{\max }{E}_{s,a\sim \hat{p}(\tau )}[\log p(s_1)+\sum _{t=1}^T\log p(s_{t+1}|s_t,a_t)+\sum _{t=1}^{T}r(s_t,a_t) \\ -\log p(s_1)-\sum _{t=1}^T\log p(s_{t+1}|s_t,a_t)-\sum _{t=1}^T\log \hat{\pi }(a_t|s_t))] \\ =\underset{\hat{p}(\tau )}{\max }{E}_{s,a\sim \hat{p}(\tau )}[\sum _{t=1}^{T}r(s_t,a_t)-\sum _{t=1}^T\log \hat{\pi }(a_t|s_t))] \\ =\underset{\hat{p}(\tau )}{\max }\sum _{t=1}^T{E}_{s,a\sim \hat{p}(\tau )}[r(s_t,a_t)]-\sum _{t=1}^T{E}_{s\sim \hat{p}(\tau )}[H(\hat{\pi }(a_t|s_t))] \end{gathered}$$

  以上证明，确定性环境下最大熵等于后验概率决策问题

\\

• 随机环境下的优化

  • 随机环境

  $$\begin{gathered} \underset{\hat{p}(\tau )}{\min }{D}_{KL}(\hat{p}(\tau )||p(\tau |O_{1:T})) \\ =\underset{\hat{p}(\tau )}{\max }{E}_{s,a\sim \hat{p}(\tau )}[\log p(s_1)]+\sum _{t=1}^T{E}_{s,a\sim \hat{p}(\tau )}[r(s_t,a_t)+\log p(s_{t+1}|s_t,a_t)]-\sum _{t=1}^T{E}_{s\sim \hat{p}(\tau )}[H(\hat{\pi }(a_t|s_t))] \end{gathered}$$

  目标函数内含有dynamic，隐式的代表了是可以通过改变dynamic来优化目标函数 (但偏离真实dynamic时会有惩罚)，这样的假设不合理，会使得学到的策略更激进

  并且此目标函数在model-free的场景下是难以直接优化的

  • 函数逼近 (限制dynamic的情况下优化)

  $$\hat{p}(\tau )=p(s_1){\Pi }_{t}p(s_{t+1}|s_t,a_t)\exp (\sum _{t}r(s_t,a_t))\approx p(\tau |O_{1:T})$$

  上述显示的规定$\hat{p}$无法改变dynamic，可以用类似确定性环境的公式，如下

  $$\begin{gathered} \underset{\hat{p}(\tau )}{\min }{D}_{KL}(\hat{p}(\tau )||p(\tau |O_{1:T})) \\ =\underset{\hat{p}(\tau )}{\max }\sum _{t=1}^T{E}_{s,a\sim \hat{p}(\tau )}[r(s_t,a_t)]-\sum _{t=1}^T{E}_{s\sim \hat{p}(\tau )}[H(\hat{\pi }(a_t|s_t))] \\ =\underset{\hat{p}(\tau )}{\max }\sum _{t=1}^T-E_{s\sim \hat{p}(\tau )}[D_{KL}(\hat{\pi }||\frac{\exp (r_t)}{exp(V_t)})]+\sum _{t=1}^T{E}_{s\sim \hat{p}(\tau )}[V_t] \end{gathered}$$

  可以看出来当$\hat{\pi }=\frac{\exp (r_t)}{exp(V_t)}$的时候为最优策略，且收益是${\sum }_{t}E[V_t]$，其中$V_t=\log {\int }_A\exp (r(s_t,a_t))$

  跟确定性环境一样，按照SAC的方法定义$Q,V$ (没看懂公式14)

  • 变分推断 (概述: 用简单函数$q$估计真实分布$p$，可以先找到$p(x)$的下界$ELBO$并优化他)

  $$\log p(x)\ge E_{z\sim q}[\log p(x,z)-\log q(z)]=ELBO$$

  用$\hat{p}(x)=p(s_1){\Pi }_{t}p(s_{t+1}|s_t,a_t)]\hat{\pi }(a_t|s_t)$估计$p(\tau )$，其中dynamic限制只能使用真实值，带入上式得到如下形式

  $$\begin{gathered} \log p(O_{1:T})\ge E_{z\sim \hat{p}}[\log p(O_{1:T},\tau )-\log \hat{p}(\tau )] \\ =E_{z\sim \hat{p}}[\sum _{t=1}^{T}r(s_t,a_t)-\log \hat{\pi }(a_t|s_t)]=ELBO \end{gathered}$$

  上面证明了最大化熵RL等价于最大化最优决策概率的下界

  其中第二行等式成立是因为dynamic相减对消了