Soft Actor-Critic(SAC算法)

1. 基本概念

1.1 soft Q-value

$${\tau }^{\pi }Q(s_t,a_t)=r(s_t,a_t)+\gamma \cdot E_{s_{t+1}\sim p}[V(s_{t+1})]$$

1.2 soft state value function

$$V(s_t)=E_{a_t\sim \pi }[Q(s_t,a_t)-\alpha \cdot log\pi (a_t|s_t)]$$

1.3 Soft Policy Evaluation

$$Q^{k+1}={\tau }^{\pi }{Q}^k$$
当k趋于无穷时，$Q^k$将收敛至$\pi$的soft Q-value。
证明：
$$r_{\pi }(s_t,a_t)=r(s_t,a_t)+\gamma \cdot E_{s_{t+1}\sim p}[H(\pi (\cdot |s_{t+1}))]$$
$$Q(s_t,a_t)=r(s_t,a_t)+\gamma \cdot E_{s_{t+1}\sim p}[H(\pi (\cdot |s_{t+1}))+E_{s_{t+1},a_{t+1}\sim {\rho }_{\pi }}[Q(s_{t+1},a_{t+1})]$$
$$Q(s_t,a_t)=r(s_t,a_t)+\gamma \cdot E_{s_{t+1},a_{t+1}\sim {\rho }_{\pi }}[-log(\pi (a_{t+1}|s_{t+1}))+E_{s_{t+1},a_{t+1}\sim {\rho }_{\pi }}[Q(s_{t+1},a_{t+1})]$$
$$Q(s_t,a_t)=r(s_t,a_t)+\gamma \cdot E_{s_{t+1},a_{t+1}\sim {\rho }_{\pi }}[Q(s_{t+1},a_{t+1})-log(\pi (a_{t+1}|s_{t+1}))$$
当|A|<∞时，可以保证熵有界，因而能保证收敛。

1.4 policy improvement

$${\pi }_{new}=argmi{n}_{{\pi }^{{}^{\prime }}\in \Pi }{D}_{KL}({\pi }^{{}^{\prime }}(\cdot |s_t)||\frac{exp(Q^{{\pi }_{old}}(s_t,\cdot ))}{Z^{{\pi }_{old}}(s_t)})$$

1.5 soft policy improvemrnt

$$Q^{{\pi }_{new}}(s_t,a_t)\ge Q^{{\pi }_{old}}(s_t,a_t)$$
s.t.为：
$${\pi }_{old}\in \Pi ,(s_t,a_t)\in S\times A,|A|<\infty$$
证明如下：
$$\begin{gathered} {\pi }_{new}=argmi{n}_{{\pi }^{{}^{\prime }}\in \Pi }{D}_{KL}({\pi }^{{}^{\prime }}(\cdot |s_t)||exp(Q^{{\pi }_{old}}(s_t,\cdot )-log(Z(s_t)))) \\ =argmi{n}_{{\pi }^{{}^{\prime }}\in \Pi }{J}_{{\pi }_{old}}({\pi }^{{}^{\prime }}(\cdot |s_t)) \end{gathered}$$
$$J_{{\pi }_{old}}({\pi }^{{}^{\prime }}(\cdot |s_t))=E_{a_t\sim {\pi }^{{}^{\prime }}}[log({\pi }^{{}^{\prime }}(s_t,a_t))-Q^{{\pi }_{old}}(s_t,a_t)+log(Z(s_t))]$$
由于一直可以取${\pi }_{new}={\pi }_{old}$,所有总能满足：
$$E_{a_t\sim {\pi }_{new}}[log({\pi }_{new}(a_t|s_t))-Q^{{\pi }_{old}}(s_t,a_t)]\le E_{a_t\in {\pi }_{old}}[log({\pi }_{old}(a_t|s_t))-Q^{{\pi }_{old}}(s_t,a_t)]$$

$$\begin{gathered} E_{a_t\sim {\pi }_{new}}[log({\pi }_{new}(a_t|s_t))-Q^{{\pi }_{old}}(s_t,a_t)]\le -V^{{\pi }_{old}}(s_t) \\ {E}_{a_t\sim {\pi }_{new}}[Q^{{\pi }_{old}}(s_t,a_t)-log({\pi }_{new}(a_t|s_t))]\ge V^{{\pi }_{old}}(s_t) \end{gathered}$$
$$\begin{gathered} Q^{{\pi }_{old}}(s_t,a_t)=r(s_t,a_t)+\gamma \cdot E_{s_{t+1}\sim p}[V^{{\pi }_{old}}(s_{t+1})] \\ \le r(s_t,a_t)+\gamma \cdot E_{s_{t+1}\sim p{E}_{a_{t+1}\sim {\pi }_{new}}}[Q^{{\pi }_{old}}(s_t,a_t)-log({\pi }_{new}(a_t|s_t)] \\ \le .......... \\ \le Q^{{\pi }_{new}}(s_t,a_t) \end{gathered}$$

1.5 soft policy iteration

假设：$|A|<\infty ；\pi \in \Pi$
经过不断地soft policy evaluation和policy improvement，最终policy会收敛至${\pi }^{\star }$,其满足
$$Q^{{\pi }^{\star }}(s_t,a_t)\ge Q^{\pi }(s_t,a_t)；其中\pi \in \Pi$$

2. soft actor critic

2.1 soft value function

1. loss function
   $$J_V(\psi )=E_{s_t\sim D}[\frac{1}{2}(V_{\psi }(s_t)-E_{a_t\sim {\pi }_{\varphi }}[Q_{\theta }(s_t,a_t)-log({\pi }_{\varphi }(a_t|s_t))){]}^2]$$
2. gradient
   $${\hat{\nabla }}_{\psi }{J}_V(\psi )={\nabla }_{\psi }{V}_{\psi }(s_t)\cdot (V_{\psi }(s_t)-Q_{\theta }(s_t,a_t)+log({\pi }_{\varphi }(a_t|s_t)))$$

2.2 soft Q-function

1. loss function
   $$J_Q(\theta )=E_{(s_t,a_t)\sim D}[\frac{1}{2}(Q_{\theta }(s_t,a_t)-\hat{Q}(s_t,a_t){)}^2]$$
   $$\hat{Q}(s_t,a_t)=r(s_t,a_t)+\gamma \cdot E_{s_{t+1}\sim p}[V_{\bar{\psi }}(s_{t+1})]$$
2. gradient
   $${\hat{\nabla }}_{\theta }{J}_Q(\theta )={\nabla }_{\theta }{Q}_{\theta }(s_t,a_t)\cdot [Q_{\theta }(s_t,a_t)-r(s_t,a_t)-\gamma \cdot V_{\bar{\psi }}(s_{t+1})]$$

2.3 policy improvement

1. loss function
   $$J_{\pi }(\varphi )=E_{s_t\sim D}[D_{KL}({\pi }_{\varphi }(\cdot |s_t)||\frac{exp(Q_{\theta }(s_t,\cdot ))}{Z_{\theta }(s_t)})]$$
   reparameterize the policy
   $$a_t=f_{\varphi }({ϵ}_t;s_t)=f_{\varphi }^{\mu }(s_t)+{ϵ}_t\cdot f_{\varphi }^{\sigma }(s_t)$$
   $$J_{\pi }(\varphi )=E_{s_t\sim D;{ϵ}_t\in N}[log({\pi }_{\varphi }(f_{\varphi }({ϵ}_t;s_t)|s_t))-Q_{\theta }(s_t,f_{\varphi }({ϵ}_t;s_t))]$$
2. gradient
   $${\nabla }_{\theta }{E}_{q_{\theta }(Z)}[f_{\theta }(Z)]=E_{q_{\theta }(Z)}[\frac{\partial f_{\theta }(Z)}{\partial \theta }]+E_{q_{\theta }(Z)}[\frac{d{f}_{\theta }(Z)}{dZ}\cdot \frac{dZ}{d\theta }]$$
   $${\hat{\nabla }}_{\varphi }{J}_{\pi }(\varphi )={\nabla }_{\varphi }log({\pi }_{\varphi }(a_t;s_t)|s_t))+{\nabla }_{\varphi }{f}_{\varphi }({ϵ}_t;s_t)\cdot ({\nabla }_{a_t}log(\pi (a_t|s_t))-{\nabla }_{a_t}{Q}_{\theta }(s_t,a_t))$$

3. 算法流程

By CyrusMay 2022.09.06
世界 再大 不过 你和我
用最小回忆 堆成宇宙
————五月天（因为你 所以我）————