迁移强化学习论文笔记（一）（Successor Features）

迁移强化学习论文笔记（一）（Successor Features）

一.Background and problem formulation

$M\equiv (\mathcal{S},\mathcal{A},p,R,\gamma )$

$\mathcal{S}$:状态空间

$\mathcal{A}$：行动空间

$p$:$p(\cdot \mid s_t,a_t)$状态转移概率

$R$:$R(s_t,a_t,s_{t+1})$奖励

二.Successor features

假设奖励函数可以写为
$$r\left(s,a,s^{\prime }\right)=\mathbf{\varphi }{\left(s,a,s^{\prime }\right)}^{\top }\mathbf{w},$$
其中$\mathbf{\varphi }(s,a,s^{\prime })$是d维向量，$\mathbf{w}$是对应的权重。利用这种形式，我们有以下结论（定义${\mathbf{\varphi }}_{t+1}=\mathbf{\varphi }(s_t,a_t,s_{t+1})$）
$$\begin{array}{rl}{Q}^{\pi }(s,a)& ={\mathrm{E}}^{\pi }\left[r_{t+1}+\gamma r_{t+2}+\dots \mid S_t=s,A_t=a\right]\\ & ={\mathrm{E}}^{\pi }\left[{\mathbf{\varphi }}_{t+1}^{\top }\mathbf{w}+\gamma {\mathbf{\varphi }}_{t+2}^{\top }\mathbf{w}+\dots \mid S_t=s,A_t=a\right]\\ & ={\mathrm{E}}^{\pi }{\left[\sum _{i=t}^{\infty }{\gamma }^{i-t}{\mathbf{\varphi }}_{i+1}\mid S_t=s,A_t=a\right]}^{\top }\mathbf{w}={\mathbf{\psi }}^{\pi }(s,a{)}^{\top }\mathbf{w}.\end{array}$$
${\mathbf{\psi }}^{\pi }(s,a)$是在策略$\pi$下$(s,a)$的Successor Features（SFs）

由定义知${\mathbf{\psi }}^{\pi }(s,a)={\mathrm{E}}^{\pi }[{\mathbf{\varphi }}_{t+1}+\gamma {\mathbf{\varphi }}_{t+2}+{\gamma }^2{\mathbf{\varphi }}_{t+3}+\cdots \mid S_t=s,A_t=a]$可得如下贝尔曼公式
$$\begin{array}{rl}{\mathbf{\psi }}^{\pi }(s,a)& ={\mathrm{E}}^{\pi }[{\mathbf{\varphi }}_{t+1}+\gamma {\mathbf{\varphi }}_{t+2}+{\gamma }^2{\mathbf{\varphi }}_{t+3}+\cdots \mid S_t=s,A_t=a]\\ & ={\mathrm{E}}_{S_{t+1},A_{t+1}}[{\mathbf{\varphi }}_{t+1}+{\mathbf{\psi }}^{\pi }(S_{t+1},A_{t+1})\mid S_t=s,A_t=a]\text{如果采取确定策略}\pi \\ & ={\mathbf{\varphi }}_{t+1}(s,a)+{\mathrm{E}}_{S_{t+1}}[{\mathbf{\psi }}^{\pi }(S_{t+1},\pi (S_{t+1}))\mid S_t=s,A_t=a]\end{array}$$
利用上式即可迭代求解${\mathbf{\psi }}^{\pi }(s,a)$,而对于$\mathbf{w}$的求解则是一个有监督学习问题很多机器学习算法都可进行。

这样对于不同的任务只要求解出不同的$\mathbf{w}$即可。

三.Generalized policy improvement

作者在论文中还证明了迁移强化学习的泛化误差界

Theorem 1. (Generalized Policy Improvement) Let ${\pi }_1,{\pi }_2,\dots ,{\pi }_n$ be $n$ decision policies and let ${\tilde{Q}}^{{\pi }_1},{\tilde{Q}}^{{\pi }_2},\dots ,{\tilde{Q}}^{{\pi }_n}$ be approximations of their respective action-value functions such that
∣ Q π i ( s , a ) − Q ~ π i ( s , a ) ∣ ≤ ϵ  for all  s ∈ S , a ∈ A , and  i ∈ { 1 , 2 , … , n } .  \left|Q^{\pi_i}(s, a)-\tilde{Q}^{\pi_i}(s, a)\right| \leq \epsilon \text { for all } s \in \mathcal{S}, a \in \mathcal{A} \text {, and } i \in\{1,2, \ldots, n\} \text {. } ​Qπi​(s,a)−Q~​πi​(s,a) ​≤ϵ for all s∈S,a∈A, and i∈{1,2,…,n}. 

Define
$$\pi (s)\in \underset{a}{\mathrm{argmax}}\underset{i}{\max }{\tilde{Q}}^{{\pi }_i}(s,a).$$

Then,
$$Q^{\pi }(s,a)\ge \underset{i}{\max }{Q}^{{\pi }_i}(s,a)-\frac{2}{1-\gamma }ϵ$$
for any $s\in \mathcal{S}$ and $a\in \mathcal{A}$, where $Q^{\pi }$ is the action-value function of $\pi$.

proof:为简化符号，定义
$$\begin{gathered} Q_{max}(s,a)={\text{max}}_i{Q}^{{\pi }_i}(s,a)(在策略{\pi }_i中的最优动作价值函数) \\ {\tilde{Q}}_{max}(s,a)={\text{max}}_i\widetilde{Q^{{\pi }_i}}(s,a)(在策略{\pi }_i中最优动作价值函数的估计值) \end{gathered}$$
借助以上符号我们有如下不等式
$$\left|Q_{\max }(s,a)-{\tilde{Q}}_{\max }(s,a)\right|=\left|\underset{i}{\max }{Q}^{{\pi }_i}(s,a)-\underset{i}{\max }{\tilde{Q}}^{{\pi }_i}(s,a)\right|\le \underset{i}{\max }\left|Q^{{\pi }_i}(s,a)-{\tilde{Q}}^{{\pi }_i}(s,a)\right|\le ϵ.$$
于是我们可得
$$Q_{\max }(s,a)-ϵ\le {\tilde{Q}}_{\max }(s,a)$$
借助贝尔曼算子$T^{\pi }$,其中
$$\begin{gathered} T^{\pi }f(s,a)=r(s,a)+\gamma {\mathrm{E}}_{s^{\prime }\sim p(s^{\prime }\mid s,a)}[V(s^{\prime })] \\ V(s^{\prime })={\mathrm{E}}_{a\sim \pi (a\mid s^{\prime })}[f(s^{\prime },a)] \\ r(s,a)={\mathrm{E}}_{s^{\prime }\sim p(s^{\prime }\mid s,a)}[r(s,a,s^{\prime })] \end{gathered}$$
因我们采用确定策略$\pi$(在所有策略中选取能使得动作价值最大的动作)，$V(s^{\prime })=f(s^{\prime },\pi (s^{\prime }))$

对于任意$(s,a)\in \mathcal{S}\times \mathcal{A}$和任意策略${\pi }_i$我们都有下式成立
$$\begin{array}{rl}{T}^{\pi }{\tilde{Q}}_{\max }(s,a)& =r(s,a)+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right){\tilde{Q}}_{\max }\left(s^{\prime },\pi \left(s^{\prime }\right)\right)\\ & =r(s,a)+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)\underset{b}{\max }{\tilde{Q}}_{\max }\left(s^{\prime },b\right)\\ & \ge r(s,a)+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)\underset{b}{\max }{Q}_{\max }\left(s^{\prime },b\right)-\gamma ϵ\\ & \ge r(s,a)+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)Q_{\max }\left(s^{\prime },{\pi }_i\left(s^{\prime }\right)\right)-\gamma ϵ\\ & \ge r(s,a)+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)Q^{{\pi }_i}\left(s^{\prime },{\pi }_i\left(s^{\prime }\right)\right)-\gamma ϵ\\ & =T^{{\pi }_i}{Q}^{{\pi }_i}(s,a)-\gamma ϵ\\ & =Q^{{\pi }_i}(s,a)-\gamma ϵ.\end{array}$$
又因$T^{\pi }{\tilde{Q}}_{\max }(s,a)\ge Q^{{\pi }_i}(s,a)-\gamma ϵ$对任意策略成立
$$\begin{array}{rl}{T}^{\pi }{\tilde{Q}}_{\max }(s,a)& \ge Q^{{\pi }_i}(s,a)-\gamma ϵfor\forall {\pi }_i\\ & \ge {\text{max}}_i{Q}^{{\pi }_i}-\gamma ϵ\\ & \ge {\tilde{Q}}_{\max }(s,a)-\gamma -\gamma ϵ\end{array}$$
为得出最终结论。我们还需要证明以下事实
$$\begin{array}{rl}{T}^{\pi }(f(s,a)+c)& =r(s,a)+\gamma {\mathrm{E}}_{s^{\prime }\sim p(s^{\prime }\mid s,a)}[f(s^{\prime },\pi (s^{\prime }))+c]\\ & =r(s,a)+\gamma {\mathrm{E}}_{s^{\prime }\sim p(s^{\prime }\mid s,a)}[f(s^{\prime },\pi (s^{\prime }))]+\gamma \cdot c\\ & =T^{\pi }(f(s,a))+\gamma \cdot c\end{array}$$
于是我们可知
$$\begin{array}{rl}{T}^{\pi }{\tilde{Q}}_{\max }(s,a)& \ge {\tilde{Q}}_{\max }(s,a)-(1+\gamma )ϵ\\ T^{\pi }(T^{\pi }{\tilde{Q}}_{\max }(s,a))& \ge T^{\pi }{\tilde{Q}}_{\max }(s,a)-\gamma (1+\gamma )ϵ\\ ⋮& \\ (T^{\pi }{)}^k({\tilde{Q}}_{\max }(s,a))& \ge (T^{\pi }{)}^{k-1}-{\gamma }^{k-1}(1+\gamma )ϵ\end{array}$$
将上式连续相加，且当$k$趋于无穷时可知
$$\begin{array}{rl}{Q}^{\pi }(s,a)& =\underset{k\to \infty }{\lim }{\left(T^{\pi }\right)}^k{\tilde{Q}}_{\max }(s,a)\\ & \ge {\tilde{Q}}_{\max }(s,a)-\frac{1+\gamma }{1-\gamma }ϵ\\ & \ge Q_{\max }(s,a)-ϵ-\frac{1+\gamma }{1-\gamma }ϵ\\ & =\underset{i}{\max }{Q}^{{\pi }_i}(s,a)-\frac{2}{1-\gamma }ϵ\end{array}$$
证毕

想要证明最后误差界，我们还需借助以下引理

Lemma 1. Let ${\delta }_{ij}={\max }_{s,a}\left|r_i(s,a)-r_j(s,a)\right|$. Then,
$$Q_i^{{\pi }_i^{\ast }}(s,a)-Q_i^{{\pi }_j^{\ast }}(s,a)\le \frac{2{\delta }_{ij}}{1-\gamma }.$$

proof为简化记号，令 $Q_i^j(s,a)\equiv Q_i^{{\pi }_j^{\ast }}(s,a)$.
$$\begin{array}{rl}{Q}_i^i(s,a)-Q_i^j(s,a)& =Q_i^i(s,a)-Q_j^j(s,a)+Q_j^j(s,a)-Q_i^j(s,a)\\ & \le \left|Q_i^i(s,a)-Q_j^j(s,a)\right|+\left|Q_j^j(s,a)-Q_i^j(s,a)\right|.\end{array}$$
令 ${\Delta }_{ij}={\max }_{s,a}\left|Q_i^i(s,a)-Q_j^j(s,a)\right|$.
$$\begin{array}{rl}\left|Q_i^i(s,a)-Q_j^j(s,a)\right|& =\left|r_i(s,a)+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)\underset{b}{\max }{Q}_i^i\left(s^{\prime },b\right)-r_j(s,a)-\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)\underset{b}{\max }{Q}_j^j\left(s^{\prime },b\right)\right|\\ & =\left|r_i(s,a)-r_j(s,a)+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)\left(\underset{b}{\max }{Q}_i^i\left(s^{\prime },b\right)-\underset{b}{\max }{Q}_j^j\left(s^{\prime },b\right)\right)\right|\\ & \le \left|r_i(s,a)-r_j(s,a)\right|+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)\left|\underset{b}{\max }{Q}_i^i\left(s^{\prime },b\right)-\underset{b}{\max }{Q}_j^j\left(s^{\prime },b\right)\right|\\ & \le \left|r_i(s,a)-r_j(s,a)\right|+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)\underset{b}{\max }\left|Q_i^i\left(s^{\prime },b\right)-Q_j^j\left(s^{\prime },b\right)\right|\\ & \le {\delta }_{ij}+\gamma {\Delta }_{ij}.\end{array}$$

从上式中可知
$${\Delta }_{ij}\le \frac{1}{1-\gamma }{\delta }_{ij}.$$

定义 ${\Delta }_{ij}^{\prime }=$ ${\max }_{s,a}\left|Q_i^i(s,a)-Q_i^j(s,a)\right|$.
$$\begin{array}{rl}\left|Q_j^j(s,a)-Q_i^j(s,a)\right|& =\left|r_j(s,a)+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)Q_j^j\left(s^{\prime },{\pi }_j^{\ast }\left(s^{\prime }\right)\right)-r_i(s,a)-\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)Q_i^j\left(s^{\prime },{\pi }_j^{\ast }\left(s^{\prime }\right)\right)\right|\\ & =\left|r_i(s,a)-r_j(s,a)+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)\left(Q_j^j\left(s^{\prime },{\pi }_j^{\ast }\left(s^{\prime }\right)\right)-Q_i^j\left(s^{\prime },{\pi }_j^{\ast }\left(s^{\prime }\right)\right)\right)\right|\\ & \le \left|r_i(s,a)-r_j(s,a)\right|+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)\left|Q_j^j\left(s^{\prime },{\pi }_j^{\ast }\left(s^{\prime }\right)\right)-Q_i^j\left(s^{\prime },{\pi }_j^{\ast }\left(s^{\prime }\right)\right)\right|\\ & \le {\delta }_{ij}+\gamma {\Delta }_{ij}^{\prime }.\end{array}$$

同样可知
$${\Delta }_{ij}^{\prime }\le \frac{1}{1-\gamma }{\delta }_{ij}.$$

证毕

Theorem 2. Let $M_i\in {\mathcal{M}}^{\varphi }$ and let $Q_i^{{\pi }_j^{\ast }}$ be the value function of an optimal policy of $M_j\in {\mathcal{M}}^{\varphi }$ when executed in $M_i$. Given the set $\left\{{\tilde{Q}}_i^{{\pi }_1^{\ast }},{\tilde{Q}}_i^{{\pi }_2^{\ast }},\dots ,{\tilde{Q}}_i^{{\pi }_n^{\ast }}\right\}$ such that
∣ Q i π j ∗ ( s , a ) − Q ~ i π j ∗ ( s , a ) ∣ ≤ ϵ  for all  s ∈ S , a ∈ A , and  j ∈ { 1 , 2 , … , n } , \left|Q_i^{\pi_j^*}(s, a)-\tilde{Q}_i^{\pi_j^*}(s, a)\right| \leq \epsilon \text { for all } s \in S, a \in A \text {, and } j \in\{1,2, \ldots, n\}, ​Qiπj∗​​(s,a)−Q~​iπj∗​​(s,a) ​≤ϵ for all s∈S,a∈A, and j∈{1,2,…,n},
let
$$\pi (s)\in \underset{a}{\mathrm{argmax}}\underset{j}{\max }{\tilde{Q}}_i^{{\pi }_j^{\ast }}(s,a).$$

Finally, let ${\varphi }_{\max }={\max }_{s,a}\Vert \varphi (s,a)\Vert$, where $\Vert \cdot \Vert$ is the norm induced by the inner product adopted. Then,
$$Q_i^{\ast }(s,a)-Q_i^{\pi }(s,a)\le \frac{2}{1-\gamma }\left({\varphi }_{\max }\underset{j}{\min }\Vert {\mathbf{w}}_i-{\mathbf{w}}_j\Vert +ϵ\right).$$

proof:
$$\begin{array}{rl}{Q}_i^{\ast }(s,a)-Q_i^{\pi }(s,a)& \le Q_i^{\ast }(s,a)-Q_i^{{\pi }_j^{\ast }}(s,a)+\frac{2}{1-\gamma }ϵ\\ & \le \frac{2}{1-\gamma }\underset{s,a}{\max }\left|r_i(s,a)-r_j(s,a)\right|+\frac{2}{1-\gamma }ϵ\\ & =\frac{2}{1-\gamma }\underset{s,a}{\max }\left|\varphi (s,a{)}^{\top }{\mathbf{w}}_i-\varphi (s,a{)}^{\top }{\mathbf{w}}_j\right|+\frac{2}{1-\gamma }ϵ\\ & =\frac{2}{1-\gamma }\underset{s,a}{\max }\left|\varphi (s,a{)}^{\top }\left({\mathbf{w}}_i-{\mathbf{w}}_j\right)\right|+\frac{2}{1-\gamma }ϵ\\ & \le \frac{2}{1-\gamma }\underset{s,a}{\max }\Vert \varphi (s,a)\Vert \Vert {\mathbf{w}}_i-{\mathbf{w}}_j\Vert +\frac{2}{1-\gamma }ϵ\\ & =\frac{2{\varphi }_{\max }}{1-\gamma }\Vert {\mathbf{w}}_i-{\mathbf{w}}_j\Vert +\frac{2}{1-\gamma }ϵ.\end{array}$$