每天一个RL基础理论(5)——Sample Complexity(中）

• 搬砖来源：https://wensun.github.io/CS6789_fall_2021.html
• 细节来源：Model-Based Reinforcement Learning with a Generative Model is Minimax Optimal 2020 JMLR
• 主题：经典方法VI&PI在转移矩阵未知情况下，使用经典算法VI&PI达到near-optimal policy所需的样本复杂度
• setting：infinite horizon discounted MDP / unknown transition probability / deterministic reward / deterministic policy
• 解决的问题：给定$\mathcal{M}=(S,A,P,r,\gamma )$，其中转移矩阵$P$未知，经典方法VI&PI需要多少的样本(transitions$(s,a,r,s^{\prime })$)才能学习到near optimal的策略(deterministic & stationary)？
• 使用的理论工具：Statistics Theory

一、样本复杂度的分析目标是什么？

1. 首先对未知的转移矩阵P，用均匀交互假设（generative model）进行建模
   $$\hat{P}(s^{\prime }\mid s,a)=\frac{\#(s^{\prime },s,a)}{N}$$
2. 在$\hat{M}=(S,A,\hat{P},r,\gamma )$中使用VI算法学习到的元素均记为$\hat{Q},\hat{V}$，其中${\hat{\pi }}^{\star },{\hat{Q}}^{\star },{\hat{V}}^{\star }$分别为基于$\hat{M}$的最优策略、Q值、V值，而我们最想要的是真实$M=(S,A,P,r,\gamma )$中对应的${\pi }^{\star }$
3. 下面用$ϵ$来量化near-optimal的“near optimal”的程度，可简记为$\Vert V^{\star }-V^{\pi }{\Vert }_{\infty }\le {ϵ}_V$
   $$V^{\star }(s)-V^{\pi }(s)\le {ϵ}_V\forall s$$
4. 第n次迭代得到的策略${\pi }_n=\underset{a}{\mathrm{argmax}}{Q}^{{\pi }_n}(s,a)$，其与最优策略${\pi }^{\star }$的性能满足：
   $$V^{{\pi }_n}\ge V^{\star }(s)-\frac{2{\gamma }^n}{1-\gamma }||Q_0-Q^{\star }|{|}_{\infty }\forall s\in S$$因为V和Q之间的关系可以通过以上公式进行转换，所以最终目标可以选择分析Q函数，毕竟在VI或PI算法均以Q为直接迭代对象
   $$Q^{\star }(s,a)-{\hat{Q}}^{{\hat{\pi }}_n}(s,a)\le {ϵ}_Q\forall s,a$$
5. 因此只需要寻找${ϵ}_Q$与$N$的关系，就可以回答样本复杂度的问题：对转移矩阵建模后，想用VI算法得到一个$ϵ$-optimal的策略，至少需要哪个量级的样本数$N$？

二、最终目标的分析与拆解

1. 先对最终目标进行放缩，找它的上界
   $$\Vert Q^{\star }-{\hat{Q}}^{{\hat{\pi }}_n}{\Vert }_{\infty }\le \Vert Q^{\star }-{\hat{Q}}^{\star }{\Vert }_{\infty }+\Vert {\hat{Q}}^{\star }-{\hat{Q}}^{{\hat{\pi }}_n}{\Vert }_{\infty }$$
2. $\Vert {\hat{Q}}^{\star }-{\hat{Q}}^{{\hat{\pi }}_n}{\Vert }_{\infty }$这项并不难，其对应的是MDP已知的planning问题，在VI的Computational Complexity中可知
   $$\begin{array}{rl}& \Vert {\hat{Q}}^{\star }-{\hat{Q}}^{{\hat{\pi }}_n}{\Vert }_{\infty }\le {\gamma }^n\Vert {\hat{Q}}_0-{\hat{Q}}^{\star }{\Vert }_{\infty }\le {\gamma }^n\frac{1}{1-\gamma }\\ & =(1-(1-\gamma ){)}^n\frac{1}{1-\gamma }\\ & \le \exp (-(1-\gamma )n)\frac{1}{1-\gamma }\le {ϵ}_{opt}\end{array}$$
3. 因此在$n\ge O((1-\gamma {)}^{-1}\ln {ϵ}_{opt}^{-1})$这个量级的迭代步数后，有$\Vert {\hat{Q}}^{\star }-{\hat{Q}}^{{\hat{\pi }}_n}{\Vert }_{\infty }\le {ϵ}_{opt}$，这个称作optimization error，也即${ϵ}_{opt}$下标的来源
4. 难点在$\Vert Q^{\star }-{\hat{Q}}^{\star }{\Vert }_{\infty }$这一项中，先将它写完整进行细看（optimal policy可以是deterministic的！）对一个$(s,a)$而言，有
   $${\hat{Q}}^{\star }(s,a)-Q^{\star }(s,a)=r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim \hat{p}(\cdot |s,a)}[\hat{Q}(s^{\prime },{\hat{\pi }}^{\star }(s^{\prime }))]-r(s,a)-\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot |s,a)}[Q(s^{\prime },{\pi }^{\star }(s^{\prime }))]$$首先，Q函数不同，其次，策略也不同，似乎一筹莫展，先回忆一下

三、相关公式的理解与梳理

2.1 Bellman Consistency Equation

$$\begin{array}{rl}{V}^{\pi }(s)& ={\mathbb{E}}_{a\sim \pi (\cdot \mid s)}\left[Q^{\pi }(s,a)\right]\text{(V-Q)}\\ Q^{\pi }(s,a)& =r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,a)}\left[V^{\pi }(s^{\prime })\right]\text{(Q-V)}\\ V^{\pi }(s)& ={\mathbb{E}}_{a\sim \pi (\cdot \mid s)}\left[r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,a)}\left[V^{\pi }(s^{\prime })\right]\right]\text{(V-V)}\\ Q^{\pi }(s,a)& =r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,a)}\left[{\mathbb{E}}_{a^{\prime }\sim \pi (\cdot \mid s^{\prime })}\left[Q^{\pi }(s^{\prime },a^{\prime })\right]\right]\text{(Q-Q)}\end{array}$$

1. 引入操作符$P^{\pi }$简写(Q-Q)式：$Q^{\pi }=r+\gamma P^{\pi }{Q}^{\pi }$，这可从矩阵形式进行理解，其中矩阵每一项的具体值为$P_{(s,a,s^{\prime },a^{\prime })}^{\pi }=p(s^{\prime }|s,a)\pi (a^{\prime }|s^{\prime })$$$Q^{\pi }(\in {\mathbb{R}}^{|S||A|\times 1})=r(\in {\mathbb{R}}^{|S||A|\times 1})+\gamma P^{\pi }(\in {\mathbb{R}}^{|S||A|\times |S||A|})Q^{\pi }(\in {\mathbb{R}}^{|S||A|\times 1})$$
2. 同理，引入操作符$P$简写(Q-V)式：$Q^{\pi }=r+\gamma P{V}^{\pi }$
3. 因此基于consistency的Q函数有closed-form的表述$Q^{\pi }=(I-\gamma P^{\pi }{)}^{-1}r$
4. $Q^{\pi }$对于其空间内的每一个点$(s,a)$定义为$$Q^{\pi }(s,a)=\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(s_t,a_t)|s_0=s,a_0=a\right]$$
5. 因此便可直接推出关于$(I-\gamma P^{\pi }{)}^{-1}$的引理，即该可逆矩阵其中每一项为（$\mathrm{Pr}$代表从$s_0=s,a_0=a$出发在转移矩阵和策略的作用下，在t时刻遇到$(s^{\prime },a^{\prime })$的概率）
   $$(I-\gamma P^{\pi }{)}_{(s,a),(s^{\prime },a^{\prime })}^{-1}=\sum _{t=0}^{\infty }{\gamma }^t\mathrm{Pr}\left(s_t=s^{\prime },a_t=a^{\prime }|s_0=s,a_0=a\right)\le \frac{1}{1-\gamma }$$
6. 在前面知道这玩意是可逆的即$$\begin{array}{rl}||(I-\gamma P^{\pi })x|{|}_{\infty }& =||x-\gamma P^{\pi }x|{|}_{\infty }\\ & \ge ||x|{|}_{\infty }-||\gamma P^{\pi }x|{|}_{\infty }\text{(两者差的最大值>= 两者最大值之差)}\\ & \ge ||x|{|}_{\infty }-\gamma ||x|{|}_{\infty }\text{（Pπ转移矩阵的定义)}\\ & =(1-\gamma )||x|{|}_{\infty }>0\end{array}$$
7. 所以易得不等式$$\Vert (I-\gamma P^{\pi }{)}^{-1}v{\Vert }_{\infty }\le \frac{\Vert v{\Vert }_{\infty }}{1-\gamma }$$

要知道这小节全都是根据定义推出的，即Bellman consistency在所有setting下均适用，没有deterministic policy的假设，得与Bellman optimality进行区分

2.2 Simulation Lemma

基于2.1 Bellman Consistency中介绍的操作符$P^{\pi }$，对于任意的策略$\pi$有如下等式
$$\begin{array}{rl}{Q}^{\pi }-{\hat{Q}}^{\pi }& ={\left(I-\gamma P^{\pi }\right)}^{-1}r-{\left(I-\gamma {\hat{P}}^{\pi }\right)}^{-1}r\\ & ={\left(I-\gamma {\hat{P}}^{\pi }\right)}^{-1}\left((I-\gamma {\hat{P}}^{\pi })(I-\gamma P^{\pi }{)}^{-1}-I\right)r\\ & ={\left(I-\gamma {\hat{P}}^{\pi }\right)}^{-1}\left((I-\gamma {\hat{P}}^{\pi })-(I-\gamma P^{\pi })\right)(I-\gamma P^{\pi }{)}^{-1}r\\ & ={\left(I-\gamma {\hat{P}}^{\pi }\right)}^{-1}\left(\left(I-\gamma {\hat{P}}^{\pi }\right)-\left(I-\gamma P^{\pi }\right)\right)Q^{\pi }\\ & =\gamma {\left(I-\gamma {\hat{P}}^{\pi }\right)}^{-1}\left(P^{\pi }-{\hat{P}}^{\pi }\right)Q^{\pi }\\ & =\gamma {\left(I-\gamma {\hat{P}}^{\pi }\right)}^{-1}(P-\hat{P})V^{\pi }\end{array}$$

$$\begin{array}{rl}{Q}^{\pi }-{\hat{Q}}^{\pi }& ={\left(I-\gamma P^{\pi }\right)}^{-1}r-(I-\gamma {\hat{P}}^{\pi }{)}^{-1}r\\ & =(I-\gamma P^{\pi }{)}^{-1}\left(I-(I-\gamma P^{\pi })(I-\gamma {\hat{P}}^{\pi }{)}^{-1}\right)r\\ & =(I-\gamma P^{\pi }{)}^{-1}\left((I-\gamma {\hat{P}}^{\pi })-(I-\gamma P^{\pi })\right)(I-\gamma {\hat{P}}^{\pi }{)}^{-1}r\\ & =\gamma (I-\gamma P^{\pi }{)}^{-1}(P^{\pi }-{\hat{P}}^{\pi }){\hat{Q}}^{\pi }\\ & =\gamma (I-\gamma P^{\pi }{)}^{-1}(P-\hat{P}){\hat{V}}^{\pi }\end{array}$$

特点是策略相同，Q不同；而Q的不同之处在于P还是$\hat{P}$，只与simulator有关，所以称为Simulation Lemma

2.3 Component-wise Bound

这是个著名的上下界，直接操作的对象是我们一筹莫展的$\Vert Q^{\star }-{\hat{Q}}^{\star }{\Vert }_{\infty }$

1. 上界
   $$\begin{array}{rl}{Q}^{\star }-{\hat{Q}}^{\star }& =Q^{{\pi }^{\star }}-{\hat{Q}}^{{\hat{\pi }}^{\star }}\le Q^{{\pi }^{\star }}-{\hat{Q}}^{{\pi }^{\star }}\text{(换成相同策略)}\\ & =\gamma {\left(I-\gamma {\hat{P}}^{{\pi }^{\star }}\right)}^{-1}(P-\hat{P})V^{{\pi }^{\star }}\text{(first simulation lemma)}\\ & =\gamma {\left(I-\gamma P^{{\pi }^{\star }}\right)}^{-1}(P-\hat{P}){\hat{V}}^{{\pi }^{\star }}\text{(second simulation leamma)}\end{array}$$
2. 下界
   $$\begin{array}{rl}{Q}^{\star }-{\hat{Q}}^{\star }& =Q^{{\pi }^{\star }}-{\hat{Q}}^{{\hat{\pi }}^{\star }}\\ & =(I-\gamma P^{{\pi }^{\star }}{)}^{-1}r-(I-\gamma {\hat{P}}^{{\hat{\pi }}^{\star }}{)}^{-1}r\\ & =(I-\gamma {\hat{P}}^{{\hat{\pi }}^{\star }}{)}^{-1}\left((I-\gamma {\hat{P}}^{{\hat{\pi }}^{\star }})(I-\gamma P^{{\pi }^{\star }}{)}^{-1}-I\right)r\\ & =(I-\gamma {\hat{P}}^{{\hat{\pi }}^{\star }}{)}^{-1}\left((I-\gamma {\hat{P}}^{{\hat{\pi }}^{\star }})-(I-\gamma P^{{\pi }^{\star }})\right)(I-\gamma P^{{\pi }^{\star }}{)}^{-1}r\\ & =\gamma (I-\gamma {\hat{P}}^{{\hat{\pi }}^{\star }}{)}^{-1}\left(P^{{\pi }^{\star }}-{\hat{P}}^{{\hat{\pi }}^{\star }}\right)Q^{\star }\\ & \ge \gamma (I-\gamma {\hat{P}}^{{\hat{\pi }}^{\star }}{)}^{-1}\left(P^{{\pi }^{\star }}-{\hat{P}}^{{\pi }^{\star }}\right)Q^{\star }\text{ (⋆)}\\ & =\gamma (I-\gamma {\hat{P}}^{{\hat{\pi }}^{\star }}{)}^{-1}\left(P-\hat{P}\right)V^{\star }\end{array}$$

因此Componet-wise bound为$$\gamma (I-\gamma {\hat{P}}^{{\hat{\pi }}^{\star }}{)}^{-1}\left(P-\hat{P}\right)V^{\star }\le Q^{\star }-{\hat{Q}}^{\star }\le \gamma {\left(I-\gamma {\hat{P}}^{{\pi }^{\star }}\right)}^{-1}(P-\hat{P})V^{\star }$$

Component-wise的bound无非是在simulation lemma的证明基础上，使用了两个小不等式：上界使用了$Q^{{\pi }^{\star }}-{\hat{Q}}^{{\hat{\pi }}^{\star }}\le Q^{{\pi }^{\star }}-{\hat{Q}}^{{\pi }^{\star }}$，下界在$(\star )$处使用了${\hat{P}}^{{\hat{\pi }}^{\star }}{Q}^{\star }\le {\hat{P}}^{{\pi }^{\star }}{Q}^{\star }$

四、在相关公式下的目标解析

1. 使用component wide bound，有两种关于目标$Q^{\star }-{\hat{Q}}^{\star }$的上界即$$\begin{array}{rl}& Q^{\star }-{\hat{Q}}^{\star }\le \gamma {\left(I-\gamma {\hat{P}}^{{\pi }^{\star }}\right)}^{-1}(P-\hat{P})V^{{\pi }^{\star }}\\ & Q^{\star }-{\hat{Q}}^{\star }\le \gamma {\left(I-\gamma P^{{\pi }^{\star }}\right)}^{-1}(P-\hat{P}){\hat{V}}^{{\pi }^{\star }}\end{array}$$
2. 接下来问题是，究竟哪个比较好bound住？
   • 是否有相关公式来表述$(P-\hat{P})V^{{\pi }^{\star }}$或$(P-\hat{P}){\hat{V}}^{{\pi }^{\star }}$
   • 是否有相关公式来表述${\left(I-\gamma {\hat{P}}^{{\pi }^{\star }}\right)}^{-1}$或${\left(I-\gamma P^{{\pi }^{\star }}\right)}^{-1}$
   • 怎么以一定的概率来bound住这些量？
3. 且听下回分解