每天一个RL基础理论(1)——Bellman Optimality

搬砖来源： https://wensun.github.io/CS6789_fall_2021.html

本系列尽量统一用下列术语表述。

一、Infinite horizon discounted MDPs

符号	含义
$S$	状态空间
$A$	动作空间
$P:S\times A\to △(S)$	$△(S)$代表有效的状态空间，符合环境规律的转移映射$P$
$r:S\times A\to [0,1]$	状态和动作 到 [0,1]映射的奖励函数$r$
$\gamma \in [0,1)$	discount fator
$\mathcal{M}=(S,A,P,r,\gamma )$	MDP
$\pi :S\to △(A)$	从状态空间到有效动作空间的策略映射$\pi$
$V^{\star }(s)$	最优策略${\pi }^{\star }$在状态结果为s下所对应的价值
$Q^{\star }(s,a)$	最优策略${\pi }^{\star }$在状态结果为s，动作结果为a下所对应的价值

• Infinite horizon：无限长的序列，用$H\to \infty$来表示
• trajectory： τ = { s 0 , a 0 , . . . , s h , a h } \tau=\{s_0,a_0,..., s_h,a_h\} τ={s0​,a0​,...,sh​,ah​}，其中$s_h$为第h时刻的状态变量，$a_h=a$意味着第h时刻的动作变量的结果为a

所以基于该MDP的setting下有：

1. value function : 表示初始时刻状态变量的取值为$s$的价值
   $$V^{\pi }(s)={\mathbb{E}}_{s_0=s,a_h\sim \pi (\cdot \mid s_h),s_{h+1}\sim p(\cdot \mid s_h,a_h)}\left[\sum _{h=0}^{\infty }{\gamma }^{h}r(s_h,a_h)\right]$$
2. Q function：表示初始时刻状态变量的取值为$s$，动作变量取值为a的价值
   $$Q^{\pi }(s,a)={\mathbb{E}}_{s_0=s,a_0=a,s_{h+1}\sim p(\cdot \mid s_h,a_h),a_{h+1}\sim \pi (\cdot \mid s_{h+1})}\left[\sum _{h=0}^{\infty }{\gamma }^{h}r(s_h,a_h)\right]$$
3. Bellman Consistency Equation：value function 和 Q function之间的迭代关系

$$\begin{array}{rl}{V}^{\pi }(s)& ={\mathbb{E}}_{a\sim \pi (\cdot \mid s)}\left[Q^{\pi }(s,a)\right]\text{(V-Q)}\\ Q^{\pi }(s,a)& =r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,a)}\left[V^{\pi }(s^{\prime })\right]\text{(Q-V)}\\ V^{\pi }(s)& ={\mathbb{E}}_{a\sim \pi (\cdot \mid s)}\left[r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,a)}\left[V^{\pi }(s^{\prime })\right]\right]\text{(V-V)}\\ Q^{\pi }(s,a)& =r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,a)}\left[{\mathbb{E}}_{a^{\prime }\sim \pi (\cdot \mid s^{\prime })}\left[Q^{\pi }(s^{\prime },a^{\prime })\right]\right]\text{(Q-Q)}\end{array}$$

二、Bellman Optimality

（证明见第四部分）

• 性质一：在infinite horizon discounted的MDP下，存在一个deterministic且stationary的最优策略${\pi }^{\star }$，使得$V^{{\pi }^{\star }}\ge V^{\pi }(s),\forall s,\pi$ ，且该optimal policy ${\pi }^{\star }$的value function为$V^{{\pi }^{\star }}$，简记为$V^{\star }$，满足如下性质：
  $$\begin{gathered} V^{\star }(s)=\underset{a}{\max }\left[r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim P(\cdot \mid s,a)}[V^{\star }(s^{\prime })]\right] \\ {Q}^{\star }(s,a)=r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot |s,a)}\left[\underset{a^{\prime }}{\max }{Q}^{\star }(s^{\prime },a^{\prime })\right] \end{gathered}$$
• 性质二：(V-version）对于任意的价值函数$V$，如果其满足$V(s)={\max }_a\left[r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim P(\cdot |s,a)}V(s^{\prime })\right],\forall s$，则有：
  $$V(s)=V^{\star }(s)$$同理有Q-version：
  对于任意的Q值函数$Q$，如果其满足$Q(s,a)=r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot |s,a)}\left[{\max }_{a^{\prime }}Q(s^{\prime },a^{\prime })\right],\forall s$，则有：
  $$Q(s,a)=Q^{\star }(s,a)$$

1. stationary的理解：在时间序列中，基于历史和现状去预测未来的前提是，未来 与 历史和现状存在一定的可延续性。更严谨的说，历史和当前时刻的状态变量$s_h$有一些基本特性要在未来的一定时刻下$s_{h+k}$保持不变。刻画这些变量的统计量为weak stationarity，而刻画这些变量的联合分布为strong stationarity。拓展延伸见如何理解时间序列的平稳性
2. 从符号来严格定义non-stationary的policy可描述为：$\pi (a|s,t)$，与时间有关；$\pi (a|s)$则意味这是稳态分布，$\pi (s)$则意味着deterministic policy
3. deterministic stationary optimal policy在infinite horizon discounted的MDP下optimal policy的存在性证明

三、Trajectory distribution & State-action distribution

• （trajectory distribution) 在初始状态为$s_0$，策略$\pi$下生成一条长度为h的trajectory τ h = { s 0 , a 0 , . . . , s h , a h } \tau_h=\{s_0,a_0,..., s_h,a_h\} τh​={s0​,a0​,...,sh​,ah​}的概率为
  $${\mathbb{P}}_{s_0}^{\pi }({\tau }_h)=\pi (a_0|s_0)\prod _{t=0}^{h}p(s_{t+1}\mid s_t,a_t)\pi (a_{t+1}\mid s_{t+1})$$
• 在初始状态为$s_0$, 策略$\pi$下在第h时刻访问到结果为$s,a$的概率为
  $${\mathbb{P}}_{s_0}^{\pi }({\tau }_{h-1},s_h=s,a_h=a)=\sum _{a_0,s_1,a_1,...,s_{h-1},a_{h-1}\in {\tau }_{h-1}}\mathbb{P}({\tau }_{h-1},s_0)p(s_t=s|s_{t-1},a_{t-1})\pi (a_t=a|s)$$(将h-1之前时刻的所有可能结果加起来）
• (state-action distribution) 在初始状态为$s_0$，策略$\pi$下，访问到结果为s,a的概率为 :
  $$d_{s_0}^{\pi }(s,a)=(1-\gamma )\sum _{h=0}^{\infty }{\gamma }^h{\mathbb{P}}_{s_0}^{\pi }({\tau }_{h-1},s_h=s,a_h=a)$$(在所有时刻访问到的概率加起来)

  其中$1-\gamma$是用来normalized的，使得$d_{s_0}^{\pi }(s,a)$为概率，即加起来为1

• 以state-action distribution的形式来表达V函数，有：
  $$V^{\pi }(s_0)=\frac{1}{1-\gamma }\sum _{s,a}{d}_{s_0}^{\pi }(s,a)r(s,a)$$(遍历所有的可能结果s,a)

四、QA和补充

Q1：为什么discount factor即$\gamma$限定的范围是$[0,1)$，为什么不要1呢？

看Q2，然后思考一下，如果$\gamma \ge 1$又会如何？

Q2：为什么reward要映射到$[0,1]$，这有什么用？

实际的reward signal都能通过reward shaping映射到该区间中，好处是我们能因此bound住value function 和 Q-function便于分析问题，即

$$\begin{array}{rl}0\le V^{\pi }(s)& ={\mathbb{E}}_{s_0=s,a_h\sim \pi (\cdot \mid s_h),s_{h+1}\sim p(\cdot \mid s_h,a_h)}\left[\sum _{h=0}^{\infty }{\gamma }^{h}r(s_h,a_h)\right]\\ & \le \sum _{h=0}^{\infty }{\gamma }^h=\frac{1}{1-\gamma }\end{array}$$

同理有：
$$0\le Q^{\pi }(s,a)\le \frac{1}{1-\gamma }$$

4.1 Infinite horizon discounted的MDP下deterministic stationary最优策略的存在性证明

• 存在性的证明思路：对于策略集合$\Pi$中的$\pi$ ($s_h,a_h$均为变量，其余为变量的取值，即结果）
  1. 首先证明：在初始时刻的结果为s,a,r, 第一时刻的状态结果为$s^{\prime }$时，其最大化future discounted value有如下性质：
     $$\underset{\pi \in \Pi }{\max }{\mathbb{E}}_{s_0=s,a_0=a,s_1=s^{\prime },a_h\sim \pi (\cdot \mid s_h),s_{h+1}\sim p(\cdot \mid s_h,a_h)}\left[\sum _{h=1}^{\infty }{\gamma }^{h}r(s_h,a_h)\right]=\gamma V^{\star }(s^{\prime })$$(这玩意说明了在一个策略集合$\pi$下，给定初始结果为s,a,r,s’时，最优策略的表现仅与下一状态的结果$s^{\prime }$有关，根据Markov Property能直觉地理解）
  2. 接着构造一个deterministic stationary的policy $\tilde{\pi }(s)$如下：
     $$\tilde{\pi }(s)=\underset{a\in A}{\mathrm{argmax}}{\mathbb{E}}_{s_0=s,a_0=a}\left[r(s,a)+\gamma V^{\star }(s_1)\right]$$
  3. 最后证明构造的$\tilde{\pi }(s)$为optimal policy，即：
     $$V^{\tilde{\pi }(s)}=V^{\star }(s)$$
• 证1:
  令$h^{\prime }=h-1$，则有：
  $$\begin{array}{rl}& \underset{\pi \in \Pi }{\max }{\mathbb{E}}_{s_0=s,a_0=a,s_1=s^{\prime },a_h\sim \pi (\cdot \mid s_h),s_{h+1}\sim p(\cdot \mid s_h,a_h)}\left[\sum _{h=1}^{\infty }{\gamma }^{h}r(s_h,a_h)\right]\\ & =\underset{\pi \in \Pi }{\max }{\mathbb{E}}_{s_0=s,a_0=a,s_1=s^{\prime },a_h\sim \pi (\cdot \mid s_h),s_{h+1}\sim p(\cdot \mid s_h,a_h)}\left[\sum _{h=1}^{\infty }{\gamma }^{h}r(s_h,a_h)+r(s_0,a_0)\right]\\ & =\underset{\pi \in \Pi }{\max }{\mathbb{E}}_{s_0=s^{\prime },a_{h^{\prime }}\sim \pi (\cdot \mid s_{h^{\prime }}),s_{h^{\prime }+1}\sim p(\cdot \mid s_{h^{\prime }},a_{h^{\prime }})}\left[\sum _{h^{\prime }=0}^{\infty }{\gamma }^{h^{\prime }+1}r(a_{h^{\prime }},s_{h^{\prime }})\right]\\ & =\underset{\pi \in \Pi }{\max }\gamma {\mathbb{E}}_{s_0=s^{\prime },a_{h^{\prime }}\sim \pi (\cdot \mid s_{h^{\prime }}),s_{h^{\prime }+1}\sim p(\cdot \mid s_{h^{\prime }},a_{h^{\prime }})}\left[\sum _{h^{\prime }=0}^{\infty }{\gamma }^{h^{\prime }}r(a_{h^{\prime }},s_{h^{\prime }})\right]\\ & =\underset{\pi \in \Pi }{\max }\gamma V^{\pi }(s^{\prime })\end{array}$$所以$$\begin{array}{rl}& \underset{\pi \in \Pi }{\max }{\mathbb{E}}_{s_0=s,a_0=a,s_1=s^{\prime },a_h\sim \pi (\cdot \mid s_h),s_{h+1}\sim p(\cdot \mid s_h,a_h)}\left[\sum _{h=1}^{\infty }{\gamma }^{h}r(s_h,a_h)\right]\\ & =\underset{\pi \in \Pi }{\max }\gamma V^{\pi }(s^{\prime })\\ & =\gamma V^{\star }(s^{\prime })\end{array}$$
• 证2:
  首先根据定义一定有：$$V^{\tilde{\pi }}(s)\le V^{\star }(s)\forall s\in S$$然后对于第0时刻的状态变量，根据定义有
  $$\begin{array}{rl}{V}^{\star }(s_0)& =\underset{\pi \in \Pi }{\max }{\mathbb{E}}_{a_h\sim \pi (\cdot \mid s_h),s_{h+1}\sim p(\cdot \mid s_h,a_h)}\left[r(s_0,a_0)+\sum _{h=1}^{\infty }{\gamma }^{h}r(s_h,a_h)\right]\\ & =\underset{\pi \in \Pi }{\max }{\mathbb{E}}_{a_0\sim \pi (\cdot \mid s_0)}\left[r(s_0,a_0)+{\mathbb{E}}_{s_h\sim p(\cdot \mid s_0,a_0),a_h\sim \pi (\cdot |s_h)}\left[\sum _{h=1}^{\infty }{\gamma }^{h}r(s_h,a_h)\right]\right]\\ & \le \underset{\pi \in \Pi }{\max }{\mathbb{E}}_{a_0\sim \pi (\cdot \mid s_0)}\left[r(s_0,a_0)+\underset{{\pi }^{\prime }\in \Pi }{\max }{\mathbb{E}}_{s_h\sim p(\cdot \mid s_0,a_0),a_h\sim {\pi }^{\prime }(\cdot |s_h)}\left[\sum _{h=1}^{\infty }{\gamma }^{h}r(s_h,a_h)\right]\right]\\ & =\underset{\pi \in \Pi }{\max }{\mathbb{E}}_{a_0\sim \pi (\cdot \mid s_0),s_1\sim p(\cdot \mid s_0,a_0)}\left[r(s_0,a_0)+\gamma V^{\star }(s_1)\right]\text{(利用证明1)}\\ & =\underset{a_0\in A}{\max }{\mathbb{E}}_{a_0\sim \pi (\cdot \mid s_0),s_1\sim p(\cdot \mid s_0,a_0)}\left[r(s_0,a_0)+\gamma V^{\star }(s_1)\right]\\ & =V^{\tilde{\pi }}(s_0)\text{(等于构造的策略所选的动作)}\end{array}$$所以
  $$V^{\star }(s_0)\le V^{\tilde{\pi }}(s_0),\forall s_0\in S$$

因此构造的策略是optimal policy。

• deterministic 体现在构造时选择的动作是argmax的，是一个动作点
• stationary policy 体现在证明1中的change of variables即$h^{\prime }=h-1$时，关于时间的变量h之间是存在可延续性的，即未来时刻的变量与当前时刻的变量之间存在一些基本特性的联系

4.2 性质一的证明

• 证明$V^{\star }(s)={\max }_a\left[r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,a)}[V^{\star }(s^{\prime })]\right]$的思路为通过不等式放缩进行夹逼，相关展开基于Bellman equation
• 分析${\max }_a\left[r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,a)}[V^{\star }(s^{\prime })]\right]={\max }_a{Q}^{\star }(s,a)$，即证$V^{\star }(s)={\max }_a{Q}^{\star }(s,a)$，等价于说策略$\hat{\pi }(s)=\underset{a}{\mathrm{argmax}}{Q}^{\star }(s,a)$是最优策略${\pi }^{\star }$，所以只需要证明$V^{\hat{\pi }(s)}=V^{\star }(s)$ 或 $\hat{\pi }(s)={\pi }^{\star }$
• 已知$V^{\hat{\pi }(s)}\le V^{\star }(s)$，只需证$V^{\hat{\pi }(s)}\ge V^{\star }(s)$，即可证得$V^{\hat{\pi }(s)}=V^{\star }(s)$，所以有$V^{\star }(s)={\max }_a\left[r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,a)}[V^{\star }(s^{\prime })]\right]$

以下证：$V^{\star }(s)\le V^{\hat{\pi }(s)}$
$$\begin{array}{rl}{V}^{\star }(s)& =r(s,{\pi }^{\star }(s))+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,{\pi }^{\star }(s))}\left[V^{\star }(s^{\prime })\right]\\ & \le \underset{a}{\max }\left[r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,a)}[V^{\star }(s^{\prime })]\right]\\ & =\underset{a}{\max }{Q}^{\star }(s,a)\\ & =r(s,\hat{\pi }(s))+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,\hat{\pi }(s))}[V^{\star }(s^{\prime })]\text{ (1)}\end{array}$$是不是感觉证完了？(1)式不正是$V^{\hat{\pi }(s)}$嘛？
可惜还真不是，根据value function的定义$V^{\hat{\pi }(s)}=r(s,\hat{\pi }(s))+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,\hat{\pi }(s))}[V^{\hat{\pi }(s)}(s^{\prime })]$

但可以对已有的不等式$V^{\star }(s)\le r(s,\hat{\pi }(s))+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,\hat{\pi }(s))}[V^{\star }(s^{\prime })]$进行套娃:
$$\begin{array}{rl}{V}^{\star }(s)& \le r(s,\hat{\pi }(s))+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,\hat{\pi }(s))}[V^{\star }(s^{\prime })]\\ & \le r(s,\hat{\pi }(s))+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,\hat{\pi }(s))}\left[r(s^{\prime },\hat{\pi }(s^{\prime }))+\gamma {\mathbb{E}}_{s^{\prime \prime }\sim p(\cdot \mid s^{\prime },\hat{\pi }(s^{\prime }))}[V^{\star }(s^{\prime \prime })]\right]\\ & \le r(s,\hat{\pi }(s))+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,\hat{\pi }(s))}\left[r(s^{\prime },\hat{\pi }(s^{\prime }))+\gamma {\mathbb{E}}_{s^{\prime \prime }\sim p(\cdot \mid s^{\prime },\hat{\pi }(s^{\prime }))}\left[r(s^{\prime \prime },\hat{\pi }(s^{\prime \prime }))+\gamma {\mathbb{E}}_{s^{\prime \prime \prime }\sim p(\cdot \mid s^{\prime \prime },\hat{\pi }(s^{\prime \prime }))}[V^{\star }(s^{\prime \prime \prime })]\right]\right]\\ & \cdots \\ & =V^{\hat{\pi }(s)}\end{array}$$

性质一本质上说明了这个deterministic stationary的optimal policy 则为${\pi }^{\star }(s)=\underset{a}{\mathrm{argmax}}{Q}^{\star }(s,a)$

4.3 性质二的证明

• 证明对于任意的价值函数$V$，如果其满足$V(s)={\max }_a\left[r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot |s,a)}V(s^{\prime })\right],\forall s$，则有$V(s)=V^{\star }(s)$的思路为对$|V(s)-V^{\star }(s)|$进行放缩，小于等于一个为0的项，展开基于Bellman equation

证：（第一步利用了性质一即$V^{\star }(s)={\max }_a\left[r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,a)}[V^{\star }(s^{\prime })]\right]$)
$$\begin{array}{rl}|V(s)-V^{\star }(s)|& =\left|\underset{a}{\max }\left[r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot |s,a)}V(s^{\prime })\right]-\underset{a}{\max }\left[r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,a)}{V}^{\star }(s^{\prime })\right]\right|\\ & \le \underset{a}{\max }\left|\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,a)}[V(s^{\prime })-V^{\star }(s^{\prime })]\right|\\ & \le \underset{a}{\max }\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,a)}\left|V(s^{\prime })-V^{\star }(s^{\prime })\right|\\ & \le \underset{a}{\max }\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot \mid s,a)}\left|\underset{a^{\prime }}{\max }\gamma {\mathbb{E}}_{s^{\prime \prime }\sim p(\cdot \mid s^{\prime },a^{\prime })}\left|V(s^{\prime \prime })-V^{\star }(s^{\prime \prime })\right|\right|\\ & \le \underset{a_1,...,a_k}{\max }{\gamma }^k{\mathbb{E}}_{s^k\sim p(\cdot \mid s_{k-1},a_{k-1})}\left[|V(s_k)-V^{\star }(s_k)|\right]\\ & \le \underset{a_1,...,a_k}{\max }{\gamma }^k\times \frac{2}{1-\gamma }\text{( ∥V∥≤1−γ1)}\\ & \le \underset{k\to \infty }{\lim }\underset{a_1,...,a_k}{\max }\frac{2{\gamma }^k}{1-\gamma }=0\end{array}$$

五、总结

1. 理解策略的 平稳性 和 确定性
2. 理解infinite horizon discounted这个setting
3. 在该MDP setting下存在一个deterministic stationary的optimal policy会满足两个性质，称为Bellman Optimality
4. 所以如果在该setting下，有满足性质的V和Q，那么它就是determinsitc stationary optimal policy的V和Q