CS6789-4(下)
- 搬砖来源:https://wensun.github.io/CS6789_fall_2021.html
- 细节来源:Model-Based Reinforcement Learning with a Generative Model is Minimax Optimal 2020 JMLR
- 主题:经典方法VI&PI在转移矩阵未知情况下,使用经典算法VI&PI达到near-optimal policy所需的样本复杂度
- setting:infinite horizon discounted MDP / unknown transition probability / deterministic reward / deterministic policy
- 解决的问题:给定 M = ( S , A , P , r , γ ) \mathcal M=(S,A,P,r,\gamma) M=(S,A,P,r,γ),其中转移矩阵 P P P未知,经典方法VI&PI需要多少的样本(transitions ( s , a , r , s ′ ) (s,a,r,s') (s,a,r,s′))才能学习到near optimal的策略(deterministic & stationary)?
- 使用的理论工具:Statistics Theory
前情提要
Q ⋆ − Q ^ ⋆ ≤ γ ( I − γ P ^ π ⋆ ) − 1 ( P − P ^ ) V π ⋆ Q ⋆ − Q ^ ⋆ ≤ γ ( I − γ P π ⋆ ) − 1 ( P − P ^ ) V ^ π ⋆ \begin{aligned} &Q^\star-\widehat{Q}^\star\leq\gamma\left(I-\gamma \widehat{P}^{\pi^\star}\right)^{-1}(P-\widehat{P}) V^{\pi^\star}\\ &Q^\star-\widehat{Q}^\star\leq\gamma\left(I-\gamma P^{\pi^\star}\right)^{-1}(P-\widehat P)\widehat V^{\pi^\star} \end{aligned} Q⋆−Q ⋆≤γ(I−γP π⋆)−1(P−P )Vπ⋆Q⋆−Q ⋆≤γ(I−γPπ⋆)−1(P−P )V π⋆
- 第一个问题:寻找与 ( P − P ^ ) V π ⋆ (P-\widehat{P}) V^{\pi^\star} (P−P )Vπ⋆或 ( P − P ^ ) V ^ π ⋆ (P-\widehat P)\widehat V^{\pi^\star} (P−P )V π⋆相关的公式进行上界放缩
- 第二个问题:寻找 ( I − γ P ^ π ⋆ ) − 1 \left(I-\gamma \widehat{P}^{\pi^\star}\right)^{-1} (I−γP π⋆)−1或 ( I − γ P π ⋆ ) − 1 \left(I-\gamma P^{\pi^\star}\right)^{-1} (I−γPπ⋆)−1相关的公式进行上界放缩
一、介绍这两个问题相关的公式
1.1 Discounted Bellman-variance Equation
学习并理解一个基础公式,称为Bellman-variance equation: Σ M π = γ 2 ( Var P ( Q M π ) + P π Σ M π ) \Sigma_M^\pi=\gamma^2\Big(\text{Var}_P(Q_M^\pi)+P^\pi\Sigma_M^\pi\Big) ΣMπ=γ2(VarP(QMπ)+PπΣMπ)
- 定义1:在某个MDP即
M
=
(
S
,
A
,
P
,
r
,
γ
)
\mathcal M=(S,A,P,r,\gamma)
M=(S,A,P,r,γ)上,对于一个stationary的策略
π
\pi
π,定义variance of discounted cumulated reward为
Σ M π ( s , a ) = E π , P [ ( ∑ t = 0 ∞ γ t r ( s t , a t ) − Q M π ( s , a ) ) 2 ∣ s 0 = s , a 0 = a ] \Sigma_M^\pi(s,a)=\mathbb E_{\pi,P}\left[\left(\sum_{t=0}^\infty\gamma^tr(s_t,a_t)-Q^\pi_M(s,a)\right)^2\Big|s_0=s,a_0=a\right] ΣMπ(s,a)=Eπ,P⎣⎡(t=0∑∞γtr(st,at)−QMπ(s,a))2∣∣∣s0=s,a0=a⎦⎤ Q M π ( s , a ) = E π , P [ ∑ h = 0 ∞ γ h r ( s h , a h ) ∣ s 0 = s , a 0 = a ] Q^\pi_M(s,a)=\mathbb E_{\pi,P}\left[\sum_{h=0}^\infty\gamma^h r(s_h,a_h)\big|s_0=s,a_0=a\right] QMπ(s,a)=Eπ,P[h=0∑∞γhr(sh,ah)∣∣s0=s,a0=a]
- 本质上来看, Σ M π ( s , a ) \Sigma_M^\pi(s,a) ΣMπ(s,a)的定义是trajectory distribution与state-action distribution的交锋。
- Q π Q^\pi Qπ是一个以 ( s , a ) (s,a) (s,a)开始累积reward的期望,是所有轨迹的平均值,是一个点; ∑ t = 0 ∞ γ t r ( s t , a t ) \sum_{t=0}^\infty\gamma^tr(s_t,a_t) ∑t=0∞γtr(st,at)是具体的以 ( s , a ) (s,a) (s,a)开始一条轨迹的累积reward,一条轨迹一个点,有很多点
- Σ M π ( s , a ) \Sigma_M^\pi(s,a) ΣMπ(s,a)这个variance衡量的是策略 π \pi π与转移矩阵的randomness,如果 π \pi π很确定,那么randomness大部分来自环境,这时若 Σ M π ( s , a ) \Sigma_M^\pi(s,a) ΣMπ(s,a)此时很大,意味着从当前的(s,a)出发,环境的随机性很大。如果转移矩阵 P P P很确定,那么randomness大部分来自策略,这时若 Σ M π ( s , a ) \Sigma_M^\pi(s,a) ΣMπ(s,a)此时很大,意味着从当前的(s,a)出发,策略的随机性很大
- 定义2:在某个MDP即
M
=
(
S
,
A
,
P
,
r
,
γ
)
\mathcal M=(S,A,P,r,\gamma)
M=(S,A,P,r,γ)上,对于一个stationary的策略
π
\pi
π,定义关于映射
V
V
V,经过一步转移
s
′
∼
p
(
⋅
∣
s
,
a
)
s'\sim p(\cdot\mid s,a)
s′∼p(⋅∣s,a)的方差
Var P ( V M π ) ( s , a ) = E s ′ ∼ p ( ⋅ ∣ s , a ) [ ( V M π ( s ′ ) − E s ′ ′ ∼ p ( ⋅ ∣ s , a ) [ V M π ( s ′ ′ ) ] ) 2 ] \text{Var}_P(V_M^\pi)(s,a)=\mathbb E_{s'\sim p(\cdot\mid s,a)}\left[\left(V_M^\pi(s')-\mathbb E_{s''\sim p(\cdot\mid s,a)}\left[V_M^\pi(s'')\right]\right)^2\right] VarP(VMπ)(s,a)=Es′∼p(⋅∣s,a)[(VMπ(s′)−Es′′∼p(⋅∣s,a)[VMπ(s′′)])2]
同理有关于映射 Q Q Q,经过一步转移后的累积方差
Var P ( Q M π ) ( s , a ) = E s ′ ∼ p ( ⋅ ∣ s , a ) , a ′ ∼ π ( ⋅ ∣ s ′ ) [ ( Q M π ( s ′ , a ′ ) − E s ′ ′ ∼ p ( ⋅ ∣ s , a ) , a ′ ′ ∼ π ( ⋅ ∣ s ′ ′ ) [ Q M π ( s ′ ′ , a ′ ′ ) ] ) 2 ] \text{Var}_P(Q_M^\pi)(s,a)=\mathbb E_{s'\sim p(\cdot\mid s,a),a'\sim\pi(\cdot|s')}\left[\left(Q_M^\pi(s',a')-\mathbb E_{s''\sim p(\cdot\mid s,a),a''\sim\pi(\cdot|s'')}\left[Q_M^\pi(s'',a'')\right]\right)^2\right] VarP(QMπ)(s,a)=Es′∼p(⋅∣s,a),a′∼π(⋅∣s′)[(QMπ(s′,a′)−Es′′∼p(⋅∣s,a),a′′∼π(⋅∣s′′)[QMπ(s′′,a′′)])2]且这两者实际上是恒等的,即 Var P ( V M π ) ( s , a ) = Var P ( Q M π ) ( s , a ) \text{Var}_P(V_M^\pi)(s,a)=\text{Var}_P(Q_M^\pi)(s,a) VarP(VMπ)(s,a)=VarP(QMπ)(s,a)
利用矩阵操作符更进一步地简写关于V和Q映射的方差,操作的点是
(
s
,
a
)
(s,a)
(s,a)所处空间:(利用方差公式
Var
X
=
E
[
X
2
]
−
(
E
[
X
]
)
2
\text{Var}X=\mathbb E[X^2]-(\mathbb E[X])^2
VarX=E[X2]−(E[X])2)
Var
P
(
V
M
π
)
=
P
(
V
M
π
)
2
−
(
P
V
M
π
)
2
\text{Var}_P(V_M^\pi)=P(V_M^\pi)^2-(PV_M^\pi)^2
VarP(VMπ)=P(VMπ)2−(PVMπ)2其中
(
⋅
)
2
(\cdot)^2
(⋅)2是element-wise的平方,而不是对整个
V
V
V向量的平方,同理有
Var
P
(
Q
M
π
)
=
P
π
(
Q
M
π
)
2
−
(
P
π
Q
M
π
)
2
\text{Var}_P(Q^\pi_M)=P^\pi(Q^\pi_M)^2-(P^\pi Q^\pi_M)^2
VarP(QMπ)=Pπ(QMπ)2−(PπQMπ)2
3. 轨迹累积reward的方差
Σ
M
π
(
s
,
a
)
\Sigma_M^\pi(s,a)
ΣMπ(s,a)与一步转移方差
Var
P
(
V
M
π
)
\text{Var}_P(V_M^\pi)
VarP(VMπ)的关系:
Σ
M
π
=
γ
2
(
Var
P
(
Q
M
π
)
+
P
π
Σ
M
π
)
\Sigma_M^\pi=\gamma^2\Big(\text{Var}_P(Q_M^\pi)+P^\pi\Sigma_M^\pi\Big)
ΣMπ=γ2(VarP(QMπ)+PπΣMπ)
在证明之前先看个简写的结论:(为了理解期望下标是什么, s 1 , a 1 s_1,a_1 s1,a1是变量)
E s 1 , a 1 , P π [ ∑ t = 1 ∞ γ t r ( s t , a t ) − γ Q π ( s 1 , a 1 ) ] = E s 1 , a 1 , P π [ ∑ t = 1 ∞ γ t r ( s t , a t ) − γ [ r ( s 1 , a 1 ) + γ E P π ( s 2 , a 2 ∣ s 1 , a 1 ) [ r ( s 2 , a 2 ) + ⋯ ] ] ] = 0 \begin{aligned} &\mathbb E_{s_1,a_1,P^\pi}\left[\sum_{t=1}^\infty \gamma^tr(s_t,a_t)-\gamma Q^\pi(s_1,a_1)\right]\\ &=\mathbb E_{s_1,a_1,P^\pi}\left[\sum_{t=1}^\infty \gamma^tr(s_t,a_t)-\gamma \left[r(s_1,a_1)+\gamma \mathbb E_{P^\pi(s_2,a_2|s_1,a_1)}[r(s_2,a_2)+\cdots]\right]\right]\\ &=0 \end{aligned} Es1,a1,Pπ[t=1∑∞γtr(st,at)−γQπ(s1,a1)]=Es1,a1,Pπ[t=1∑∞γtr(st,at)−γ[r(s1,a1)+γEPπ(s2,a2∣s1,a1)[r(s2,a2)+⋯]]]=0
证明:对于一个(s,a)有
Σ M π ( s , a ) = E π , P [ ( ∑ t = 0 ∞ γ t r ( s t , a t ) − Q M π ( s , a ) ) 2 ∣ s 0 = s , a 0 = a ] = E s 0 = s , a 0 = a , P π [ ( ∑ t = 0 ∞ γ t r ( s t , a t ) − Q M π ( s , a ) ) 2 ] = E s 0 = s , a 0 = a , P π [ ( ∑ t = 1 ∞ γ t r ( s t , a t ) + r ( s , a ) − Q M π ( s , a ) ) 2 ] = E s 0 = s , a 0 = a , P π [ ( ∑ t = 1 ∞ γ t r ( s t , a t ) − γ Q π ( s 1 , a 1 ) − ( Q M π ( s , a ) − r ( s , a ) − γ Q π ( s 1 , a 1 ) ) ) 2 ] = E P π ( s 1 , a 1 ∣ s , a ) , P π [ ( ∑ t = 1 ∞ γ t r ( s t , a t ) − γ Q π ( s 1 , a 1 ) ) 2 ] + E P π ( s 1 , a 1 ∣ s , a ) , P π [ ( Q M π ( s , a ) − r ( s , a ) − γ Q π ( s 1 , a 1 ) ) 2 ] − 2 E P π ( s 1 , a 1 ∣ s , a ) , P π [ ( ∑ t = 1 ∞ γ t r ( s t , a t ) − γ Q π ( s 1 , a 1 ) ) ( Q M π ( s , a ) − r ( s , a ) − γ Q π ( s 1 , a 1 ) ) ] (该项为0) = E P π ( s 1 , a 1 ∣ s , a ) , P π [ ( ∑ t = 1 ∞ γ t r ( s t , a t ) − γ Q π ( s 1 , a 1 ) ) 2 ] + E P π ( s 1 , a 1 ∣ s , a ) , P π [ ( Q M π ( s , a ) − r ( s , a ) − γ Q π ( s 1 , a 1 ) ) 2 ] = γ 2 E P π ( s 1 , a 1 ∣ s , a ) , P π [ ( ∑ t = 1 ∞ γ t − 1 r ( s t , a t ) − Q π ( s 1 , a 1 ) ) 2 ] + E P π ( s 1 , a 1 ∣ s , a ) , P π [ ( Q M π ( s , a ) − r ( s , a ) − γ Q π ( s 1 , a 1 ) ) 2 ] = γ 2 P π Σ M π + γ 2 Var P ( Q M π ) \begin{aligned} \Sigma_M^\pi(s,a)&=\mathbb E_{\pi,P}\left[\left(\sum_{t=0}^\infty\gamma^tr(s_t,a_t)-Q^\pi_M(s,a)\right)^2\Big|s_0=s,a_0=a\right]\\ &= \mathbb E_{s_0=s,a_0=a,P^\pi}\left[\left(\sum_{t=0}^\infty\gamma^tr(s_t,a_t)-Q^\pi_M(s,a)\right)^2\right]\\ &= \mathbb E_{s_0=s,a_0=a,P^\pi}\left[\left(\sum_{t=1}^\infty\gamma^tr(s_t,a_t)+r(s,a)-Q^\pi_M(s,a)\right)^2\right]\\ &=\mathbb E_{s_0=s,a_0=a,P^\pi}\left[\left(\sum_{t=1}^\infty\gamma^tr(s_t,a_t)-\gamma Q^\pi(s_1,a_1)-\Big(Q^\pi_M(s,a)-r(s,a)-\gamma Q^\pi(s_1,a_1)\Big)\right)^2\right]\\ &=\mathbb E_{P^\pi(s_1,a_1|s,a),P^\pi}\left[\Big(\sum_{t=1}^\infty\gamma^tr(s_t,a_t)-\gamma Q^\pi(s_1,a_1)\Big)^2\right]+E_{P^\pi(s_1,a_1|s,a),P^\pi}\left[\Big(Q^\pi_M(s,a)-r(s,a)-\gamma Q^\pi(s_1,a_1)\Big)^2\right]\\ &-2E_{P^\pi(s_1,a_1|s,a),P^\pi}\left[\Big(\sum_{t=1}^\infty\gamma^tr(s_t,a_t)-\gamma Q^\pi(s_1,a_1)\Big)\Big(Q^\pi_M(s,a)-r(s,a)-\gamma Q^\pi(s_1,a_1)\Big)\right]\text{(该项为0)}\\ &=\mathbb E_{P^\pi(s_1,a_1|s,a),P^\pi}\left[\Big(\sum_{t=1}^\infty\gamma^tr(s_t,a_t)-\gamma Q^\pi(s_1,a_1)\Big)^2\right]+E_{P^\pi(s_1,a_1|s,a),P^\pi}\left[\Big(Q^\pi_M(s,a)-r(s,a)-\gamma Q^\pi(s_1,a_1)\Big)^2\right]\\ &=\gamma^2\mathbb E_{P^\pi(s_1,a_1|s,a),P^\pi}\left[\Big(\sum_{t=1}^\infty\gamma^{t-1}r(s_t,a_t)-Q^\pi(s_1,a_1)\Big)^2\right]+E_{P^\pi(s_1,a_1|s,a),P^\pi}\left[\Big(Q^\pi_M(s,a)-r(s,a)-\gamma Q^\pi(s_1,a_1)\Big)^2\right]\\ &=\gamma^2P^\pi\Sigma_M^\pi+\gamma^2\text{Var}_P(Q_M^\pi) \end{aligned} ΣMπ(s,a)=Eπ,P⎣⎡(t=0∑∞γtr(st,at)−QMπ(s,a))2∣∣∣s0=s,a0=a⎦⎤=Es0=s,a0=a,Pπ⎣⎡(t=0∑∞γtr(st,at)−QMπ(s,a))2⎦⎤=Es0=s,a0=a,Pπ⎣⎡(t=1∑∞γtr(st,at)+r(s,a)−QMπ(s,a))2⎦⎤=Es0=s,a0=a,Pπ⎣⎡(t=1∑∞γtr(st,at)−γQπ(s1,a1)−(QMπ(s,a)−r(s,a)−γQπ(s1,a1)))2⎦⎤=EPπ(s1,a1∣s,a),Pπ[(t=1∑∞γtr(st,at)−γQπ(s1,a1))2]+EPπ(s1,a1∣s,a),Pπ[(QMπ(s,a)−r(s,a)−γQπ(s1,a1))2]−2EPπ(s1,a1∣s,a),Pπ[(t=1∑∞γtr(st,at)−γQπ(s1,a1))(QMπ(s,a)−r(s,a)−γQπ(s1,a1))](该项为0)=EPπ(s1,a1∣s,a),Pπ[(t=1∑∞γtr(st,at)−γQπ(s1,a1))2]+EPπ(s1,a1∣s,a),Pπ[(QMπ(s,a)−r(s,a)−γQπ(s1,a1))2]=γ2EPπ(s1,a1∣s,a),Pπ[(t=1∑∞γt−1r(st,at)−Qπ(s1,a1))2]+EPπ(s1,a1∣s,a),Pπ[(QMπ(s,a)−r(s,a)−γQπ(s1,a1))2]=γ2PπΣMπ+γ2VarP(QMπ)
- Q π ( s 1 , a 1 ) = r ( s 1 , a 1 ) + γ E P π ( s 2 , a 2 ∣ s 1 , a 1 ) [ Q π ( s 2 , a 2 ) ] Q^\pi(s_1,a_1)=r(s_1,a_1)+\gamma \mathbb E_{P^\pi(s_2,a_2|s_1,a_1)}[Q^\pi(s_2,a_2)] Qπ(s1,a1)=r(s1,a1)+γEPπ(s2,a2∣s1,a1)[Qπ(s2,a2)]
- Q π ( s , a ) = r ( s , a ) + γ E P π ( s 1 , a 1 ∣ s , a ) [ Q π ( s 1 , a 1 ) ] Q^\pi(s,a)=r(s,a)+\gamma \mathbb E_{P^\pi(s_1,a_1|s,a)}[Q^\pi(s_1,a_1)] Qπ(s,a)=r(s,a)+γEPπ(s1,a1∣s,a)[Qπ(s1,a1)]
- 这是为了完整展开最后一步这项 E P π ( s 1 , a 1 ∣ s , a ) , P π [ ( Q M π ( s , a ) − r ( s , a ) − γ Q π ( s 1 , a 1 ) ) 2 ] = E P π ( s 1 , a 1 ∣ s , a ) , P π [ γ E P π ( s 2 , a 2 ∣ s 1 , a 1 ) [ Q π ( s 2 , a 2 ) ] − γ Q π ( s 1 , a 1 ) ] E_{P^\pi(s_1,a_1|s,a),P^\pi}\left[\Big(Q^\pi_M(s,a)-r(s,a)-\gamma Q^\pi(s_1,a_1)\Big)^2\right]=E_{P^\pi(s_1,a_1|s,a),P^\pi}\left[ \gamma \mathbb E_{P^\pi(s_2,a_2|s_1,a_1)}[Q^\pi(s_2,a_2)]-\gamma Q^\pi(s_1,a_1)\right] EPπ(s1,a1∣s,a),Pπ[(QMπ(s,a)−r(s,a)−γQπ(s1,a1))2]=EPπ(s1,a1∣s,a),Pπ[γEPπ(s2,a2∣s1,a1)[Qπ(s2,a2)]−γQπ(s1,a1)]满足Q一步转移方差的定义
1.2 Hoeffding‘s inequality
参考:A Gentle Introduction to Concentration Inequalities
具体证明过程用到统计学习理论:马尔可夫不等式——切比雪夫不等式——矩母函数MGF——大数定理,得到常用的Hoeffding不等式:
假设有一系列的有界的随机变量
Z
1
,
.
.
.
,
Z
N
,
Z
i
∈
[
a
i
,
b
i
]
Z_1,...,Z_N,Z_i\in [a_i,b_i]
Z1,...,ZN,Zi∈[ai,bi],对
∀
t
>
0
\forall t>0
∀t>0有:
Pr
(
∣
1
N
∑
i
=
1
N
Z
i
−
E
Z
∣
≥
t
)
≤
2
exp
(
−
2
N
2
t
2
∑
i
=
1
N
(
b
i
−
a
i
)
2
)
\Pr(|\frac{1}{N}\sum_{i=1}^NZ_i-\mathbb EZ|\geq t)\leq 2\exp(-\frac{2N^2t^2}{\sum_{i=1}^N(b_i-a_i)^2})
Pr(∣N1i=1∑NZi−EZ∣≥t)≤2exp(−∑i=1N(bi−ai)22N2t2)
1.3 Bernstein’s inequality
参考:A Gentle Introduction to Concentration Inequalities
假设有一系列的有界独立的随机变量
Z
1
,
.
.
.
,
Z
N
,
E
Z
i
=
0
,
∣
Z
i
∣
≤
l
,
σ
2
=
1
n
∑
i
=
1
n
Var
(
Z
i
)
Z_1,...,Z_N,\mathbb EZ_i=0,|Z_i|\leq l,\sigma^2=\frac{1}{n}\sum_{i=1}^n\text{Var}(Z_i)
Z1,...,ZN,EZi=0,∣Zi∣≤l,σ2=n1∑i=1nVar(Zi),则对
∀
t
>
0
\forall t>0
∀t>0有:
Pr
(
1
n
∑
i
=
1
n
Z
i
≥
t
)
≤
exp
(
−
n
t
2
2
σ
2
+
2
l
t
3
)
\Pr(\frac{1}{n}\sum_{i=1}^nZ_i\geq t)\leq\exp(-\frac{nt^2}{2\sigma^2+\frac{2lt}{3}})
Pr(n1i=1∑nZi≥t)≤exp(−2σ2+32ltnt2)
二、回到目标上界的分析
目标是找Q值差异的上界,这个上界(state,action)处的样本数N相关。
目标:
Q
⋆
−
Q
^
⋆
≤
γ
(
I
−
γ
P
^
π
⋆
)
−
1
(
P
−
P
^
)
V
π
⋆
Q^\star-\widehat{Q}^\star\leq\gamma\left(I-\gamma \widehat{P}^{\pi^\star}\right)^{-1}(P-\widehat{P}) V^{\pi^\star}
Q⋆−Q
⋆≤γ(I−γP
π⋆)−1(P−P
)Vπ⋆
这里主要是两个思路:
- 对 ( P − P ^ ) V π ⋆ (P-\widehat{P}) V^{\pi^\star} (P−P )Vπ⋆使用Hoeffding + 利用 ( I − γ P ^ π ⋆ ) − 1 \left(I-\gamma \widehat{P}^{\pi^\star}\right)^{-1} (I−γP π⋆)−1的值域进行简单放缩
- 对 ( P − P ^ ) V π ⋆ (P-\widehat{P}) V^{\pi^\star} (P−P )Vπ⋆使用Bernstein + 使用Bellman-variance equation处理 ( I − γ P ^ π ⋆ ) − 1 \left(I-\gamma \widehat{P}^{\pi^\star}\right)^{-1} (I−γP π⋆)−1
2.1 使用Hoeffding进行放缩
- 在上一篇文章Sample Complexity(中)已知,对于所有策略
π
\pi
π有:
∥ ( I − γ P π ) − 1 v ∥ ∞ ≤ ∥ v ∥ ∞ 1 − γ \|(I-\gamma P^\pi)^{-1}v\|_\infty \leq \frac{\|v\|_\infty}{1-\gamma} ∥(I−γPπ)−1v∥∞≤1−γ∥v∥∞ - 所以相当于直接取
(
I
−
γ
P
^
π
∗
)
−
1
(I-\gamma\widehat P^{\pi^*})^{-1}
(I−γP
π∗)−1的最大值,即
∥ Q ⋆ − Q ^ ⋆ ∥ ∞ ≤ γ 1 − γ ∥ ( P − P ^ ) V π ⋆ ∥ ∞ \|Q^\star-\widehat{Q}^\star\|_\infty\leq \frac{\gamma}{1-\gamma}\|(P-\widehat{P}) V^{\pi^\star}\|_\infty ∥Q⋆−Q ⋆∥∞≤1−γγ∥(P−P )Vπ⋆∥∞ - 完整写开
∥
(
P
−
P
^
)
V
π
⋆
∥
∞
\|(P-\widehat{P}) V^{\pi^\star}\|_\infty
∥(P−P
)Vπ⋆∥∞如下:
∥ ( P − P ^ ) V π ⋆ ∥ ∞ = max s , a ∣ E s ′ ∼ P ( ⋅ ∣ s , a ) [ V ⋆ ( s ′ ) ] − E s ′ ∼ P ^ ( ⋅ ∣ s , a ) [ V ⋆ ( s ′ ) ] ∣ \|(P-\widehat{P}) V^{\pi^\star}\|_\infty=\max_{s,a}\left|\mathbb E_{s'\sim P(\cdot|s,a)}[V^\star(s')]-\mathbb E_{s'\sim \widehat P(\cdot|s,a)}[V^\star(s')]\right| ∥(P−P )Vπ⋆∥∞=s,amax∣∣∣Es′∼P(⋅∣s,a)[V⋆(s′)]−Es′∼P (⋅∣s,a)[V⋆(s′)]∣∣∣ - 其中
P
^
(
s
′
∣
s
,
a
)
=
#
(
s
′
,
s
,
a
)
N
\widehat P(s'\mid s,a)=\frac{\# (s',s,a)}{N}
P
(s′∣s,a)=N#(s′,s,a),所以有:
∣ E s ′ ∼ P ( ⋅ ∣ s , a ) [ V ⋆ ( s ′ ) ] − E s ′ ∼ P ^ ( ⋅ ∣ s , a ) [ V ⋆ ( s ′ ) ] ∣ = ∣ ∑ s ′ ( p ( s ′ ∣ s , a ) − # ( s ′ , s , a ) N ) V ⋆ ( s ′ ) ∣ \left|\mathbb E_{s'\sim P(\cdot|s,a)}[V^\star(s')]-\mathbb E_{s'\sim \widehat P(\cdot|s,a)}[V^\star(s')]\right|=\left|\sum_{s'}\Big(p(s'|s,a)-\frac{\# (s',s,a)}{N}\Big)V^\star(s')\right| ∣∣∣Es′∼P(⋅∣s,a)[V⋆(s′)]−Es′∼P (⋅∣s,a)[V⋆(s′)]∣∣∣=∣∣∣∣∣s′∑(p(s′∣s,a)−N#(s′,s,a))V⋆(s′)∣∣∣∣∣ - 由Hoeffding inequality,可知对于一个(s,a)确定的向量
P
(
⋅
∣
s
,
a
)
P(\cdot|s,a)
P(⋅∣s,a),可看作一个多维变量
X
X
X,有
Pr ( p ( ⋅ ∣ s , a ) − # ( s ′ , s , a ) N ≥ t ) ≤ 2 exp ( − 2 N t 2 ) \Pr\Big(p(\cdot|s,a)-\frac{\# (s',s,a)}{N}\geq t\Big)\leq2\exp(-2Nt^2) Pr(p(⋅∣s,a)−N#(s′,s,a)≥t)≤2exp(−2Nt2) - 因此对于所有可能的
(
s
,
a
)
(s,a)
(s,a)构成的矩阵都需要满足上述的约束,才有
Pr ( ∥ ( P − P ^ ) ∥ ≥ t ) ≤ 2 ∣ S ∣ ∣ A ∣ exp ( − 2 N t 2 ) \Pr\Big(\|(P-\widehat{P}) \|\geq t\Big)\leq2|S||A|\exp(-2Nt^2) Pr(∥(P−P )∥≥t)≤2∣S∣∣A∣exp(−2Nt2) - 令
δ
=
2
∣
S
∣
∣
A
∣
exp
(
−
2
N
t
2
)
\delta=2|S||A|\exp(-2Nt^2)
δ=2∣S∣∣A∣exp(−2Nt2),解得
t
=
2
ln
∣
S
∣
∣
A
∣
δ
N
t=\sqrt{\frac{2\ln\frac{|S||A|}{\delta}}{N}}
t=N2lnδ∣S∣∣A∣,因此至少有
1
−
δ
1-\delta
1−δ的概率有如下不等式成立
∥ ( P − P ^ ) ∥ ≤ 2 ln ∣ S ∣ ∣ A ∣ δ N \|(P-\widehat{P}) \|\leq\sqrt{\frac{2\ln\frac{|S||A|}{\delta}}{N}} ∥(P−P )∥≤N2lnδ∣S∣∣A∣ - 所以,因为 ∥ V ∥ ≤ 1 1 − γ \|V\|\leq \frac{1}{1-\gamma} ∥V∥≤1−γ1,所以至少有 1 − δ 1-\delta 1−δ的概率有 ∥ ( P − P ^ ) V π ⋆ ∥ ∞ ≤ 1 1 − γ 2 ln ∣ S ∣ ∣ A ∣ δ N \|(P-\widehat{P}) V^{\pi^\star}\|_\infty \leq\frac{1}{1-\gamma}\sqrt{\frac{2\ln\frac{|S||A|}{\delta}}{N}} ∥(P−P )Vπ⋆∥∞≤1−γ1N2lnδ∣S∣∣A∣
- 最后有 1 − δ 1-\delta 1−δ的概率,有如下成立 Q ⋆ − Q ^ ⋆ ≤ γ ( I − γ P ^ π ⋆ ) − 1 ( P − P ^ ) V π ⋆ ≤ γ ( 1 − γ ) 2 2 ln ∣ S ∣ ∣ A ∣ δ N Q^\star-\widehat{Q}^\star\leq\gamma\left(I-\gamma \widehat{P}^{\pi^\star}\right)^{-1}(P-\widehat{P}) V^{\pi^\star}\leq\frac{\gamma}{(1-\gamma)^2}\sqrt{\frac{2\ln\frac{|S||A|}{\delta}}{N}} Q⋆−Q ⋆≤γ(I−γP π⋆)−1(P−P )Vπ⋆≤(1−γ)2γN2lnδ∣S∣∣A∣
这个是upper bound,关注点是称为effective horizon的 1 1 − γ \frac{1}{1-\gamma} 1−γ1,是其平方项,下面使用了相关等式的放缩会更加紧致,这个紧致是相对于N来说的
总算用样本bound住了!这个statisticscal error 加上之前的optimization error就ok了!
2.2 使用Bernstein进行放缩
- 由Bellman-variance equation即
Σ
M
π
=
γ
2
(
Var
P
(
V
M
π
)
+
P
π
Σ
M
π
)
\Sigma_M^\pi=\gamma^2\Big(\text{Var}_P(V_M^\pi)+P^\pi\Sigma_M^\pi\Big)
ΣMπ=γ2(VarP(VMπ)+PπΣMπ),可解得:
Σ M π = ( I − γ 2 P π ) − 1 γ 2 Var P ( V M π ) \Sigma_M^\pi=(I-\gamma^2P^\pi)^{-1}\gamma^2\text{Var}_P(V_M^\pi) ΣMπ=(I−γ2Pπ)−1γ2VarP(VMπ)
这回答了第二个问题:寻找 ( I − γ P ^ π ⋆ ) − 1 \left(I-\gamma \widehat{P}^{\pi^\star}\right)^{-1} (I−γP π⋆)−1或 ( I − γ P π ⋆ ) − 1 \left(I-\gamma P^{\pi^\star}\right)^{-1} (I−γPπ⋆)−1相关的公式
- 对
∥
(
P
−
P
^
)
V
π
⋆
∥
\|(P-\widehat{P}) V^{\pi^\star}\|
∥(P−P
)Vπ⋆∥使用Bernstein inequality有
∥ ( P − P ^ ) V π ⋆ ∥ ≤ 2 ln ∣ S ∣ ∣ A ∣ δ N Var P ( V ∗ ) + 1 1 − γ 2 ln 2 ∣ S ∣ ∣ A ∣ δ 3 N \|(P-\widehat{P}) V^{\pi^\star}\|\leq\sqrt{\frac{2\ln\frac{|S||A|}{\delta}}{N}}\sqrt{\text{Var}_P(V^*)}+\frac{1}{1-\gamma}\frac{2\ln 2\frac{|S||A|}{\delta}}{3N} ∥(P−P )Vπ⋆∥≤N2lnδ∣S∣∣A∣VarP(V∗)+1−γ13N2ln2δ∣S∣∣A∣ - 取dominant的量级,因此有
∥ Q ⋆ − Q ^ ⋆ ∥ ∞ ≤ γ 2 ln ∣ S ∣ ∣ A ∣ δ N ∥ ( I − γ P ^ π ⋆ ) − 1 Var P ( V ∗ ) ∥ ∞ \|Q^\star-\widehat Q^\star\|_\infty\leq \gamma \sqrt{\frac{2\ln\frac{|S||A|}{\delta}}{N}}\sqrt{\|(I-\gamma\widehat P^{\pi^\star})^{-1}\text{Var}_P(V^*)\|_\infty} ∥Q⋆−Q ⋆∥∞≤γN2lnδ∣S∣∣A∣∥(I−γP π⋆)−1VarP(V∗)∥∞ - 可由Bellman variance equation推得:
∥ ( I − γ P ^ π ⋆ ) − 1 Var P ( V ∗ ) ∥ ∞ ≤ 2 ( 1 − γ ) 3 \|(I-\gamma\widehat P^{\pi^\star})^{-1}\text{Var}_P(V^*)\|_\infty\leq \sqrt{\frac{2}{(1-\gamma)^3}} ∥(I−γP π⋆)−1VarP(V∗)∥∞≤(1−γ)32 - 因此,把常数项用c简写为
∥ Q ⋆ − Q ^ ⋆ ∥ ∞ ≤ γ c ( 1 − γ ) 3 ln c ∣ S ∣ ∣ A ∣ / δ N + c ( 1 − γ ) 3 ln c ∣ S ∣ ∣ A ∣ / δ N \|Q^\star-\widehat Q^\star\|_\infty\leq \gamma \sqrt{\frac{c}{(1-\gamma)^3}\frac{\ln c|S||A|/\delta}{N}}+\frac{c}{(1-\gamma)^3}\frac{\ln c|S||A|/\delta}{N} ∥Q⋆−Q ⋆∥∞≤γ(1−γ)3cNlnc∣S∣∣A∣/δ+(1−γ)3cNlnc∣S∣∣A∣/δ
2.3 样本复杂度的最终形式
令 ϵ = γ c ( 1 − γ ) 3 ln c ∣ S ∣ ∣ A ∣ / δ N + c ( 1 − γ ) 3 ln c ∣ S ∣ ∣ A ∣ / δ N \epsilon=\gamma \sqrt{\frac{c}{(1-\gamma)^3}\frac{\ln c|S||A|/\delta}{N}}+\frac{c}{(1-\gamma)^3}\frac{\ln c|S||A|/\delta}{N} ϵ=γ(1−γ)3cNlnc∣S∣∣A∣/δ+(1−γ)3cNlnc∣S∣∣A∣/δ得 N = O ( 1 ( 1 − γ ) 3 ln ∣ S ∣ ∣ A ∣ / δ ϵ 2 ) N=O(\frac{1}{(1-\gamma)^3}\frac{\ln |S||A|/\delta}{\epsilon^2}) N=O((1−γ)31ϵ2ln∣S∣∣A∣/δ)
所以有 ∀ ϵ < 1 \forall \epsilon < 1 ∀ϵ<1,至少需要 N ≥ 1 ( 1 − γ ) 3 ln ∣ S ∣ ∣ A ∣ / δ ϵ 2 N\geq \frac{1}{(1-\gamma)^3}\frac{\ln |S||A|/\delta}{\epsilon^2} N≥(1−γ)31ϵ2ln∣S∣∣A∣/δ个样本,才能在 1 − δ 1-\delta 1−δ的概率下,满足 ∥ Q ⋆ − Q ^ ⋆ ∥ ∞ ≤ ϵ \|Q^\star-\widehat Q^\star\|_\infty\leq \epsilon ∥Q⋆−Q ⋆∥∞≤ϵ
三、Generative model的Sample Complexity的总结
- 均匀交互假设,能在任意的 ( s , a ) (s,a) (s,a)处调用stimulator,肆意地获取下一状态,回避了exploration的问题
- 转移矩阵P未知,用最直接的方式进行建模 P ^ ( s ′ ∣ s , a ) = # ( s ′ , s , a ) N \widehat P(s'\mid s,a)=\frac{\# (s',s,a)}{N} P (s′∣s,a)=N#(s′,s,a)
- 学习的算法选择的是Q-value iteration 或 Policy Iteration
- 在第一篇文章中每天一个RL基础理论(4)——Sample Complexity(上),出发点是想bound住每一个策略的误差Uniform Value Accuracy,进而最优策略也被bound住了,命题如下,证明的入手点是从
l
1
−
n
o
r
m
l_1 -norm
l1−norm相关的不等式去bound住transition model的建模误差Model Accuracy
- 在第二篇文章中每天一个RL基础理论(5)——Sample Complexity(中),出发点是想直接bound住optimal policy的误差即可,没必要bound住迭代过程中的每一个策略,然后使用component-wise bound 以及 simulation lemma对目标进行了分析与拆解
Q ⋆ − Q ^ ⋆ ≤ γ ( I − γ P ^ π ⋆ ) − 1 ( P − P ^ ) V π ⋆ Q ⋆ − Q ^ ⋆ ≤ γ ( I − γ P π ⋆ ) − 1 ( P − P ^ ) V ^ π ⋆ \begin{aligned} &Q^\star-\widehat{Q}^\star\leq\gamma\left(I-\gamma \widehat{P}^{\pi^\star}\right)^{-1}(P-\widehat{P}) V^{\pi^\star}\\ &Q^\star-\widehat{Q}^\star\leq\gamma\left(I-\gamma P^{\pi^\star}\right)^{-1}(P-\widehat P)\widehat V^{\pi^\star} \end{aligned} Q⋆−Q ⋆≤γ(I−γP π⋆)−1(P−P )Vπ⋆Q⋆−Q ⋆≤γ(I−γPπ⋆)−1(P−P )V π⋆ - 在这篇文章中,分别选择了Hoeffding 与 Berstein inequality对上界的关键量进行了概率意义的限制,其中Berstein因为额外利用了分布方差的量从而bound得更紧致,但同时也必须要引入Bellman variance相关的公式与不等式进行放缩。最终整体命题如下:
关于uncertainty的参数 δ ≥ 0 \delta \geq 0 δ≥0,一些常数的简写量 c > 0 c>0 c>0,有 1 − δ 1-\delta 1−δ的概率有如下成立:
- (Value estimation)
∥ Q ⋆ − Q ^ ⋆ ∥ ∞ ≤ γ c ( 1 − γ ) 3 ln c ∣ S ∣ ∣ A ∣ / δ N + c ( 1 − γ ) 3 ln c ∣ S ∣ ∣ A ∣ / δ N \|Q^\star-\widehat Q^\star\|_\infty\leq \gamma \sqrt{\frac{c}{(1-\gamma)^3}\frac{\ln c|S||A|/\delta}{N}}+\frac{c}{(1-\gamma)^3}\frac{\ln c|S||A|/\delta}{N} ∥Q⋆−Q ⋆∥∞≤γ(1−γ)3cNlnc∣S∣∣A∣/δ+(1−γ)3cNlnc∣S∣∣A∣/δ - (Sub-optimality)如果
N
≥
1
(
1
−
γ
)
2
N\geq \frac{1}{(1-\gamma)^2}
N≥(1−γ)21
∥ Q ⋆ − Q π ^ ⋆ ∥ ∞ ≤ γ c ( 1 − γ ) 3 ln c ∣ S ∣ ∣ A ∣ / δ N \|Q^\star- Q^{\widehat \pi^\star}\|_\infty\leq \gamma \sqrt{\frac{c}{(1-\gamma)^3}\frac{\ln c|S||A|/\delta}{N}} ∥Q⋆−Qπ ⋆∥∞≤γ(1−γ)3cNlnc∣S∣∣A∣/δ
换一种表述方式
3. (Value estimation)在
ϵ
≤
1
\epsilon \leq 1
ϵ≤1下,如果
总
的
样
本
复
杂
度
=
∣
S
∣
∣
A
∣
N
≥
c
∣
S
∣
∣
A
∣
(
1
−
γ
)
3
ln
c
∣
S
∣
∣
A
∣
/
δ
ϵ
2
总的样本复杂度=|S||A|N\geq \frac{c|S||A|}{(1-\gamma)^3}\frac{\ln c|S||A|/\delta}{\epsilon^2}
总的样本复杂度=∣S∣∣A∣N≥(1−γ)3c∣S∣∣A∣ϵ2lnc∣S∣∣A∣/δ则在
1
−
δ
1-\delta
1−δ概率下有:
∥
Q
⋆
−
Q
^
⋆
∥
∞
≤
ϵ
\|Q^\star-\widehat Q^\star\|_\infty \leq \epsilon
∥Q⋆−Q
⋆∥∞≤ϵ
4. Sub-Optimality在
ϵ
≤
1
1
−
γ
\epsilon \leq \sqrt{\frac{1}{1-\gamma}}
ϵ≤1−γ1下,如果
总
的
样
本
复
杂
度
=
∣
S
∣
∣
A
∣
N
≥
c
∣
S
∣
∣
A
∣
(
1
−
γ
)
3
ln
c
∣
S
∣
∣
A
∣
/
δ
ϵ
2
总的样本复杂度=|S||A|N\geq \frac{c|S||A|}{(1-\gamma)^3}\frac{\ln c|S||A|/\delta}{\epsilon^2}
总的样本复杂度=∣S∣∣A∣N≥(1−γ)3c∣S∣∣A∣ϵ2lnc∣S∣∣A∣/δ则在
1
−
δ
1-\delta
1−δ概率下有:
∥
Q
⋆
−
Q
π
^
⋆
∥
∞
≤
ϵ
\|Q^\star- Q^{\widehat \pi^\star}\|_\infty \leq \epsilon
∥Q⋆−Qπ
⋆∥∞≤ϵ
- 一个关于样本获取的generative model的强假设,一个是naively对transition model的建模,然后在Q-value的iteration算法下,才推出这个bound。。
- 这个bound还是可以分析一下的,样本复杂度与effective horizon呈三次方的关系,和环境的状态与动作数量呈线性关系