policy-base SOTA
学习周博雷老师课程总结
Natural Policy Gradient
为了改善PG算法的不稳定的缺点(如果更新到一个bad policy,就会采集到的一个坏的数据集,进而恶性循环。)
这样我们在两个策略之间限制一个距离(KL-divergence),保证优化不会跑的太偏,优化的方法就变为下面这样一个等式,就是说我们在上一个策略周围找一个使得目标函数值最大的方向进行前进,下面就是Natural Policy Gradient的实现方式:
d
∗
=
arg
max
J
(
θ
+
d
)
,
d^{*}=\arg \max J(\theta+d),
d∗=argmaxJ(θ+d),s.t.
K
L
(
π
θ
∥
π
θ
+
d
)
=
c
K L\left(\pi_{\theta} \| \pi_{\theta+d}\right)=c
KL(πθ∥πθ+d)=c
TRPO
TRPO=Natural Policy Gradient + importance sampling
根据sampling定理:
E
x
∼
p
[
f
(
x
)
]
=
∫
p
(
x
)
f
(
x
)
d
x
=
∫
q
(
x
)
p
(
x
)
q
(
x
)
f
(
x
)
d
x
=
E
x
∼
q
[
p
(
x
)
q
(
x
)
f
(
x
)
]
\mathbb{E}_{x \sim p}[f(x)]=\int p(x) f(x) d x=\int q(x) \frac{p(x)}{q(x)} f(x) d x=\mathbb{E}_{x \sim q}\left[\frac{p(x)}{q(x)} f(x)\right]
Ex∼p[f(x)]=∫p(x)f(x)dx=∫q(x)q(x)p(x)f(x)dx=Ex∼q[q(x)p(x)f(x)]
可以把目标函数写成:
J
(
θ
)
=
E
a
∼
π
θ
[
r
(
s
,
a
)
]
=
E
a
∼
π
^
[
π
θ
(
s
,
a
)
π
^
(
s
,
a
)
r
(
s
,
a
)
]
J(\theta)=\mathbb{E}_{a \sim \pi_{\theta}}[r(s, a)]=\mathbb{E}_{a \sim \hat{\pi}}\left[\frac{\pi_{\theta}(s, a)}{\hat{\pi}(s, a)} r(s, a)\right]
J(θ)=Ea∼πθ[r(s,a)]=Ea∼π^[π^(s,a)πθ(s,a)r(s,a)]
这样我们就能用off-policy来进行价值函数的更新,即一个behavior policy,一个training policy,下面就是TRPO的优化公式:
J
θ
old
(
θ
)
=
E
t
[
π
θ
(
a
t
∣
s
t
)
π
θ
old
(
a
t
∣
s
t
)
R
t
]
subject to
K
L
(
π
θ
old
(
.
∣
s
t
)
∥
π
θ
(
.
∣
s
t
)
)
≤
δ
\begin{aligned} J_{\theta_{\text {old}}}(\theta) &=\mathbb{E}_{t}\left[\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} R_{t}\right] \\ \text { subject to } & K L\left(\pi_{\theta_{\text {old}}}\left(. \mid s_{t}\right) \| \pi_{\theta}\left(. \mid s_{t}\right)\right) \leq \delta \end{aligned}
Jθold(θ) subject to =Et[πθold(at∣st)πθ(at∣st)Rt]KL(πθold(.∣st)∥πθ(.∣st))≤δ
KFAC
提升TRPO的计算效率,
PPO
TRPO的简化版本
TRPO:
maximize
θ
E
t
[
π
θ
(
a
t
∣
s
t
)
π
θ
old
(
a
t
∣
s
t
)
A
t
]
subject to
E
t
[
K
L
[
π
θ
old
(
.
∣
s
t
)
,
π
θ
(
.
∣
s
t
)
]
]
≤
δ
\begin{array}{l} \operatorname{maximize}_{\theta} \mathbb{E}_{t}\left[\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} A_{t}\right] \\ \text { subject to } \mathbb{E}_{t}\left[K L\left[\pi_{\theta_{\text {old}}}\left(. \mid s_{t}\right), \pi_{\theta}\left(. \mid s_{t}\right)\right]\right] \leq \delta \end{array}
maximizeθEt[πθold(at∣st)πθ(at∣st)At] subject to Et[KL[πθold(.∣st),πθ(.∣st)]]≤δ
PPO的优化简化:
maximize
θ
E
t
[
π
θ
(
a
t
∣
s
t
)
π
θ
o
l
d
(
a
t
∣
s
t
)
A
t
]
−
β
E
t
[
K
L
[
π
θ
old
(
.
∣
s
t
)
,
π
θ
(
.
∣
s
t
)
]
]
\operatorname{maximize}_{\theta} \mathbb{E}_{t}\left[\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{o l d}}\left(a_{t} \mid s_{t}\right)} A_{t}\right]-\beta \mathbb{E}_{t}\left[K L\left[\pi_{\theta_{\text {old}}}\left(. \mid s_{t}\right), \pi_{\theta}\left(. \mid s_{t}\right)\right]\right]
maximizeθEt[πθold(at∣st)πθ(at∣st)At]−βEt[KL[πθold(.∣st),πθ(.∣st)]]
简化后的PPO用一阶优化过程SGD,而TRPO用了二阶的信息,所以二阶的较慢。
带clipping的PPO
r
t
(
θ
)
=
π
θ
(
a
t
∣
s
t
)
π
θ
old
(
a
t
∣
s
t
)
r_{t}(\theta)=\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)}
rt(θ)=πθold(at∣st)πθ(at∣st)
就是将价值函数的取值进行一个限制,则带有clipping的PPO的优化为:
L
t
(
θ
)
=
min
(
r
t
(
θ
)
A
^
t
,
clip
(
r
t
(
θ
)
,
1
−
ϵ
,
1
+
ϵ
)
A
^
t
)
L_{t}(\theta)=\min \left(r_{t}(\theta) \hat{A}_{t}, \operatorname{clip}\left(r_{t}(\theta), 1-\epsilon, 1+\epsilon\right) \hat{A}_{t}\right)
Lt(θ)=min(rt(θ)A^t,clip(rt(θ),1−ϵ,1+ϵ)A^t)
这里用clipping代替了KL-divergence这一项。