ICML 2024
paper
code
解决离线到在线过程中,判别器对齐问题。
Intro
传统采用离线模仿学习结合基于GAIL的在线模仿学习,容易因为在线阶段初始化的判别器表现具有随机性,与离线获得的策略不一致。因此,本文提出的OLLIE,便是利用混合质量的数据,实现判别器与策略之间的对齐,从何防止O2O的performance drop。
Method
GAIL
GAIL是一种传统在线模仿学习算法,其目标是对抗的学习一个判别器,用于区分专家数据与在线数据
min
π
max
D
E
ρ
π
[
log
D
(
s
,
a
)
]
+
E
ρ
ˉ
e
[
log
(
1
−
D
(
s
,
a
)
)
]
.
(
2
)
\min_\pi\max_D\mathbb{E}_{\rho^\pi}[\log D(s,a)]+\mathbb{E}_{\bar{\rho}^e}[\log(1-D(s,a))].(2)
πminDmaxEρπ[logD(s,a)]+Eρˉe[log(1−D(s,a))].(2).
最优判别器输出表示为
D
∗
(
s
,
a
)
=
ρ
π
(
s
,
a
)
ρ
π
(
s
,
a
)
+
ρ
~
e
(
s
,
a
)
.
D^*(s,a)=\frac{\rho^\pi(s,a)}{\rho^\pi(s,a)+\tilde{\rho}^e(s,a)}.
D∗(s,a)=ρπ(s,a)+ρ~e(s,a)ρπ(s,a).
Offline IL
假设专家数据
D
e
D_e
De以及混合数据
D
o
≐
D
e
∪
D
s
\mathcal{D}_{o}\doteq\mathcal{D}_{e}\cup\mathcal{D}_{s}
Do≐De∪Ds的状态动作分布分别为
ρ
~
e
>
0
\tilde{\rho}^{e}>0
ρ~e>0以及
ρ
~
o
>
0
\tilde{\rho}^o>0
ρ~o>0。离线学习的目标可以看作状态动作分布匹配问题,通过一个逆KL散度表达
min
π
D
K
L
(
ρ
π
∥
ρ
~
e
)
=
E
(
s
,
a
)
∼
ρ
π
[
log
ρ
π
(
s
,
a
)
ρ
~
e
(
s
,
a
)
]
\operatorname*{min}_{\pi}D_{\mathrm{KL}}(\rho^{\pi}\|\tilde{\rho}^{e})=\mathbb{E}_{(s,a)\sim\rho^{\pi}}\left[\log{\frac{\rho^{\pi}(s,a)}{\tilde{\rho}^{e}(s,a)}}\right]
πminDKL(ρπ∥ρ~e)=E(s,a)∼ρπ[logρ~e(s,a)ρπ(s,a)]
为了让混合数据参与上式的处理过程,在对数项的分子分母同时添加
ρ
~
o
\tilde{\rho}^o
ρ~o,原问题改造为
max
π
E
(
s
,
a
)
∼
ρ
π
[
R
~
(
s
,
a
)
]
−
D
K
L
(
ρ
π
∥
ρ
~
o
)
(
5
)
\max_\pi\mathbb{E}_{(s,a)\sim\rho^\pi}\big[\tilde{R}(s,a)\big]-D_{\mathrm{KL}}(\rho^\pi\|\tilde{\rho}^o)~~~~(5)
πmaxE(s,a)∼ρπ[R~(s,a)]−DKL(ρπ∥ρ~o) (5)
其中
R
~
(
s
,
a
)
≐
log
ρ
ˉ
e
(
s
,
a
)
ρ
ˉ
o
(
s
,
a
)
\tilde{R}(s,a)\doteq\log\frac{\bar{\rho}^e(s,a)}{\bar{\rho}^o(s,a)}
R~(s,a)≐logρˉo(s,a)ρˉe(s,a)。对于低维表格环境可以通过统计计算状态动作分布,而对于高维环境则可以通过训练一个判别器进行区分
max
d
E
ρ
~
e
[
log
d
(
s
,
a
)
]
+
E
ρ
~
o
[
log
(
1
−
d
(
s
,
a
)
)
]
(
6
)
\max_d\mathbb{E}_{\tilde{\rho}^e}\big[\log d(s,a)\big]+\mathbb{E}_{\tilde{\rho}^o}\big[\log(1-d(s,a))\big]\quad(6)
dmaxEρ~e[logd(s,a)]+Eρ~o[log(1−d(s,a))](6)
进而得到
R
~
(
s
,
a
)
=
log
ρ
~
e
(
s
,
a
)
ρ
~
o
(
s
,
a
)
=
log
d
∗
(
s
,
a
)
1
−
d
∗
(
s
,
a
)
.
(
7
)
\tilde{R}(s,a)=\log\frac{\tilde{\rho}^{e}(s,a)}{\tilde{\rho}^{o}(s,a)}=\log\frac{d^{*}(s,a)}{1-d^{*}(s,a)}.\quad(7)
R~(s,a)=logρ~o(s,a)ρ~e(s,a)=log1−d∗(s,a)d∗(s,a).(7)。
而对于KL正则,在Offline设定下无法保证策略性能。因此,算法提出状态动作的分布需要满足贝尔曼流约束(Bellman flow constraints),即
Z
≐
{
ρ
:
ρ
(
s
,
a
)
≥
0
,
f
s
(
ρ
)
=
0
,
∀
s
∈
S
,
a
∈
A
}
(
8
)
w
h
e
r
e
f
s
(
ρ
)
≐
(
1
−
γ
)
μ
(
s
)
+
γ
∑
a
,
s
′
T
(
s
∣
s
′
,
a
)
ρ
(
s
′
,
a
)
−
∑
a
ρ
(
s
,
a
)
.
\mathcal{Z}\doteq\left\{\rho:\rho(s,a)\geq0,f_{s}(\rho)=0,\forall s\in\mathcal{S},a\in\mathcal{A}\right\}\quad(8)\\\mathrm{where~~~}f_s(\rho)\doteq(1-\gamma)\mu(s)+\gamma\sum_{a,s^{\prime}}T(s|s^{\prime},a)\rho(s^{\prime},a)-\sum_a\rho(s,a).
Z≐{ρ:ρ(s,a)≥0,fs(ρ)=0,∀s∈S,a∈A}(8)where fs(ρ)≐(1−γ)μ(s)+γa,s′∑T(s∣s′,a)ρ(s′,a)−a∑ρ(s,a).
上述条件下,策略与其平稳状态-动作分布之间存在一一对应关系:如果
ρ
∈
Z
\rho \in \mathcal{Z}
ρ∈Z,则
ρ
\rho
ρ是策略
π
ρ
(
a
∣
s
)
≐
ρ
(
s
,
a
)
/
∑
a
′
ρ
(
s
,
a
′
)
\pi_{\rho}(a|s)\doteq\rho(s,a)/\sum_{a^{\prime}}\rho(s,a^{\prime})
πρ(a∣s)≐ρ(s,a)/∑a′ρ(s,a′)下的状态动作平稳分布,而
π
ρ
\pi_{\rho}
πρ是唯一
ρ
\rho
ρ的平稳策略。因此,问题 (5) 可以等价地写成以下形式:
max
ρ
≥
0
E
(
s
,
a
)
∼
ρ
[
R
~
(
s
,
a
)
]
−
D
K
L
(
ρ
∥
ρ
~
o
)
(9)
s
.
t
.
f
s
(
ρ
)
=
0
,
∀
s
∈
S
.
(10)
\max_{\rho\geq0}\mathbb{E}_{(s,a)\sim\rho}\big[\tilde{R}(s,a)\big]-D_{\mathrm{KL}}(\rho\|\tilde{\rho}^{o})~~~~\text{(9)}\\\mathrm{s.t.}f_{s}(\rho)=0,\forall s\in\mathcal{S}.~~~~~\text{(10)}
ρ≥0maxE(s,a)∼ρ[R~(s,a)]−DKL(ρ∥ρ~o) (9)s.t.fs(ρ)=0,∀s∈S. (10)
由于目标和约束分别在
ρ
\rho
ρ上是凹的和仿射的,问题 (9)-(10) 是一个凸优化问题。考虑上述问题的拉格朗日:
L
(
ρ
,
ν
)
≐
E
s
,
a
∼
ρ
[
R
~
(
s
,
a
)
]
−
D
K
L
(
ρ
∥
ρ
~
o
)
+
∑
s
ν
(
s
)
f
s
(
ρ
)
L(\rho,\nu)\doteq\mathbb{E}_{s,a\sim\rho}[\tilde{R}(s,a)]-D_{\mathrm{KL}}(\rho\|\tilde{\rho}^o)+\sum_s\nu(s)f_s(\rho)
L(ρ,ν)≐Es,a∼ρ[R~(s,a)]−DKL(ρ∥ρ~o)+s∑ν(s)fs(ρ)
其中
D
K
L
(
ρ
∥
ρ
~
o
)
=
∑
s
,
a
ρ
(
s
,
a
)
log
(
ρ
(
s
,
a
)
/
ρ
~
o
(
s
,
a
)
)
D_{\mathrm{KL}}(\rho\|\tilde{\rho}^{o})=\sum_{s,a}\rho(s,a)\log(\rho(s,a)/\tilde{\rho}^{o}(s,a))
DKL(ρ∥ρ~o)=∑s,aρ(s,a)log(ρ(s,a)/ρ~o(s,a)),进一步将上述等式
L
(
ρ
,
v
)
L(\rho,v)
L(ρ,v)转化为
L
(
ρ
,
ν
)
=
∑
s
,
a
ρ
(
s
,
a
)
(
δ
ν
(
s
,
a
)
−
log
ρ
(
s
,
a
)
ρ
~
o
(
s
,
a
)
)
+
(
1
−
γ
)
∑
s
ν
(
s
)
μ
(
s
)
(
11
)
L(\rho,\nu)=\sum_{s,a}\rho(s,a)\bigg(\delta_\nu(s,a)-\log\frac{\rho(s,a)}{\tilde{\rho}^o(s,a)}\bigg)+(1-\gamma)\sum_s\nu(s)\mu(s) ~~~~~(11)
L(ρ,ν)=s,a∑ρ(s,a)(δν(s,a)−logρ~o(s,a)ρ(s,a))+(1−γ)s∑ν(s)μ(s) (11)
其中
δ
ν
(
s
,
a
)
≐
R
~
(
s
,
a
)
+
γ
∑
s
′
ν
(
s
′
)
T
(
s
′
∣
s
,
a
)
−
ν
(
s
)
\delta_{\nu}(s,a)\doteq\tilde{R}(s,a)+\gamma\sum_{s'}\nu(s')T(s'|s,a)-\nu(s)
δν(s,a)≐R~(s,a)+γ∑s′ν(s′)T(s′∣s,a)−ν(s)
对其求最优
ρ
\rho
ρ, 令
∂
L
∂
ρ
(
s
,
a
)
=
0
\frac{\partial L}{\partial\rho(s,a)}=0
∂ρ(s,a)∂L=0得到
ρ
(
s
,
a
)
=
ρ
~
o
(
s
,
a
)
exp
(
δ
ν
(
s
,
a
)
−
1
)
.
(
13
)
\rho(s,a)=\tilde{\rho}^o(s,a)\exp\left(\delta_\nu(s,a)-1\right).\quad(13)
ρ(s,a)=ρ~o(s,a)exp(δν(s,a)−1).(13)
将等式(13)带入等式(11)得到关于v的优化问题
min
ν
L
(
ν
)
≐
E
(
s
,
a
)
∼
ρ
~
o
[
exp
(
δ
ν
(
s
,
a
)
−
1
)
]
+
(
1
−
γ
)
E
s
∼
μ
[
ν
(
s
)
]
.
(14)
\begin{aligned}\min_{\nu}L(\nu)&\doteq\mathbb{E}_{(s,a)\sim\tilde{\rho}^o}\left[\exp\left(\delta_\nu(s,a)-1\right)\right]+(1-\gamma)\mathbb{E}_{s\sim\mu}\left[\nu(s)\right].&\text{(14)}\end{aligned}
νminL(ν)≐E(s,a)∼ρ~o[exp(δν(s,a)−1)]+(1−γ)Es∼μ[ν(s)].(14)
直接优化问题 (14) 是有问题的,因为
δ
ν
\delta_\nu
δν中的期望由于双采样导致随机梯度存在偏差,且等式 (14) 中的指数项在实践中很容易导致数值不稳定。因此,文章引入凸共轭(convex conjugate)解决:
将x用
δ
ν
\delta_\nu
δν代替得到
exp
(
δ
ν
(
s
,
a
)
−
1
)
=
max
y
(
s
,
a
)
δ
ν
(
s
,
a
)
y
(
s
,
a
)
−
y
(
s
,
a
)
log
y
(
s
,
a
)
(
16
)
\begin{aligned}\exp\left(\delta_{\nu}(s,a)-1\right)&=\max_{y(s,a)}\delta_{\nu}(s,a)y(s,a)-y(s,a)\log y(s,a)\quad(16)\end{aligned}
exp(δν(s,a)−1)=y(s,a)maxδν(s,a)y(s,a)−y(s,a)logy(s,a)(16)
等式(16)带入(14)得到一个min-max问题
min
ν
max
y
F
(
ν
,
y
)
≐
E
(
s
,
a
)
∼
ρ
~
o
[
δ
ν
(
s
,
a
)
y
(
s
,
a
)
−
y
(
s
,
a
)
⋅
log
y
(
s
,
a
)
]
+
(
1
−
γ
)
E
s
∼
μ
[
ν
(
s
)
]
.
(
17
)
\min\limits_\nu\max\limits_yF(\nu,y)\doteq\mathbb{E}_{(s,a)\sim\tilde{\rho}^o}\big[\delta_\nu(s,a)y(s,a)-y(s,a)\cdot\log y(s,a)\Big]+(1-\gamma)\mathbb{E}_{s\sim\mu}[\nu(s)].\quad(17)
νminymaxF(ν,y)≐E(s,a)∼ρ~o[δν(s,a)y(s,a)−y(s,a)⋅logy(s,a)]+(1−γ)Es∼μ[ν(s)].(17)
由于 F (·, y) 在固定 y 下是凸的,并且 F (ν, ·) 在固定 ν 的凹的,极大极小定理成立 ,问题 (17) 实际上是一个凸凹随机鞍点 (SSP) 问题。对于固定转换
(
s
,
a
,
s
′
)
(s, a, s')
(s,a,s′),将
δ
ν
(
s
,
a
,
s
′
)
\delta_\nu(s, a, s')
δν(s,a,s′)表示为
δ
~
ν
(
s
,
a
,
s
′
)
≐
R
~
(
s
,
a
)
+
γ
ν
(
s
′
)
−
ν
(
s
)
.
(
18
)
\tilde{\delta}_\nu(s,a,s')\doteq\tilde{R}(s,a)+\gamma\nu(s')-\nu(s).\quad(18)
δ~ν(s,a,s′)≐R~(s,a)+γν(s′)−ν(s).(18).
那(17)的无偏对应为:
min
ν
max
y
F
~
(
ν
,
y
)
≐
E
(
s
,
a
,
s
′
)
∼
D
o
[
δ
~
ν
(
s
,
a
,
s
′
)
y
(
s
,
a
)
−
y
(
s
,
a
)
log
y
(
s
,
a
)
]
+
(
1
−
γ
)
E
s
∼
D
o
(
s
0
)
[
ν
(
s
)
]
.
(
19
)
\min_{\nu}\max_{y}\tilde{F}(\nu,y)\doteq\mathbb{E}_{(s,a,s^{\prime})\sim\mathcal{D}_{o}}\big[\tilde{\delta}_{\nu}(s,a,s^{\prime})y(s,a)-y(s,a)\log y(s,a)\big]+(1-\gamma)\mathbb{E}_{s\sim\mathcal{D}_{o}(s_{0})}[\nu(s)].(19)
νminymaxF~(ν,y)≐E(s,a,s′)∼Do[δ~ν(s,a,s′)y(s,a)−y(s,a)logy(s,a)]+(1−γ)Es∼Do(s0)[ν(s)].(19)
Offline Policy Extraction
根据等式(13)可以获得最优策略与状态动作分布见存在如下关系
ρ
∗
(
s
,
a
)
=
ρ
~
o
(
s
,
a
)
exp
(
δ
ν
∗
(
s
,
a
)
−
1
)
.
(
20
)
\rho^*(s,a)=\tilde{\rho}^o(s,a)\exp\left(\delta_{\nu^*}(s,a)-1\right).\quad(20)
ρ∗(s,a)=ρ~o(s,a)exp(δν∗(s,a)−1).(20)
π
∗
(
a
∣
s
)
=
ρ
∗
(
s
,
a
)
∑
a
′
ρ
∗
(
s
,
a
′
)
∝
ρ
~
o
(
s
,
a
)
exp
(
δ
ν
∗
(
s
,
a
)
−
1
)
.
(
21
)
\pi^*(a|s)=\frac{\rho^*(s,a)}{\sum_{a^{\prime}}\rho^*(s,a^{\prime})}\propto\tilde{\rho}^o(s,a)\exp\left(\delta_{\nu^*}(s,a)-1\right).\quad\quad(21)
π∗(a∣s)=∑a′ρ∗(s,a′)ρ∗(s,a)∝ρ~o(s,a)exp(δν∗(s,a)−1).(21)
当
∂
F
∂
y
(
s
,
a
)
=
0
\frac{\partial F}{\partial y(s,a)}=0
∂y(s,a)∂F=0,可以得到
y
∗
(
s
,
a
)
=
exp
(
δ
ν
∗
(
s
,
a
)
−
1
)
(
22
)
y^*(s,a)=\exp(\delta_{\nu^*}(s,a)-1) \quad (22)
y∗(s,a)=exp(δν∗(s,a)−1)(22)
因此
π
∗
(
a
∣
s
)
=
ρ
~
o
(
s
,
a
)
y
∗
(
s
,
a
)
z
(
s
)
(
23
)
\pi^*(a|s)=\frac{\tilde{\rho}^o(s,a)y^*(s,a)}{z(s)}\quad(23)
π∗(a∣s)=z(s)ρ~o(s,a)y∗(s,a)(23)
那么策略优化可以采用两种形式
- Reverse KL-divergence
类似SAC中的重参数策略优化 min π J ( π ) = E s ∼ D o [ D KL ( π ( ⋅ ∣ s ) ∥ q ( s , ⋅ ) z ( s ) ) ] ( 25 ) \min_\pi J(\pi)=\mathbb{E}_{s\sim\mathcal{D}_o}\left[D_{\text{KL}}\left(\pi(\cdot|s)\left\|\frac{q(s,\cdot)}{z(s)}\right)\right]\quad(25)\right. minπJ(π)=Es∼Do[DKL(π(⋅∣s) z(s)q(s,⋅))](25)。其中公式(7)带入公式(23)得到 q ( s , a ) ≐ ρ ~ e ( s , a ) y ∗ ( s , a ) ( 1 d ∗ ( s , a ) − 1 ) q(s,a)\doteq\tilde{\rho}^e(s,a)y^*(s,a)\left(\frac{1}{d^*(s,a)}-1\right) q(s,a)≐ρ~e(s,a)y∗(s,a)(d∗(s,a)1−1)。 - Forward KL-divergence
E s ∼ ρ ∗ [ D K L ( π ∗ ( ⋅ ∣ s ) ∥ π ( ⋅ ∣ s ) ) ] = E s ∼ ρ ∗ [ E a ∼ π ∗ ( ⋅ ∣ s ) [ log π ∗ ( a ∣ s ) − log π ( a ∣ s ) ] ] ⇔ E ( s , a ) ∼ ρ ∗ [ − log π ( a ∣ s ) ] = E ( s , a ) ∼ ρ ~ o [ − ρ ∗ ( s , a ) ρ ~ o ( s , a ) log π ( a ∣ s ) ] (26) \begin{aligned} &\mathbb{E}_{s\sim\rho^*}\left[D_{\mathrm{KL}}(\pi^*(\cdot|s)\|\pi(\cdot|s))\right] \\ &=\mathbb{E}_{s\sim\rho^*}\left[\mathbb{E}_{a\sim\pi^*(\cdot|s)}\left[\log\pi^*(a|s)-\log\pi(a|s)\right]\right] \\ &\Leftrightarrow\mathbb{E}_{(s,a)\sim\rho^*}\begin{bmatrix}-\log\pi(a|s)\end{bmatrix} \\ &=\mathbb{E}_{(s,a)\sim\tilde{\rho}^{o}}\left[-\frac{\rho^{*}(s,a)}{\tilde{\rho}^{o}(s,a)}\log\pi(a|s)\right]& \text{(26)} \end{aligned} Es∼ρ∗[DKL(π∗(⋅∣s)∥π(⋅∣s))]=Es∼ρ∗[Ea∼π∗(⋅∣s)[logπ∗(a∣s)−logπ(a∣s)]]⇔E(s,a)∼ρ∗[−logπ(a∣s)]=E(s,a)∼ρ~o[−ρ~o(s,a)ρ∗(s,a)logπ(a∣s)](26)
由于 ρ ∗ ( s , a ) ρ ~ o ( s , a ) = exp ( δ ν ∗ ( s , a ) − 1 ) = y ∗ ( s , a ) . ( 27 ) \frac{\rho^*(s,a)}{\tilde{\rho}^o(s,a)}=\exp\left(\delta_{\nu^*}(s,a)-1\right)=y^*(s,a).\quad(27) ρ~o(s,a)ρ∗(s,a)=exp(δν∗(s,a)−1)=y∗(s,a).(27),问题转化为加权模仿学习的优化问题
max π J ( π ) = E ( s , a ) ∼ D o [ y ∗ ( s , a ) log π ( a ∣ s ) ] ( 28 ) \max_\pi J(\pi)=\mathbb{E}_{(s,a)\sim\mathcal{D}_o}\begin{bmatrix}y^*(s,a)\log\pi(a|s)\end{bmatrix}\quad(28) πmaxJ(π)=E(s,a)∼Do[y∗(s,a)logπ(a∣s)](28)
Aligned Discriminator
离线获得最优策略
π
∗
\pi^*
π∗后,便是解决在线判别器对齐问题。对于GAIL,
D
0
(
s
,
a
)
≐
ρ
∗
(
s
,
a
)
ρ
∗
(
s
,
a
)
+
ρ
~
e
(
s
,
a
)
=
(
1
+
ρ
~
e
(
s
,
a
)
ρ
∗
(
s
,
a
)
)
−
1
(
29
)
D_0(s,a)\doteq\frac{\rho^*(s,a)}{\rho^*(s,a)+\tilde{\rho}^e(s,a)}=\left(1+\frac{\tilde{\rho}^e(s,a)}{\rho^*(s,a)}\right)^{-1}(29)
D0(s,a)≐ρ∗(s,a)+ρ~e(s,a)ρ∗(s,a)=(1+ρ∗(s,a)ρ~e(s,a))−1(29)
由等式(20)(22)对上式推到得到
D
0
(
s
,
a
)
=
(
1
+
ρ
~
e
(
s
,
a
)
ρ
~
o
(
s
,
a
)
⋅
ρ
~
o
(
s
,
a
)
ρ
∗
(
s
,
a
)
)
−
1
=
(
1
+
d
∗
(
s
,
a
)
1
−
d
∗
(
s
,
a
)
⋅
1
exp
(
δ
ν
∗
(
s
,
a
)
−
1
)
)
−
1
=
(
1
+
d
∗
(
s
,
a
)
1
−
d
∗
(
s
,
a
)
⋅
1
y
∗
(
s
,
a
)
)
−
1
.
(
30
)
\begin{aligned} D_{0}(s,a)& =\left(1+\frac{\tilde{\rho}^{e}(s,a)}{\tilde{\rho}^{o}(s,a)}\cdot\frac{\tilde{\rho}^{o}(s,a)}{\rho^{*}(s,a)}\right)^{-1} \\ &=\left(1+\frac{d^*(s,a)}{1-d^*(s,a)}\cdot\frac{1}{\exp\left(\delta_{\nu^*}(s,a)-1\right)}\right)^{-1} \\ &=\left(1+\frac{d^*(s,a)}{1-d^*(s,a)}\cdot\frac1{y^*(s,a)}\right)^{-1}.\quad(30) \end{aligned}
D0(s,a)=(1+ρ~o(s,a)ρ~e(s,a)⋅ρ∗(s,a)ρ~o(s,a))−1=(1+1−d∗(s,a)d∗(s,a)⋅exp(δν∗(s,a)−1)1)−1=(1+1−d∗(s,a)d∗(s,a)⋅y∗(s,a)1)−1.(30)
判别器的输出天然与离线下学习的
d
d
d以及
y
y
y挂钩,自然而然实现对齐。
Implementation with Function Approximation
在实际操作中,使用参数化模型对
ϕ
,
d
,
ν
\phi, d , \nu
ϕ,d,ν建模。使用前向策略优化(Forward KL-divergence)。在在线微调过程中,通过离线学习 得到的由参数
ϕ
d
\phi_d
ϕd 和
ϕ
y
\phi_y
ϕy 构造的
d
d
d与
y
y
y来初始化判别器:
D
ϕ
y
,
ϕ
d
(
s
,
a
)
=
(
1
+
ϕ
d
(
s
,
a
)
1
−
ϕ
d
(
s
,
a
)
⋅
1
ϕ
y
(
s
,
a
)
)
−
1
.
(31)
D_{\phi_y,\phi_d}(s,a)=\left(1+\frac{\phi_d(s,a)}{1-\phi_d(s,a)}\cdot\frac{1}{\phi_y(s,a)}\right)^{-1}.\text{(31)}
Dϕy,ϕd(s,a)=(1+1−ϕd(s,a)ϕd(s,a)⋅ϕy(s,a)1)−1.(31)
伪代码
结果
Offline to Online
不同数量专家数据下进行在线微调
后续还有AntMaze、FrankaKitchen.以及vision-based输入下的微调结果。效果都非常好。