在IWAE中仅仅考虑了两层的隐藏层,但是随着层数多了,高层很多都训练不好,因此提出了LVAE。首先比较下模型与IWAE的区别。各层
建模如下
p
θ
(
z
)
=
p
θ
(
z
L
)
∏
i
=
1
L
−
1
p
θ
(
z
i
∣
z
i
+
1
)
p
θ
(
z
i
∣
z
i
+
1
)
=
N
(
z
∣
μ
p
,
i
(
z
i
+
1
)
,
σ
p
,
i
2
(
z
i
+
1
)
)
,
p
θ
(
z
L
)
=
N
(
z
L
∣
0
,
I
)
p
θ
(
x
∣
z
1
)
=
N
(
x
∣
μ
p
,
0
(
z
1
)
,
σ
p
,
0
2
(
z
1
)
)
or
P
θ
(
x
∣
z
1
)
=
B
(
x
∣
μ
p
,
0
(
z
1
)
)
\begin{aligned} p_{\theta}(\mathbf{z}) &=p_{\theta}\left(\mathbf{z}_{L}\right) \prod_{i=1}^{L-1} p_{\theta}\left(\mathbf{z}_{i} | \mathbf{z}_{i+1}\right) \\ p_{\theta}\left(\mathbf{z}_{i} | \mathbf{z}_{i+1}\right) &=\mathcal{N}\left(\mathbf{z} | \mu_{p, i}\left(\mathbf{z}_{i+1}\right), \sigma_{p, i}^{2}\left(\mathbf{z}_{i+1}\right)\right), \quad p_{\theta}\left(\mathbf{z}_{L}\right)=\mathcal{N}\left(\mathbf{z}_{L} | \mathbf{0}, \mathbf{I}\right) \\ p_{\theta}\left(\mathbf{x} | \mathbf{z}_{1}\right) &=\mathcal{N}\left(\mathbf{x} | \mu_{p, 0}\left(\mathbf{z}_{1}\right), \sigma_{p, 0}^{2}\left(\mathbf{z}_{1}\right)\right) \text { or } P_{\theta}\left(\mathbf{x} | \mathbf{z}_{1}\right)=\mathcal{B}\left(\mathbf{x} | \mu_{p, 0}\left(\mathbf{z}_{1}\right)\right) \end{aligned}
pθ(z)pθ(zi∣zi+1)pθ(x∣z1)=pθ(zL)i=1∏L−1pθ(zi∣zi+1)=N(z∣μp,i(zi+1),σp,i2(zi+1)),pθ(zL)=N(zL∣0,I)=N(x∣μp,0(z1),σp,02(z1)) or Pθ(x∣z1)=B(x∣μp,0(z1))一般的VAE的inference部分建模如下
q
ϕ
(
z
∣
x
)
=
q
ϕ
(
z
1
∣
x
)
∏
i
=
2
L
q
ϕ
(
z
i
∣
z
i
−
1
)
q
ϕ
(
z
1
∣
x
)
=
N
(
z
1
∣
μ
q
,
1
(
x
)
,
σ
q
,
1
2
(
x
)
)
q
ϕ
(
z
i
∣
z
i
−
1
)
=
N
(
z
i
∣
μ
q
,
i
(
z
i
−
1
)
,
σ
q
,
i
2
(
z
i
−
1
)
)
,
i
=
2
…
L
\begin{aligned} q_{\phi}(\mathbf{z} | \mathbf{x}) &=q_{\phi}\left(\mathbf{z}_{1} | \mathbf{x}\right) \prod_{i=2}^{L} q_{\phi}\left(\mathbf{z}_{i} | \mathbf{z}_{i-1}\right) \\ q_{\phi}\left(\mathbf{z}_{1} | \mathbf{x}\right) &=\mathcal{N}\left(\mathbf{z}_{1} | \mu_{q, 1}(\mathbf{x}), \sigma_{q, 1}^{2}(\mathbf{x})\right) \\ q_{\phi}\left(\mathbf{z}_{i} | \mathbf{z}_{i-1}\right) &=\mathcal{N}\left(\mathbf{z}_{i} | \mu_{q, i}\left(\mathbf{z}_{i-1}\right), \sigma_{q, i}^{2}\left(\mathbf{z}_{i-1}\right)\right), i=2 \ldots L \end{aligned}
qϕ(z∣x)qϕ(z1∣x)qϕ(zi∣zi−1)=qϕ(z1∣x)i=2∏Lqϕ(zi∣zi−1)=N(z1∣μq,1(x),σq,12(x))=N(zi∣μq,i(zi−1),σq,i2(zi−1)),i=2…L但是这样训练会出现问题,难以优化,因此本文提出一个新的inference模型。首先用正常的神经网络得到
d
n
=
MLP
(
d
n
−
1
)
μ
^
q
,
i
=
Linear
(
d
i
)
,
i
=
1
…
L
σ
^
q
,
i
2
=
Softplus
(
Linear
(
d
i
)
)
,
i
=
1
…
L
\begin{aligned} \mathbf{d}_{n} &=\operatorname{MLP}\left(\mathbf{d}_{n-1}\right) \\ \hat{\mu}_{q, i} &=\operatorname{Linear}\left(\mathbf{d}_{i}\right), i=1 \ldots L \\ \hat{\sigma}_{q, i}^{2} &=\operatorname{Softplus}\left(\operatorname{Linear}\left(\mathbf{d}_{i}\right)\right), i=1 \ldots L \end{aligned}
dnμ^q,iσ^q,i2=MLP(dn−1)=Linear(di),i=1…L=Softplus(Linear(di)),i=1…L其中
d
0
=
x
\mathbf{d}_{0}=\mathbf{x}
d0=x。然后
q
ϕ
(
z
∣
x
)
=
q
ϕ
(
z
L
∣
x
)
∏
i
=
1
L
−
1
q
ϕ
(
z
i
∣
z
i
+
1
,
x
)
σ
q
,
i
=
1
σ
^
q
,
i
−
2
+
σ
p
,
i
−
2
μ
q
,
i
=
μ
^
q
,
i
σ
^
q
,
i
−
2
+
μ
p
,
i
σ
p
,
i
−
2
σ
^
q
,
i
−
2
+
σ
p
,
i
−
2
q
ϕ
(
z
i
∣
⋅
)
=
N
(
z
i
∣
μ
q
,
i
,
σ
q
,
i
2
)
\begin{array}{c}{q_{\phi}(\mathbf{z} | \mathbf{x})=q_{\phi}\left(\mathbf{z}_{L} | \mathbf{x}\right) \prod_{i=1}^{L-1} q_{\phi}\left(\mathbf{z}_{i} | \mathbf{z}_{i+1}, \mathbf{x}\right)} \\ {\sigma_{q, i}=\frac{1}{\hat{\sigma}_{q, i}^{-2}+\sigma_{p, i}^{-2}}} \\ {\mu_{q, i}=\frac{\hat{\mu}_{q, i} \hat{\sigma}_{q, i}^{-2}+\mu_{p, i} \sigma_{p, i}^{-2}}{\hat{\sigma}_{q, i}^{-2}+\sigma_{p, i}^{-2}}} \\ {q_{\phi}\left(\mathbf{z}_{i} | \cdot\right)=\mathcal{N}\left(\mathbf{z}_{i} | \mu_{q, i}, \sigma_{q, i}^{2}\right)}\end{array}
qϕ(z∣x)=qϕ(zL∣x)∏i=1L−1qϕ(zi∣zi+1,x)σq,i=σ^q,i−2+σp,i−21μq,i=σ^q,i−2+σp,i−2μ^q,iσ^q,i−2+μp,iσp,i−2qϕ(zi∣⋅)=N(zi∣μq,i,σq,i2)其中
μ
q
,
L
=
μ
^
q
,
L
\mu_{q, L}=\hat{\mu}_{q, L}
μq,L=μ^q,L,
σ
q
,
L
2
=
σ
^
q
,
L
2
\sigma_{q, L}^{2}=\hat{\sigma}_{q, L}^{2}
σq,L2=σ^q,L2。可以看出
σ
q
,
i
,
μ
q
,
i
\sigma_{q, i},\mu_{q, i}
σq,i,μq,i由生成模型(“先验”)的
σ
p
,
i
,
μ
p
,
i
\sigma_{p,i},\mu_{p,i}
σp,i,μp,i和(“似然”)
σ
^
q
,
i
,
μ
^
q
,
i
\hat{\sigma}_{q,i},\hat{\mu}_{q, i}
σ^q,i,μ^q,i组成“后验”。
随着网络的加深,有些隐变量会变得uninformative,这种uninformative是从一开始就变成这样,一旦uninformative就会一直uninformative,随着训练的进行不会再激活它们。本文中先以standard deterministic auto-encoder进行参数初始化,然后
L
(
θ
,
ϕ
;
x
)
W
U
=
−
β
K
L
(
q
ϕ
(
z
∣
x
)
∥
p
θ
(
z
)
)
+
E
q
ϕ
(
z
∣
x
)
[
log
p
θ
(
x
∣
z
)
]
\mathcal{L}(\theta, \phi ; \mathbf{x})_{W U}=-\beta K L\left(q_{\phi}(z | x) \| p_{\theta}(\mathbf{z})\right)+E_{q_{\phi}(z | x)}\left[\log p_{\theta}(\mathbf{x} | \mathbf{z})\right]
L(θ,ϕ;x)WU=−βKL(qϕ(z∣x)∥pθ(z))+Eqϕ(z∣x)[logpθ(x∣z)]其中
β
\beta
β为前
N
t
N_t
Nt线性从0增加到1,这种方式称为warm-up(WU)。
从上面的效果可以看出,本文的方法能够得到更多有意义的神经元。不仅如此,可视化效果可以看出,LVAE能够学习到结构性的高层特征
Ladder VAE
最新推荐文章于 2023-11-22 10:47:19 发布