动机
当我们存在隐变量的时候,我们从数据中观察到的
p
(
x
)
p(x)
p(x),可能是有隐变量z所影响得到的
p
(
x
∣
z
)
p(x|z)
p(x∣z),当我们要计算
L
(
θ
)
=
∑
i
=
1
N
log
(
p
(
x
i
∣
θ
)
)
=
∑
i
=
1
N
log
[
∑
z
i
p
(
x
i
,
z
i
∣
θ
)
]
=
∑
i
=
1
N
log
[
∑
z
i
[
p
(
x
i
∣
z
i
,
θ
)
p
(
z
i
∣
θ
)
]
]
\begin{aligned} \mathcal{L(\theta)}&=\sum_{i=1}^{N}\log(p(x_i|\theta))\\&=\sum_{i=1}^{N}\log \Bigl[\sum_{z_i}p(x_i,z_i|\theta) \Bigr]\\&=\sum_{i=1}^{N}\log \Bigl[\sum_{z_i}\bigl[p(x_i|z_i,\theta)p(z_i|\theta)\bigr] \Bigr] \end{aligned}
L(θ)=i=1∑Nlog(p(xi∣θ))=i=1∑Nlog[zi∑p(xi,zi∣θ)]=i=1∑Nlog[zi∑[p(xi∣zi,θ)p(zi∣θ)]]
如果p是个指数族分布,log就可以直接作用于p,似然度就很好求,但是log被
∑
z
\sum_z
∑z截断,又因为z是隐变量,我们不知道z的取值,所以需要找其他的方法去优化。
想法
我们引入一个
q
i
(
z
i
)
q_i(z_i)
qi(zi)代表
z
i
z_i
zi的分布
log
p
(
x
i
∣
θ
)
=
log
p
(
x
i
,
z
i
∣
θ
)
−
log
p
(
z
i
∣
x
i
,
θ
)
=
log
p
(
x
i
,
z
i
∣
θ
)
q
i
(
z
i
)
−
log
p
(
z
i
∣
x
i
,
θ
)
q
i
(
z
i
)
=
log
p
(
x
i
,
z
i
∣
θ
)
−
log
q
i
(
z
i
)
−
log
p
(
z
i
∣
x
i
,
θ
)
q
i
(
z
i
)
=
∫
q
i
(
z
i
)
log
p
(
x
i
,
z
i
∣
θ
)
d
z
−
∫
q
i
(
z
i
)
log
q
i
(
z
i
)
d
z
−
∫
q
i
(
z
i
)
log
p
(
z
i
∣
x
i
,
θ
)
q
i
(
z
i
)
d
z
=
E
z
i
(
log
p
(
x
i
,
z
i
∣
θ
)
)
+
H
(
z
i
)
⏟
ELBO
+
K
L
(
q
i
(
z
i
)
∣
∣
p
(
z
i
∣
x
i
,
θ
)
)
\begin{aligned} \log p(x_i|\theta) &= \log p(x_i,z_i|\theta)-\log p(z_i|x_i,\theta)\\ &= \log \frac{p(x_i,z_i|\theta)}{q_i(z_i)} - \log \frac{p(z_i|x_i,\theta)}{q_i(z_i)}\\ &= \log p(x_i,z_i|\theta) - \log q_i(z_i) - \log \frac{p(z_i|x_i,\theta)}{q_i(z_i)}\\ &= \int q_i(z_i)\log p(x_i,z_i|\theta) \mathrm{d}z - \int q_i(z_i)\log q_i(z_i) \mathrm{d}z - \int q_i(z_i)\log \frac{p(z_i|x_i,\theta)}{q_i(z_i)} \mathrm{d}z \\ &= \underbrace{E_{z_i}(\log p(x_i,z_i|\theta)) + H(z_i)}_\text{ELBO} + KL(q_i(z_i)||p(z_i|x_i,\theta)) \end{aligned}
logp(xi∣θ)=logp(xi,zi∣θ)−logp(zi∣xi,θ)=logqi(zi)p(xi,zi∣θ)−logqi(zi)p(zi∣xi,θ)=logp(xi,zi∣θ)−logqi(zi)−logqi(zi)p(zi∣xi,θ)=∫qi(zi)logp(xi,zi∣θ)dz−∫qi(zi)logqi(zi)dz−∫qi(zi)logqi(zi)p(zi∣xi,θ)dz=ELBO
Ezi(logp(xi,zi∣θ))+H(zi)+KL(qi(zi)∣∣p(zi∣xi,θ))
因为对于KL散度是非负数,那么就有
log
p
(
x
i
∣
θ
)
≥
E
z
i
(
p
(
x
i
,
z
i
∣
θ
)
)
+
H
(
z
i
)
\log p(x_i|\theta) \geq E_{z_i}(p(x_i,z_i|\theta)) + H(z_i)
logp(xi∣θ)≥Ezi(p(xi,zi∣θ))+H(zi)
那么我们让KL散度为0,也就是
q
i
(
z
i
)
=
p
(
z
i
∣
x
i
,
θ
)
q_i(z_i)=p(z_i|x_i,\theta)
qi(zi)=p(zi∣xi,θ),我们所要求的似然值就等于
E
z
i
(
p
(
x
i
,
z
i
∣
θ
)
)
+
H
(
z
i
)
E_{z_i}(p(x_i,z_i|\theta)) + H(z_i)
Ezi(p(xi,zi∣θ))+H(zi),也就是ELBO,那么我们接下来的任务就是最大化ELBO。
对于
H
(
z
i
)
H(z_i)
H(zi)来说是个常数,我们不用管,那么最后的任务就是最大化
E
z
i
(
p
(
x
i
,
z
i
∣
θ
)
)
E_{z_i}(p(x_i,z_i|\theta))
Ezi(p(xi,zi∣θ))。
又因为我们让KL散度等于0,存在
q
i
(
z
i
)
=
p
(
z
i
∣
x
i
,
θ
)
q_i(z_i)=p(z_i|x_i,\theta)
qi(zi)=p(zi∣xi,θ)
E
L
B
O
i
=
E
z
i
(
p
(
x
i
,
z
i
∣
θ
)
)
+
c
o
n
s
t
=
∫
q
i
(
z
i
)
log
p
(
x
i
,
z
i
∣
θ
)
d
z
+
c
o
n
s
t
=
∫
p
(
z
i
∣
x
i
,
θ
)
log
p
(
x
i
,
z
i
∣
θ
)
d
z
+
c
o
n
s
t
\begin{aligned} ELBO_i&=E_{z_i}(p(x_i,z_i|\theta))+const\\ &= \int q_i(z_i)\log p(x_i,z_i|\theta) \mathrm{d}z+const\\ &= \int p(z_i|x_i,\theta)\log p(x_i,z_i|\theta) \mathrm{d}z+const \end{aligned}
ELBOi=Ezi(p(xi,zi∣θ))+const=∫qi(zi)logp(xi,zi∣θ)dz+const=∫p(zi∣xi,θ)logp(xi,zi∣θ)dz+const
嘴炮实现
利用初始化的 θ \theta θ计算上面的式子,再根据上面的式子更新 θ \theta θ,迭代到收敛为止。
目前的想法
貌似要知道隐变量到底是如何作用到结果的,感觉有点奇怪。而且初值敏感好像很多迭代的算法对初值敏感。