变分推断variational Inference

十一、变分推断

1.背景

$$\{\begin{array}{l}频率角度，优化问题\{\begin{array}{l}回归\{\begin{array}{l}Model\\ 策略\\ 算法\{\begin{array}{l}解析解\\ 数值解\end{array}\end{array}\\ SVM\\ EM\\ 等等\end{array}\\ 贝叶斯角度，积分问题\{\begin{array}{l}贝叶斯Infernece(求贝叶斯后验)\\ P(\theta \mid x)=\frac{P(x\mid \theta )P(\theta )}{P(x)}\\ 贝叶斯决策(预测，最后还是求贝叶斯后验)\\ P(\tilde{x}\mid x)={\int }_{\theta }P(\tilde{x},\theta \mid x)d\theta ={\int }_{\theta }P(\tilde{x}\mid \theta )P(\theta \mid x)d\theta =E_{\theta \mid x}[P(\tilde{x}\mid \theta )]\end{array}\end{array}$$

$$Inference\{\begin{array}{l}精确推断（后验简单）\\ 近似推断/近似推断的期望(参数空间、隐变量非常复杂)\\ \{\begin{array}{l}确定性近似\to VI\\ 随机近似\to MCMC,MH,Gibbs\end{array}\end{array}$$

2.公式推导

$x$:observed data
$z$:later variable + parameter
$(x,z)$:complete data

ELBO + KL
$$\log P(x)=L(q)+KL(q||p)$$
$\hat{q}(z)=\arg {\max }_{q(z)}L(q)\to \hat{q}(z)\approx p(z\mid x)$

基于物理的平均场理论
$q(z)={\prod }_{i=1}^M{q}_i(z_i)$，计算时固定一维$q_j(z_j)$
$$L(q)={\int }_{z}q(z)\log P(x,z)dz-{\int }_{z}q(z)\log q(z)dz$$

∫ z q ( z ) log ⁡ P ( x , z ) d z = ∫ z ∏ i = 1 M q i ( z i ) log ⁡ P ( x , z ) d z = ∫ z j q j ( z j ) d z j ( ∫ z i ∏ i M q i ( z i ) log ⁡ P ( x , z ) d z i ) ( i ≠ j