Variational Inference Review

最新推荐文章于 2023-03-22 21:33:46 发布

一个不愿透露姓名的孩子

最新推荐文章于 2023-03-22 21:33:46 发布

阅读量233

点赞数

分类专栏：统计学

本文链接：https://blog.csdn.net/weixin_44207974/article/details/114363964

版权

变分推断是一种通过找到密度族中最接近目标密度的分布来近似复杂密度的方法。相对于MCMC，它更易于计算且适合大规模数据。本文主要介绍了变分推断的问题、变分目标函数、平均场变分家族以及坐标上升变分推断（CAVI），并通过贝叶斯高斯混合模型举例说明。

摘要由CSDN通过智能技术生成

Variational Inference Review

Idea: posit a family of density and find one from the family that is the closest (in K-L divergence) to the target density.
For statisticians: VI provides a method to approximate complicate densities. Compared with MCMC, it’s easier to compute and scale to big data.

Problem of Approximate Inference

$x$ -observations
$z$ -latent variables
want to estimate
$p(z|x)=\frac{p(z,x)}{p(x)}$

pick a family of distribution $\mathcal{Q}$ , and approximate $p (z ∣ x)$ by
$q^*(z)=\argmin_{q(z) \in \mathcal{Q}}\ KL(q(z)||p(z|x))$

Variational Object Function

$KL(q(z)||p(z|x))=E_q[\log q(z)]-E_q[\log p(z|x)] \\ = E_q[\log q(z)]-E_q[\log p(z,x)]+\log p(x) \\ = -ELBO(q)+const.$

ELBO = evidence lower bound, and minimizing K-L is equivalent to maximizing ELBO:
$ELBO(q)=E_q[\log p(z,x)]-E_q[\log q(z)] \\ = E_q[\log p(z)]+E_q[\log p(x|z)]-E_q[\log q(z)] \\ = \underbrace{E[\log p(x|z)]}_{expected\ likehood}-\underbrace{KL(q(z)||p(z))}_{penalize\ deviation\ from\ prior}$