期望传播算法及其推导

个人博客：www.qiuyun-blog.cn

Notations:

1. $\text{Diag}(\mathbf{a})$: a diagonal matrix with $\mathbf{a}$ being its diagonal element.
2. $\text{diag}(\mathbf{A})$: a vector from the diagonal element of $\mathbf{A}$.
3. $\mathbf{a}\odot \mathbf{b}$: componentwise multiply.
4. $\mathbf{a}\oslash \mathbf{b}$: componentwise divide.

Recap of Variational Inference

As mentioned in [1], we have introduced variational inference and its application in Bayesian linear regression. In this blog, we focus on a variational inference perspective on expectation propagation.

In signal processing regime, the posterior distribution is interested. However, it is difficult to obtain owing to many high-dimension integral. For example, we consider linear Gaussian model
$$\mathbf{y}=\mathbf{H}\mathbf{x}+\mathbf{w}$$
Its posterior distribution denoted by
$$p(\mathbf{x}|\mathbf{y})=\frac{p(\mathbf{y}|\mathbf{x})p(\mathbf{x})}{\int p(\mathbf{y}|\mathbf{x})p(\mathbf{x})\text{d}\mathbf{x}}$$
where $p(\mathbf{y}|\mathbf{x})=p_{\mathbf{w}}(\mathbf{y}-\mathbf{H}\mathbf{x})$. Unless both $p(\mathbf{y}|\mathbf{x})$ and $p(\mathbf{x})$ are Gaussian, we can’t obtain the close-form of $p(\mathbf{x}|\mathbf{y})$ directly. For that, some approximations are necessary.

To thid end, we use $q(\mathbf{x})$ to approximate the posterior distribution and KL-divergence to measure the difference between $q(\mathbf{x})$ and $p(\mathbf{x}|\mathbf{y})$. For simplification, we generally restrict the form of $q(\mathbf{x})$ from the distribution family $\mathcal{S}$, i.e.,
$$q(\mathbf{x})=\underset{q(\mathbf{x})\in \mathcal{S}}{\mathrm{argmin}}{\mathcal{D}}_{\text{KL}}(p||q)$$
Obviously, a distribution family with excellent properties will greatly reduce the amount of computation. Fortunately, exponential family is one of that.

Exponential Family

The exponential family over $\mathbf{x}$ parametered by $\mathbf{\eta }$ is defined by
$$p(\mathbf{x};\mathbf{\eta })=h(\mathbf{x})g(\mathbf{\eta })\exp \left({\mathbf{\eta }}^T\mathbf{u}(\mathbf{x})\right)$$
where $g(\mathbf{\eta })$ is normalization constant
$$g(\mathbf{\eta })\left(\int h(\mathbf{x})\exp \left({\mathbf{\eta }}^T\mathbf{u}(\mathbf{x})\right)\text{d}\mathbf{x}\right)=1$$
Taking the gradient of both side of the above w.r.t. $\mathbf{\eta }$, we get
$$\nabla g(\mathbf{\eta })\int h(\mathbf{x})\exp \left({\mathbf{\eta }}^T\mathbf{u}(\mathbf{x})\right)\text{d}\mathbf{x}+g(\mathbf{\eta })\int h(\mathbf{x})\left({\mathbf{\eta }}^T\mathbf{u}(\mathbf{x})\right)\mathbf{u}(\mathbf{x})\text{d}\mathbf{x}=0$$
Rearranging yields
$$\begin{gathered} -\frac{1}{g(\mathbf{\eta })}\nabla g(\mathbf{\eta })=g(\mathbf{\eta })\int \mathbf{u}(\mathbf{x})h(\mathbf{x})\exp \left({\mathbf{\eta }}^T\mathbf{u}(\mathbf{x})\right)\text{d}\mathbf{x} \\ =\frac{\int \mathbf{u}(\mathbf{x})h(\mathbf{x})\exp \left({\mathbf{\eta }}^T\mathbf{u}(\mathbf{x})\right)\text{d}\mathbf{x}}{\int h(\mathbf{x})\exp \left({\mathbf{\eta }}^T\mathbf{u}(\mathbf{x})\right)\text{d}\mathbf{x}} \\ =\mathbb{E}[\mathbf{u}(\mathbf{x})] \end{gathered}$$
Using the fact $\nabla \log g(\mathbf{\eta })=\frac{1}{g(\mathbf{\eta })}\nabla g(\mathbf{\eta })$, we have
$$-\nabla \log g(\mathbf{\eta })=\mathbb{E}[\mathbf{u}(\mathbf{x})]\cdots \cdots \cdots \cdots (\ast 1)$$

A Variational Inference Perspective on EP

For the distribution of $q(\mathbf{x})$ in variational inference, We take exponential family distribution into account
$$q(\mathbf{x})=h(\mathbf{x})g(\mathbf{\eta })\exp \left({\mathbf{\eta }}^T\mathbf{u}(\mathbf{x})\right)$$
we then write ${\mathcal{D}}_{\text{KL}}(p||q)$ as
$${\mathcal{D}}_{\text{KL}}(p||q)=-\log g(\mathbf{\eta })-{\mathbf{\eta }}^T{\mathbb{E}}_{p(\mathbf{x})}[\mathbf{u}(\mathbf{x})]+\text{const}$$
Taking the gradient of the both side of above w.r.t. $\mathbf{\eta }$ to zero yields
$$-\nabla \log g(\mathbf{\eta })={\mathbb{E}}_{p(\mathbf{x})}[\mathbf{u}(\mathbf{x})]$$
As mentioned in $(\ast 1)$, we then get
E q ( x ) [ u ( x ) ] = E p ( x ) [ u ( x ) ] \mathbb{E}_{q(\mathbf{x})}[\boldsymbol{u}(\mathbf{x})]=\mathbb{E}_{p(\mathbf{x})}[\boldsymbol{u}(\mathbf{x})]