统计学习方法第十八章——概率潜在语义分析

18.2 概率潜在语义分析的算法

生成模型的对数似然函数是：
$$\begin{array}{rl}L& =\sum _{i=1}^M\sum _{j=1}^{N}n(w_i,d_j)logP(w_i,d_j)\\ & =\sum _{i=1}^M\sum _{j=1}^{N}n(w_i,d_j)log[\sum _{k=1}P(w_i|z_k)P(z_k|d_j)P(d_j)]\\ & =\sum _{i=1}^M\sum _{j=1}^{N}n(w_i,d_j)[logP(d_j)+log(\sum _{k=1}P(w_i|z_k)P(z_k|d_j))]\\ & =\sum _{i=1}^M\sum _{j=1}^{N}n(w_i,d_j)logP(d_j)+\sum _{i=1}^M\sum _{j=1}^{N}n(w_i,d_j)log(\sum _{k=1}P(w_i|z_k)P(z_k|d_j))\end{array}$$
又因为前半部分是一个常数，与模型参数无关，于是将其省去，就得到书上的似然函数：
$$L=\sum _{i=1}^M\sum _{j=1}^{N}n(w_i,d_j)log[\sum _{k=1}P(w_i|z_k)P(z_k|d_j)]$$
E步：计算Q函数
$$\begin{array}{rl}L& =\sum _{i=1}^M\sum _{j=1}^{N}n(w_i,d_j)log[\sum _{k=1}P(w_i|z_k)P(z_k|d_j)]\\ & =\sum _{i=1}^M\sum _{j=1}^{N}n(w_i,d_j)log[\sum _{k=1}P(z_k|w_i,d_j)\frac{P(w_i|z_k)P(z_k|d_j)}{P(z_k|w_i,d_j)}]\end{array}$$
上式中, 由Jensen不等式:
$$\log \sum _j{\lambda }_j{y}_j\ge \sum _j{\lambda }_j\log y_j{\lambda }_j\ge 0,\sum _j{\lambda }_j=1$$

$$\begin{array}{rl}L& =\sum _{i=1}^M\sum _{j=1}^{N}n(w_i,d_j)log[\sum _{k=1}P(z_k|w_i,d_j)\frac{P(w_i|z_k)P(z_k|d_j)}{P(z_k|w_i,d_j)}]\\ & ⩾\sum _{i=1}^M\sum _{j=1}^{N}n(w_i,d_j)\sum _{k=1}^{K}P(z_k|w_i,d_j)log[\frac{P(w_i|z_k)P(z_k|d_j)}{P(z_k|w_i,d_j)}]\end{array}$$

得到L的下界：
$$\begin{array}{rl}L& =\sum _{i=1}^M\sum _{j=1}^{N}n(w_i,d_j)\sum _{k=1}^{K}P(z_k|w_i,d_j)log[\frac{P(w_i|z_k)P(z_k|d_j)}{P(z_k|w_i,d_j)}]\\ & =\sum _{i=1}^M\sum _{j=1}^{N}n(w_i,d_j)\sum _{k=1}^{K}P(z_k|w_i,d_j)[log[P(w_i|z_k)P(z_k|d_j)]-logP(z_k|w_i,d_j)]\\ & =\sum _{i=1}^M\sum _{j=1}^{N}n(w_i,d_j)\sum _{k=1}^{K}P(z_k|w_i,d_j)log[P(w_i|z_k)P(z_k|d_j)]-\sum _{i=1}^M\sum _{j=1}^{N}n(w_i,d_j)\sum _{k=1}^{K}P(z_k|w_i,d_j)logP(z_k|w_i,d_j)\end{array}$$
又因为在极大化Q函数时，对$P\left(w_i\mid z_k\right)\text{ 和 }P\left(z_k\mid d_j\right)\text{ 求偏导数 }$，后半部分偏导数为0，所以可以直接在这里将其省去，当然也可以留着，反正后面求导都会为0。因此
$$Q=\sum _{i=1}^M\sum _{j=1}^{N}n(w_i,d_j)\sum _{k=1}^{K}P(z_k|w_i,d_j)log[P(w_i|z_k)P(z_k|d_j)]$$
就得到了书上的$Q^{\prime }$函数。其中
$$P\left(z_k\mid w_i,d_j\right)=\frac{P\left(w_i\mid z_k\right)P\left(z_k\mid d_j\right)}{{\sum }_{k=1}^{K}P\left(w_i\mid z_k\right)P\left(z_k\mid d_j\right)}$$
M步：极大化Q函数

因为变量 $P\left(w_i\mid z_k\right),P\left(z_k\mid d_j\right)$ 形成概率分布, 满足约束条件
$$\begin{array}{rl}& \sum _{i=1}^{M}P\left(w_i\mid z_k\right)=1,k=1,2,\cdots ,K\\ & \sum _{k=1}^{K}P\left(z_k\mid d_j\right)=1,j=1,2,\cdots ,N\end{array}$$
应用拉格朗日法, 引入拉格朗日乘子 ${\tau }_k$ 和 ${\rho }_j$, 定义拉格朗日函数 $\Lambda$
$$\Lambda =Q^{\prime }+\sum _{k=1}^K{\tau }_k\left(1-\sum _{i=1}^{M}P\left(w_i\mid z_k\right)\right)+\sum _{j=1}^N{\rho }_j\left(1-\sum _{k=1}^{K}P\left(z_k\mid d_j\right)\right)$$
将拉格朗日函数 $\Lambda$ 分别对 $P\left(w_i\mid z_k\right)$ 和 $P\left(z_k\mid d_j\right)$ 求偏导数, 并令其等于 0 , 得到下面的方程组
$$\begin{array}{rl}& \sum _{j=1}^{N}n\left(w_i,d_j\right)P\left(z_k\mid w_i,d_j\right)-{\tau }_{k}P\left(w_i\mid z_k\right)=0,i=1,2,\cdots ,M;k=1,2,\cdots ,K\\ & \sum _{i=1}^{M}n\left(w_i,d_j\right)P\left(z_k\mid w_i,d_j\right)-{\rho }_{j}P\left(z_k\mid d_j\right)=0,j=1,2,\cdots ,N;k=1,2,\cdots ,K\end{array}$$
现求解${\tau }_k和{\rho }_j$，两边分别同时对i和k求和得到：
$$\begin{array}{rl}& \sum _{i=1}^M\sum _{i=1}^{M}n(w_i,d_j)P(z_k|w_j,d_j)=\sum _{i=1}^M{\tau }_{k}P(w_i|z_k)={\tau }_k\\ & \sum _{k=1}^K\sum _{i=1}^{M}n(w_i,d_j)P(z_k|w_i,d_j)=\sum _{k=1}^K{\rho }_{j}P(z_k|d_j)={\rho }_j\end{array}$$
于是得到：
$$\begin{array}{rl}{\rho }_j& =\sum _{k=1}^K\sum _{i=1}^{M}n(w_i,d_j)P(z_k|w_j,d_j)=\sum _{i=1}^{M}n(w_i,d_j)=n(d_j)\\ {\tau }_k& =\sum _{j=1}^N\sum _{i=1}^{M}n(w_i,d_j)P(z_k|w_i,d_j)\end{array}$$

将求得的${\tau }_k和{\rho }_j$代回方程组得参数估计公式：
$$\begin{array}{rl}& P\left(w_i\mid z_k\right)=\frac{{\sum }_{j=1}^{N}n\left(w_i,d_j\right)P\left(z_k\mid w_i,d_j\right)}{{\sum }_{m=1}^M{\sum }_{j=1}^{N}n\left(w_m,d_j\right)P\left(z_k\mid w_m,d_j\right)}\\ & P\left(z_k\mid d_j\right)=\frac{{\sum }_{i=1}^{M}n\left(w_i,d_j\right)P\left(z_k\mid w_i,d_j\right)}{n\left(d_j\right)}\end{array}$$