统计推断(六) Modeling

1. Modeling problem

• formulation

  • a set of distributions
    P = { p y ( ⋅ ; x ) ∈ P y : x ∈ X } \mathcal{P}=\left\{p_{\mathrm{y}}(\cdot ; x) \in \mathcal{P}^{y}: x \in \mathcal{X}\right\} P={py​(⋅;x)∈Py:x∈X}

  • approximation
    $$\underset{q\in {\mathcal{P}}^y}{\min }\underset{x\in \mathcal{X}}{\max }D\left(p_{\mathrm{y}}(\cdot ;x)\Vert q(\cdot )\right)$$

• solution

Theorem: 对任意 $q\in {\mathcal{P}}^y$ 都存在一个混合模型 $q_w(\cdot )={\sum }_{x\in \mathcal{X}}w(x)p_y(\cdot ;x)$ 满足
$$D\left(p_y(\cdot ;x)\Vert q_w(\cdot )\right)\le D\left(p_y(\cdot ;x)\Vert q(\cdot )\right)\text{ for all }x\in \mathcal{X}$$
Proof: 应用 Pythagoras 定理

然后很容易有
$$\underset{x\in \mathcal{X}}{\max }\underset{q\in {\mathcal{P}}^y}{\min }D\left(p_y(\cdot ;x)\Vert q(\cdot )\right)=\underset{x\in \mathcal{X}}{\max }0=0$$

$$\underset{q\in \mathcal{P}}{\min }\underset{x\in \mathcal{X}}{\max }D\left(p_{\mathrm{y}}(\cdot ;x)\Vert q(\cdot )\right)=\underset{q\in \mathcal{P}}{\min }\underset{w\in {\mathcal{P}}^{\mathcal{X}}}{\max }\sum _{x}w(x)D\left(p_{\mathrm{y}}(\cdot ;x)\Vert q(\cdot )\right)$$

Theorem (Redundancy-Capacity Theorem): 以下等式成立，且两侧最优的 $w,q$ s是相同的
$$\begin{array}{rl}{R}^{+}\triangleq \underset{q\in {\mathcal{P}}^{\mathcal{Y}}}{\min }\underset{w\in {\mathcal{P}}^{\mathcal{X}}}{\max }& \sum _{x}w(x)D\left(p_{\mathrm{y}}(\cdot ;x)\Vert q(\cdot )\right)\\ & =\underset{w\in \mathcal{P}}{\max }\underset{q\in \mathcal{P}}{\min }\sum _{x}w(x)D\left(p_{\mathrm{y}}(\cdot ;x)\Vert q(\cdot )\right)\triangleq R^{-}\end{array}$$
Proof:

1. 利用后面的 Equidistance property 证明 $R^{+}\le R^{-}$
2. 根据 minimax 和 maxmini 的性质，有 $R^{+}\ge R^{-}$
3. 一定有 $R^{+}\ge R^{-}$
4. 证明两个不等式的取等条件是在同样的 $w,q$ 处取到

2. Model capacity

首先计算 $R^{-}$ 内部的 min
$$\begin{array}{rl}& \underset{q\in {\mathcal{P}}^{\mathcal{Y}}}{\min }\sum _{x}w(x)D\left(p_{\mathbf{y}}(\cdot ;x)\Vert q(\cdot )\right)\\ =& \underset{q\in {\mathcal{P}}^{\mathcal{Y}}}{\min }\sum _{x,y}w(x)p_{\mathbf{y}}(y;x)\log \frac{p_y(y;x)}{q(y)}\\ =& \text{ constant }-\underset{q\in {\mathcal{P}}^{\mathcal{Y}}}{\max }\sum _y{q}_w(y)\log q(y)\\ =& \text{ constant }-\underset{q\in {\mathcal{P}}^{\mathcal{Y}}}{\max }{\mathbb{E}}_{q_w}[\log q(y)]\end{array}$$
根据 Gibbs 不等式
$$q^{\ast }(\cdot )=q_w(\cdot )\triangleq \sum _{x\in \mathcal{X}}w(x)p_y(\cdot ;x)$$
再考虑 $R^{-}$ 外部的 max，此时可以转化为 Bayesian 角度！
$$
\begin{aligned} R^{-} &=\max {w \in \mathcal{P}^{\mathcal{X}}} \sum{x} w(x) D\left(p_{y}(\cdot ; x) | q_{w}(\cdot)\right) \ &=\max {w \in \mathcal{P}^{\mathcal{X}}} \sum{x, y} w(x) p_{y}(y ; x) \log \frac{p_{y}(y ; x)}{\sum_{x^{\prime}} w\left(x^{\prime}\right) p_{y}\left(y ; x^{\prime}\right)} \

&\overset{\text{Bayesian}}{=}\max {p{\mathbf{x}}} \sum_{x} p_{\mathbf{x}}(x) D\left(p_{y | \mathbf{x}}(\cdot | x) | p_{y}(\cdot)\right) \ &=\max {p{\mathbf{x}}} \sum_{x, y} p_{\mathbf{x}}(x) p_{\mathbf{y} | \mathbf{x}}(y | x) \log \frac{p_{y | x}(y | x)}{p_{\mathbf{y}}(y)} \ &=\max {p{\mathbf{x}}} \sum_{x, y} p_{\mathbf{x}, \mathbf{y}}(x, y) \log \frac{p_{\mathbf{x}, \mathbf{y}}(x, y)}{p_{\mathbf{x}}(x) p_{y}(y)}=\max {p{\mathbf{x}}} I(x ; y)=C
\end{aligned}
KaTeX parse error: Can't use function '$' in math mode at position 24: …ition**: 对一个模型 $̲p_{\mathsf{y|x}…
C \triangleq \max {p{x}} I(x ; y)
$$

• Model capacity: C
• least informative prior: $p_x^{\ast }$

Theorem(Equidistance property): C对应的最优的 $p^{\ast }$ 和 $w^{\ast }$ 满足
$$D(p_y(\cdot ;x)||q^{\ast }(\cdot ))\le C\forall x\in \mathcal{X}$$
其中等号对于满足 $w^{\ast }(x)>0$ 的 x 成立

Proof:

1. $I(x,y)$ 关于 $p_x(a)\forall a$ 是 concave 的
2. 构造拉格朗日函数 $\mathcal{L}=I(x,y)-\lambda ({\sum }_x{p}_x(x)-1)$，也关于 $p_x(a)$ concave
3. ${\min }_{p_x}I(x,y)$ 的极值点应满足 ${\frac{\partial I(x;y)}{\partial p_x(a)}|}_{p_x=p_x^{\ast }}-\lambda =0,\text{ for all }a\in \mathcal{X}\text{ such that }{p}_x^{\ast }(a)>0$，或者 ${\frac{\partial I(x;y)}{\partial p_x(a)}|}_{p_x=p_x^{\ast }}-\lambda \le 0,\text{ for all }a\in \mathcal{X}\text{ such that }{p}_x^{\ast }(a)=0$
4. $\frac{\partial I(x;y)}{\partial p_x(a)}=D\left(p_{y|x}(\cdot ;a)\Vert p_y\right)-\log e$ 并根据 3 中取等号的特点恰好可以得到定理中的式子

3. Inference with mixture models

• Formulation: 有观测 $y_{-}$，想要预测 $y_{+}$

• Solution

  • 根据前面得到的最优先验 $w^{\ast }$ 来估计 $y=[y_{-},y_{+}]$ 的分布
    $$q_{\mathbf{y}}^{\ast }(\mathbf{y})=\sum _x{w}^{\ast }(x)p_{\mathbf{y}}(\mathbf{y};x)$$

  • 然后可以获得后验概率
    $$\begin{array}{rl}{q}_{\mathrm{y}+|{\mathrm{y}}_{-}}^{\ast }\left(\cdot |y_{-}\right)& \triangleq \frac{q_{\mathrm{y}}^{\ast }\left(y_{+},y_{-}\right)}{q_{\mathrm{y}-}^{\ast }\left(y_{-}\right)}=\frac{{\sum }_x{w}^{\ast }(x)p_{\mathrm{y}}\left(y_{+},y_{-};x\right)}{{\sum }_a{w}^{\ast }(a)p_{{\mathrm{y}}_{-}}\left(y_{-};a\right)}\\ & =\sum _x{w}^{\ast }\left(x|y_{-}\right)p_{{\mathrm{y}}_{+}|y_{-}}\left(y_{+}|y_{-};x\right)\end{array}$$

  • 相当于是做了 soft decision，因为 ML 估计中只会取 $p_{{\mathrm{y}}_{+}|y_{-}}(\cdot |y_{-};{\hat{x}}_{ML})$

4. Maximum entropy distribution

• 最大熵等价于均匀分布向对应的模型集合上的 I-projection
  $$\begin{gathered} D(p\Vert U)=\sum _{y}p(y)\log p(y)+\log |\mathcal{Y}|=\log |\mathcal{Y}|-H(p) \\ {p}^{\ast }=\underset{p\in {\mathcal{L}}_{\mathrm{t}}}{\mathrm{argmax}}H(p)=\underset{p\in {\mathcal{L}}_{\mathrm{t}}}{\mathrm{argmin}}D(p\Vert U) \end{gathered}$$

---

其他内容请看：
统计推断(一) Hypothesis Test
统计推断(二) Estimation Problem
统计推断(三) Exponential Family
统计推断(四) Information Geometry
统计推断(五) EM algorithm
统计推断(六) Modeling
统计推断(七) Typical Sequence
统计推断(八) Model Selection
统计推断(九) Graphical models
统计推断(十) Elimination algorithm
统计推断(十一) Sum-product algorithm