机器学习白板推导 shuhuai008

参考文献

shuhuai008

续篇

前言

数据$X={\left(\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1p}\\ x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{N1}& x_{N2}& \cdots & x_{Np}\end{array}\right)}_{N\times P}$，P为特征数，N为样本数。

在这片大陆上，有两大学派，一个是频率学派，另一个是统计学派。

频率派认为$\theta$是未知的常量，数据$X$为r.v（random variable，随机变量），$x\sim p(x\mid \theta )$
如：极大似然估计（Maximum Likelihood Estimation,MLE）
$${\theta }_{MLE}=\arg \underset{\theta }{\max }\log P(x\mid \theta )$$

贝叶斯派则认为参数$\theta$为r.v，$\theta \sim p(\theta )$
如：最大后验概率MAP
$$P(\theta \mid X)=\frac{P(X\mid \theta )P(\theta )}{P(X)}$$
$P(\theta \mid X)$为后验posterior，$P(X\mid \theta )$为likehood，$p(\theta )$为prior，$P(X)={\int }_{\theta }P(X\mid \theta )P(\theta )\mathrm{d}\theta$。
$${\theta }_{MAP}=\arg \underset{\theta }{\max }P(X\mid \theta )P(\theta )$$

贝叶斯估计：
$$P(\theta \mid X)=\frac{P(X\mid \theta )P(\theta )}{{\int }_{\theta }P(X\mid \theta )P(\theta )\mathrm{d}\theta }$$
贝叶斯预测：$\tilde{x}$预测数据,$\theta$为中间变量
$$\begin{array}{rl}P(\tilde{x}\mid X)& ={\int }_{\theta }P(\tilde{x},\theta \mid X)P(\theta )\mathrm{d}\theta \\ & ={\int }_{\theta }P(\tilde{x}\mid \theta )P(\theta \mid X)\mathrm{d}\theta \end{array}$$

频率派>统计机器学习>优化：

1. model
2. loss function
3. algorithm

贝叶斯派>概率图模型>求积分>MCMC

一、概率基础

1.一维高斯分布

$X=(x_1,x_2,\dots ,x_N{)}^{\top }=\left(\begin{array}{c}{x}_1^{\top }\\ x_2^{\top }\\ ⋮\\ x_N^{\top }\end{array}\right)$
$x_i\in {\mathbb{R}}^p$

$\theta \stackrel{\text{i.i.d}}{\sim }\mathcal{N}(\mu ,\Sigma )$
i.i.d.(independent and identically distributed 独立同分布)

令$p=1$,$\theta =(\mu ,{\sigma }^2)$
一维高斯分布：
$$p(x)=\frac{1}{\sqrt{2\pi }\sigma }\exp \left\{-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right\}$$

极大似然估计
$${\theta }_{MLE}=\arg \underset{\theta }{\max }P(X\mid \theta )$$

对数似然：
$$\begin{array}{rl}\log P(X\mid \theta )& =\log \prod _{i=1}^{N}P(x_i\mid \theta )\\ & =\sum _{i=1}^N\log P(x_i\mid \theta )\\ & =\sum _{i=1}^N\log \frac{1}{\sqrt{2\pi }\sigma }\exp -\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\\ & =\sum _{i=1}^N\left[\log \frac{1}{\sqrt{2\pi }}+\log \frac{1}{\sigma }-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\right]\end{array}$$
求解对数似然${\mu }_{MLE}$:

$$\begin{array}{rl}{\mu }_{MLE}& =\arg \underset{\mu }{\max }\log P(X\mid \theta )\\ & =\arg \underset{\mu }{\max }\sum _{i=1}^N-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\\ & =\arg \underset{\mu }{\min }\sum _{i=1}^N\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\end{array}$$

对${\mu }_{MLE}$求偏导:

$$\begin{array}{rl}\frac{\partial }{\partial \mu }\sum _{i=1}^N(x_i-\mu {)}^2& =\sum _{i=1}^{N}2\cdot (x_i-\mu )\cdot (-1)=0\\ & \Rightarrow \sum _{i=1}^N(x_i-\mu )=0\\ & \Rightarrow {\mu }_{MLE}=\frac{1}{N}\sum _{i=1}^N{x}_i\end{array}$$
对${\mu }_{MLE}$求期望：
$$E[{\mu }_{MLE}]=\frac{1}{N}\sum _{i=1}^{N}E[x_i]=\frac{1}{N}\sum _{i=1}^N\mu =\mu$$
因此${\mu }_{MLE}$无偏。

求解对数似然${\sigma }_{MLE}$:
$$\begin{array}{rl}{\mu }_{MLE}& =\arg \underset{\sigma }{\max }\log P(X\mid \theta )\\ & =\arg \underset{\sigma }{\max }\sum _{i=1}^N-\log \sigma -\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\end{array}$$
对数似然${\sigma }_{MLE}$偏导:

令$L(\sigma )=-\log \sigma -\frac{(x_i-\mu {)}^2}{2{\sigma }^2}$
$$\begin{array}{rl}\frac{\partial L}{\partial \sigma }& =\sum _{i=1}^N[-\frac{1}{\sigma }+\frac{1}{2}(x_i-\mu {)}^2\cdot (-2)\cdot {\sigma }^{-3}]\\ & =\sum _{i=1}^N[-{\sigma }^2+(x_i-\mu {)}^2]=0\\ & \Rightarrow \sum _{i=1}^N{\sigma }^2=\sum _{i=1}^N(x_i-\mu {)}^2\end{array}$$
$${\sigma }_{MLE}^2=\frac{1}{N}\sum _{i=1}^N(x_i-\mu {)}^2=\frac{1}{N}\sum _{i=1}^N{x}_i^2-{\mu }_{MLE}^2$$
${\sigma }_{MLE}^2$求期望:
$$\begin{array}{rl}E[{\sigma }_{MLE}^2]& =E[\frac{1}{N}\sum _{i=1}^N{x}_i^2-{\mu }_{MLE}^2]\\ & =E[\frac{1}{N}\sum _{i=1}^N(x_i^2-{\mu }^2)-({\mu }_{MLE}^2-{\mu }^2)]\\ & =E[\frac{1}{N}\sum _{i=1}^N(x_i^2-{\mu }^2)]-E[{\mu }_{MLE}^2-{\mu }^2]\\ & =\frac{1}{N}\sum _{i=1}^{N}E[x_i^2-{\mu }^2]-(E[{\mu }_{MLE}^2]-E[{\mu }^2])\\ & =\frac{1}{N}\sum _{i=1}^N(E[x_i^2]-{\mu }^2)-(E[{\mu }_{MLE}^2]-{\mu }^2)\\ & =\frac{1}{N}\sum _{i=1}^{N}Var(x_i)-E[{\mu }_{MLE}^2-E^2[{\mu }_{MLE}]\\ & =\frac{1}{N}\sum _{i=1}^{N}Var(x_i)-Var({\mu }_{MLE})\\ & ={\sigma }^2-\frac{1}{N}{\sigma }^2\end{array}$$
其中$Var({\mu }_{MLE})=Var(\frac{1}{N}{\sum }_{i=1}^N{x}_i)=\frac{1}{N^2}{\sum }_{i=1}^{N}Var(x_i)=\frac{1}{N^2}{\sum }_{i=1}^N{\sigma }^2=\frac{1}{N}{\sigma }^2$

因此${\sigma }_{MLE}=\frac{N-1}{N}{\sigma }^2$是有偏估计。

极大似然估计
$$\begin{gathered} {\mu }_{MLE}\to E[{\mu }_{MLE}] \\ {\sigma }_{MLE}^2\to E[{\sigma }_{MLE}^2] \end{gathered}$$

2.高维高斯分布

$$p(x)=\frac{1}{(2\pi {)}^{p/2}{\left|\Sigma \right|}^{1/2}}\exp (-\frac{1}{2}(x-\mu {)}^{\top }{\Sigma }^{-1}(x-\mu ))$$

$x\in {\mathbb{R}}^p$，r.v
$x=\left(\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_p\end{array}\right),\mu =\left(\begin{array}{c}{\mu }_1\\ {\mu }_2\\ ⋮\\ {\mu }_p\end{array}\right),\Sigma ={\left(\begin{array}{cccc}{\sigma }_{11}& {\sigma }_{12}& \cdots & {\sigma }_{1p}\\ {\sigma }_{21}& {\sigma }_{22}& \cdots & {\sigma }_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ {\sigma }_{p1}& {\sigma }_{p2}& \cdots & {\sigma }_{pp}\end{array}\right)}_{p\times p},\Sigma$需要为正定矩阵，不过一般都是半正定矩阵。

$(x-\mu {)}^{\top }{\Sigma }^{-1}(x-\mu )$表示$x$与$\mu$的马氏距离

当$\Sigma =1$时，为欧式距离

特征分解
$$\begin{gathered} U=(U_1,U_2,\dots ,U_p{)}_{p\times p},i=1,2,\dots ,p \\ U{U}^{\top }=U^{\top }U=I \\ \Lambda =diag({\lambda }_i) \end{gathered}$$
$$\begin{array}{rl}\Sigma & =U\Lambda U^{\top }=\left(\begin{array}{cccc}{\mu }_1& {\mu }_2& \cdots & {\mu }_p\end{array}\right)\left(\begin{array}{cccc}{\lambda }_1& & & \\ & {\lambda }_2& & \\ & & \ddots & \\ & & & {\lambda }_p\end{array}\right)\left(\begin{array}{c}{\mu }_1^{\top }\\ {\mu }_2^{\top }\\ ⋮\\ {\mu }_p^{\top }\end{array}\right)\\ & =\left(\begin{array}{cccc}{\mu }_1{\lambda }_1& {\mu }_2{\lambda }_2& \cdots & {\mu }_p{\lambda }_p\end{array}\right)\left(\begin{array}{c}{\mu }_1^{\top }\\ {\mu }_2^{\top }\\ ⋮\\ {\mu }_p^{\top }\end{array}\right)=\sum _{i=1}^p{u}_i{\lambda }_i{u}_i^{\top }\end{array}$$
$${\Sigma }^{-1}=(U\Lambda U^{\top }{)}^{-1}=(U^{\top }{)}^{-1}{\Lambda }^{-1}{U}^{-1}=U{\Lambda }^{-1}{U}^{\top }=\sum _{i=1}^p{u}_i\frac{1}{{\lambda }_i}{u}_i^{\top }$$
$$\begin{array}{rl}(x-\mu {)}^{\top }{\Sigma }^{-1}(x-\mu )& =(x-\mu {)}^{\top }\sum _{i=1}^p{u}_i\frac{1}{{\lambda }_i}{u}_i^{\top }(x-\mu )\\ & =\sum _{i=1}^p(x-\mu {)}^{\top }{u}_i\frac{1}{{\lambda }_i}{u}_i^{\top }(x-\mu )\\ & =\sum _{i=1}^p{y}_i\frac{1}{{\lambda }_i}{y}_i^{\top }=\sum _{i=1}^p\frac{y_i^2}{{\lambda }_i}\end{array}$$
当$p=2$时，$\frac{y_1^2}{{\lambda }_1}+\frac{y_2^2}{{\lambda }_2}$是一个椭圆，表示数据$x$的聚集为椭圆，由数据$x$的坐标变换到$y$坐标，数据的主方向和$y$坐标相同。
${\lambda }_1={\lambda }_2$时，数据聚集为圆，各向同性（无新坐标）。

$$p(x)\to \Sigma \to {\Sigma }^{-1}\to (x-\mu {)}^{\top }{\Sigma }^{-1}(x-\mu )=\sum _{i=1}^p\frac{y_i^2}{{\lambda }_i}$$

3.局限性

参数多
各向同性
多类别数据无法表示

4.边缘概率&条件概率

$$p(x)=\frac{1}{(2\pi {)}^{p/2}{\left|\Sigma \right|}^{1/2}}\exp (-\frac{1}{2}(x-\mu {)}^{\top }{\Sigma }^{-1}(x-\mu ))$$
$x\in {\mathbb{R}}^p$，r.v
$x=\left(\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_p\end{array}\right),\mu =\left(\begin{array}{c}{\mu }_1\\ {\mu }_2\\ ⋮\\ {\mu }_p\end{array}\right),\Sigma ={\left(\begin{array}{cccc}{\sigma }_{11}& {\sigma }_{12}& \cdots & {\sigma }_{1p}\\ {\sigma }_{21}& {\sigma }_{22}& \cdots & {\sigma }_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ {\sigma }_{p1}& {\sigma }_{p2}& \cdots & {\sigma }_{pp}\end{array}\right)}_{p\times p}$

已知$x=\left(\begin{array}{c}{x}_a\\ x_b\end{array}\right),\mu =\left(\begin{array}{c}{\mu }_a\\ {\mu }_b\end{array}\right),\Sigma =\left(\begin{array}{cc}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ba}& {\Sigma }_{bb}\end{array}\right),x_a:m维,x_b:n维，m+n=p$

求：
$p(x_a),p(x_b\mid x_a)$
$p(x_b),p(x_a\mid x_b)$

---

定理：
$已知x\sim \mathcal{N}(\mu ,\Sigma ),y=Ax+B$
结论：
$$\begin{gathered} y\sim \mathcal{N}(A\mu +B,A\Sigma A^{\top }) \\ E[y]=E[Ax+B]=AE[x]+B=A\mu +B \\ \begin{array}{rl}Var[y]& =Var[Ax+B]=Var[Ax]+Var[B]\\ & =AVar[x]A^{\top }=A\Sigma A^{\top }\end{array} \end{gathered}$$

---

• 求：$p(x_a)$

$x_a=\left(\begin{array}{cc}{I}_m& 0\end{array}\right)\left(\begin{array}{c}{x}_a\\ x_b\end{array}\right)$
$E[x_a]=\left(\begin{array}{cc}{I}_m& 0\end{array}\right)\left(\begin{array}{c}{\mu }_a\\ {\mu }_b\end{array}\right)={\mu }_a$
$Var[x_a]=\left(\begin{array}{cc}{I}_m& 0\end{array}\right)\left(\begin{array}{cc}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ba}& {\Sigma }_{bb}\end{array}\right)\left(\begin{array}{c}{I}_m\\ 0\end{array}\right)={\Sigma }_{aa}$
$\therefore x_a\sim \mathcal{N}({\mu }_a,{\Sigma }_{aa})$

---

• 求：$p(x_{b\cdot a})$
  令$\begin{array}{r}\{\begin{array}{ll}{x}_{b\cdot a}& =x_b-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a\\ {\mu }_{b\cdot a}& ={\mu }_b-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\mu }_a\\ {\Sigma }_{bb\cdot a}& ={\Sigma }_{bb}-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\Sigma }_{ab}\end{array}\end{array}$
  $x_{b\cdot a}=\left(\begin{array}{cc}{\Sigma }_{ba}{\Sigma }_{aa}& I_n\end{array}\right)\left(\begin{array}{c}{x}_a\\ x_b\end{array}\right)=Ax$
  $\begin{array}{rl}E[x_{b\cdot a}]& =\left(\begin{array}{cc}{\Sigma }_{ba}{\Sigma }_{aa}^{-1}& I_n\end{array}\right)\left(\begin{array}{c}{\mu }_a\\ {\mu }_b\end{array}\right)\\ & ={\mu }_b-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\mu }_a={\mu }_{b\cdot a}=A\mu +B\end{array}$
  $\begin{array}{rl}Var[x_{b\cdot a}]& =\left(\begin{array}{cc}{\Sigma }_{ba}{\Sigma }_{aa}^{-1}& I_n\end{array}\right)\left(\begin{array}{cc}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ba}& {\Sigma }_{bb}\end{array}\right)\left(\begin{array}{c}{\Sigma }_{ba}{\Sigma }_{aa}^{-1}\\ I_n\end{array}\right)\\ & ={\Sigma }_{bb}-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\Sigma }_{ab}\\ & ={\Sigma }_{bb\cdot a}=A\Sigma A^{\top }\end{array}$
  $\therefore x_{b\cdot a}\sim \mathcal{N}({\mu }_{b\cdot a},{\Sigma }_{bb\cdot a})$

---

• 求：$p(x_b\mid x_a)$
  $\begin{array}{rl}{x}_b& =x_{b\cdot a}+{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a\\ & =Ax+B\end{array}$
  $\begin{array}{rl}E[x_b\mid x_a]& ={\mu }_{b\cdot a}+{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a\\ & =A\mu +B\end{array}$
  $\begin{array}{rl}Var[x_b\mid x_a]& =Var[x_{b\cdot a}]\\ & =A\Sigma A^{\top }=\Sigma \end{array}$
  $$

$$x\to \begin{array}{c}y=Ax+B\end{array}\}\to \{\begin{array}{c}E[y]\\ Var[y]\end{array}\to y\sim \mathcal{N}(E[y],Var[y])$$

$$\begin{array}{rl}\begin{array}{c}\left(\genfrac{}{}{0ex}{}{x_a}{x_b}\right)\to x_a\end{array}\}\to & \{\begin{array}{c}E[x_a]\\ Var[x_a]\end{array}\to x_a\sim \mathcal{N}(E[x_a],Var[x_a])\\ \begin{array}{c}{x}_{b\cdot a}\end{array}\}\to & \{\begin{array}{c}E[x_{b\cdot a}]\\ Var[x_{b\cdot a}]\end{array}\to x_{b\cdot a}\sim \mathcal{N}(E[x_{b\cdot a}],Var[x_{b\cdot a}])\\ \begin{array}{c}{x}_b\mid x_a\end{array}\}\to & \{\begin{array}{c}E[x_b\mid x_a]\\ Var[x_b\mid x_a]\end{array}\to x_b\mid x_a\sim \mathcal{N}(E[x_b\mid x_a],Var[x_b\mid x_a])\end{array}$$

5.联合概率

已知
$$\begin{gathered} p(x)\sim \mathcal{N}(x\mid \mu ,{\Lambda }^{-1}) \\ p(y\mid x)\sim \mathcal{N}(y\mid Ax+B,{\Lambda }^{-1}) \end{gathered}$$
求
$p(y),p(x\mid y)$

---

$$\begin{gathered} y=Ax+b+\epsilon \\ x,y,\epsilon \sim r.v \\ \epsilon \sim \mathcal{N}(0,L^{-1}) \end{gathered}$$

$\begin{array}{rl}E[y]& =E[Ax+b+\epsilon ]\\ & =E[Ax+b]+E[\epsilon ]\\ & =A\mu +b\end{array}$
$\begin{array}{rl}Var[y]& =Var[Ax+b+\epsilon ]\\ & =Var[Ax+b]+Var[\epsilon ]\\ & =A{\Lambda }^{-1}{A}^{\top }+L^{-1}\end{array}$
$\therefore y\sim \mathcal{N}(A\mu +b,A{\Lambda }^{-1}{A}^{\top }+L^{-1})$

---

$z=\left(\begin{array}{c}x\\ y\end{array}\right)\sim \mathcal{N}(\left[\begin{array}{c}\mu \\ A\mu +b\end{array}\right],\left(\begin{array}{cc}{L}^{-1}& \Delta \\ \Delta & A{\Lambda }^{-1}{A}^{\top }+L^{-1}\end{array}\right))$

$\begin{array}{rl}\Delta & =Cov(x,y)\\ & =E[(x-E[x])(y-E[y]{)}^{\top }]\\ & =E[(x-\mu )(y-A\mu -b{)}^{\top }]\\ & =E[(x-\mu )(Ax+b+\epsilon -A\mu -b{)}^{\top }]\\ & =E[(x-\mu )(Ax+\epsilon -A\mu {)}^{\top }]\\ & =E[(x-\mu )(Ax-A\mu {)}^{\top }]-E[(x-\mu )\epsilon ]\\ & =E[(x-\mu )(x-\mu {)}^{\top }]A^{\top }\\ & =Var[x]A^{\top }\\ & ={\Lambda }^{-1}{A}^{\top }\end{array}$

$z=\left(\begin{array}{c}x\\ y\end{array}\right)=p(x,y)=\epsilon \sim \mathcal{N}(\left[\begin{array}{c}\mu \\ A\mu +b\end{array}\right],\left(\begin{array}{cc}{L}^{-1}& {\Lambda }^{-1}{A}^{\top }\\ {\Lambda }^{-1}{A}^{\top }\Delta & A{\Lambda }^{-1}{A}^{\top }+L^{-1}\end{array}\right))$

---

$p(x\mid y)=\frac{p(x,y)}{p(y)}$
$\therefore p(x\mid y)\sim \mathcal{N}(,)$

x → y } → { E [ y ] V a r [ y ] → y ∼ N ( E [ y ] , V a r [ y ] ) ( x y ) } → { ( E [ x ] E [ y ] ) ( V a r [ x ] Δ Δ V a r [ y ] ) → y ∼ N ( ( E [ x ] E [ y ] ) , ( V a r [ x ] Δ Δ V a r [ y ] ) )