LLE 证明

a LLE 权重 求解

$x_i\in {\mathbb{R}}^{d\times 1}$, $w_i\in {\mathbb{R}}^{1\times n}$, $X=[x_1\dots x_n]$, $w_i=[w_{i1}\dots w_{in}]$.

$$\begin{array}{rl}\underset{w_i}{\min }& \sum _{i=1}^n{\Vert x_i-\sum _{j=1}^n{w}_{ij}{x}_j\Vert }_2^2\\ \mathrm{s}.\mathrm{t}.& \sum _{j=1}^n{w}_{ij}=1\end{array}$$

$$\begin{array}{rl}\sum _{i=1}^n{\Vert x_i-\sum _{j=1}^n{w}_{ij}{x}_j\Vert }_2^2& =\sum _{i=1}^n{\Vert x_i-\sum _{j=1}^n{w}_{ij}{x}_j\Vert }_2^2\\ & =\sum _{i=1}^n{\Vert \sum _{j=1}^n{w}_{ij}{x}_i-\sum _{j=1}^n{w}_{ij}{x}_j\Vert }_2^2\\ & =\sum _{i=1}^n{\Vert \sum _{j=1}^n{w}_{ij}(x_i-x_j)\Vert }_2^2\\ & =\sum _{i=1}^n{\Vert (x_i{1}^T-X)w_i^T\Vert }_2^2\\ & =\sum _{i=1}^n{w}_i(x_i{1}^T-X{)}^T(x_i{1}^T-X)w_i^T\end{array}$$

$$\sum _{j=1}^n{w}_{ij}=1\iff w_{i}1=1$$

$$L=\sum _{i=1}^n\left(w_i(x_i{1}^T-X{)}^T(x_i{1}^T-X)w_i^T+{\mu }_i(w_{i}1-1)\right)$$

$$\begin{array}{rl}0=\frac{\partial }{\partial w_i}L& =\frac{\partial }{\partial w_i}\sum _{i=1}^n\left(w_i(x_i{1}^T-X{)}^T(x_i{1}^T-X)w_i^T+{\mu }_i(w_{i}1-1)\right)\\ & =\frac{\partial }{\partial w_i}\left(w_i(x_i{1}^T-X{)}^T(x_i{1}^T-X)w_i^T+{\mu }_i(w_{i}1-1)\right)\\ & =2w_i(x_i{1}^T-X{)}^T(x_i{1}^T-X)+{\mu }_i{1}^T\end{array}$$

$$w_i=-\frac{1}{2}{\mu }_i{1}^T(x_i{1}^T-X{)}^{-1}(x_i{1}^T-X{)}^{-T}$$

$$1=w_{i}1=-\frac{1}{2}{\mu }_i{1}^T(x_i{1}^T-X{)}^{-1}(x_i{1}^T-X{)}^{-T}1$$

$$-\frac{1}{2}{\mu }_i=\frac{1}{1^T(x_i{1}^T-X{)}^{-1}(x_i{1}^T-X{)}^{-T}1}$$

$$\begin{array}{rl}{w}_i& =-\frac{1}{2}{\mu }_i{1}^T(x_i{1}^T-X{)}^{-1}(x_i{1}^T-X{)}^{-T}\\ & =\frac{1^T(x_i{1}^T-X{)}^{-1}(x_i{1}^T-X{)}^{-T}}{1^T(x_i{1}^T-X{)}^{-1}(x_i{1}^T-X{)}^{-T}1}\\ & =\frac{1^T[(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}}{1^T[(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}1}\end{array}$$

$$\begin{array}{rl}{w}_{ij}& ={\left(\frac{1^T[(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}}{1^T[(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}1}\right)}_j\\ & =\frac{{\left(1^T[(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}\right)}_j}{1^T[(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}1}\\ & =\frac{{\left(1^T[(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}\right)}_j}{\sum _{j=1}^n{\left(1^T[(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}\right)}_j}\\ & =\frac{\sum _{k=1}^n{\left([(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}\right)}_{kj}}{\sum _{j=1}^n\sum _{k=1}^n{\left([(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}\right)}_{kj}}\end{array}$$

实际上$[(x_i{1}^T-X{)}^T(x_i{1}^T-X)]$不可逆, 原因是$(x_i{1}^T-X)$第$i$列是$0$, 所以我们实际采用

$$w_i=\frac{1^T[(x_i{1}^T-X^{(i)}{)}^T(x_i{1}^T-X^{(i)}){]}^{-1}}{1^T[(x_i{1}^T-X^{(i)}{)}^T(x_i{1}^T-X^{(i)}){]}^{-1}1}$$

$$w_{ij}=\frac{\sum _{k=1}^{n^{(i)}}{\left([(x_i{1}^T-X^{(i)}{)}^T(x_i{1}^T-X^{(i)}){]}^{-1}\right)}_{kj}}{\sum _{j^{\prime }=1}^{n^{(i)}}\sum _{k=1}^{n^{(i)}}{\left([(x_i{1}^T-X^{(i)}{)}^T(x_i{1}^T-X^{(i)}){]}^{-1}\right)}_{k{j}^{\prime }}}$$

其中$X^{(i)}$仅包含$x_i$的近邻(不包括$x_i$自身), 共$n^{(i)}$个近邻.

b LLE 权重 旋转/平移/缩放 不变性

$w_i(X)=\frac{1^T[(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}}{1^T[(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}1}$

记对$x_i$的变换为$f(x_i)$, 记对$X$的变换为$F(X)$. $w_i(T(X))=w_i(X)$, 的充分条件有

$$\begin{array}{rl}& \frac{1^T[(f(x_i)1^T-F(X){)}^T(f(x_i)1^T-F(X)){]}^{-1}}{1^T[(f(x_i)1^T-F(X){)}^T(f(x_i)1^T-F(X)){]}^{-1}1}=\frac{1^T[(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}}{1^T[(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}1}\\ \Leftarrow & (f(x_i)1^T-F(X){)}^T(f(x_i)1^T-F(X))=(x_i{1}^T-X{)}^T(x_i{1}^T-X)\\ \Leftarrow & f(x_i)1^T-F(X)=x_i{1}^T-X\end{array}$$

旋转$f(x_i)=Q{x}_i,F(X)=QX$, 证明充分条件

$$\begin{array}{rl}& (f(x_i)1^T-F(X){)}^T(f(x_i)1^T-F(X))\\ & =(Q{x}_i{1}^T-QX{)}^T(Q{x}_i{1}^T-QX)\\ & =[Q(x_i{1}^T-X){]}^T[Q(x_i{1}^T-X)]\\ & =(x_i{1}^T-X{)}^T{Q}^{T}Q(x_i{1}^T-X)\\ & =(x_i{1}^T-X{)}^{T}I(x_i{1}^T-X)\\ & =(x_i{1}^T-X{)}^T(x_i{1}^T-X)\end{array}$$

平移$f(x_i)=x_i+v,F(X)=X+v{1}^T$, 证明充分条件

$$\begin{array}{rl}& f(x_i)1^T-F(X)\\ & =(x_i+v)1^T-(X+v{1}^T)\\ & =x_i{1}^T+v{1}^T-X-v{1}^T\\ & =x_i{1}^T-X\end{array}$$

伸缩$f(x_i)=a{x}_i,F(X)=aX$, 证明充要条件

$$\begin{array}{rl}& \frac{1^T[(f(x_i)1^T-F(X){)}^T(f(x_i)1^T-F(X)){]}^{-1}}{1^T[(f(x_i)1^T-F(X){)}^T(f(x_i)1^T-F(X)){]}^{-1}1}\\ & =\frac{1^T[(a{x}_i{1}^T-aX{)}^T(a{x}_i{1}^T-aX){]}^{-1}}{1^T[(a{x}_i{1}^T-aX{)}^T(a{x}_i{1}^T-aX){]}^{-1}1}\\ & =\frac{1^T[a^2(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}}{1^T[a^2(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}1}\\ & =\frac{a^{-2}{1}^T[(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}}{a^{-2}{1}^T[(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}1}\\ & =\frac{1^T[(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}}{1^T[(x_i{1}^T-X{)}^T(x_i{1}^T-X){]}^{-1}1}\end{array}$$

c LLE 低维表示 意义

$$\begin{array}{rlr}\underset{y_i}{\max }& \sum _{i=1}^n{\Vert y_i-\sum _{j=1}^n{w}_{ij}{y}_j\Vert }_2^2& \\ \mathrm{s}.\mathrm{t}.& \sum _{i=1}^n{y}_i=0& Y1=0\\ & \sum _{i=1}^n{y}_i{y}_i^T=I& Y{Y}^T=I\end{array}$$

为什么保持了局部几何性质？
${\sum }_{i=1}^n{\Vert x_i-{\sum }_{j=1}^n{w}_{ij}{x}_j\Vert }_2^2={\sum }_{i=1}^n{\Vert y_i-{\sum }_{j=1}^n{w}_{ij}{y}_j\Vert }_2^2$
$x_i$和$y_i$共享了局部的权重$w_i$, 保持邻域内样本之间的线性关系.
样本点$x_i$的坐标能通过它的邻域样本$x_j$通过线性组合而重构出来, 降维后样本点$y_i$的坐标能通过它的邻域样本$y_j$通过相同的线性组合而重构出来. 从而, 原空间邻域内样本之间的线性关系在降维后的低维空间以保持.

从统计的角度

• $Y1=0$消去各个维度均值不确定性.
• $Y{Y}^T=I$消去各个维度方差不确定性,消去各个维度之间线性相关性不确定性.

从几何的角度

• $Y1=0$消去平移不确定性.
• $Y{Y}^T=I$消去伸缩不确定性.

证明:

• $Y1=0$消去平移不确定性?
  $Y^{\prime }=Y+v{1}^T$, $Y^{\prime }1=0.$.
  $Y^{\prime }1=Y1+v{1}^{T}1=0+v=v\stackrel{\mathrm{s}\mathrm{e}\mathrm{t}}{=}0\Rightarrow v=0$
  $v$没有自由度, 所以消去了平移不确定性.
• $Y{Y}^T=I$消去旋转不确定性?
  $Y^{\prime }=QY$, $Q^{T}Q=I$, $Y^{\prime }1=0$, $Y^{\prime }{Y}^{\prime T}=I$.
  $Y^{\prime }1=QY1=0\stackrel{\mathrm{s}\mathrm{e}\mathrm{t}}{\equiv }0$
  $Y^{\prime }{Y}^{\prime T}=QY{Y}^T{Q}^T=Q{Q}^T=I\stackrel{\mathrm{s}\mathrm{e}\mathrm{t}}{\equiv }I$
  $Q$仍有自由度, 所以没有消去旋转不确定性.
• $Y{Y}^T=I$消去伸缩不确定性?
  $Y^{\prime }=aY$, $a\ne 0$, $Y^{\prime }{Y}^{\prime T}=I$.
  $Y^{\prime }{Y}^{\prime T}=(aY)(aY{)}^T=a^{2}Y{Y}^T=a^{2}I\stackrel{\mathrm{s}\mathrm{e}\mathrm{t}}{=}I\Rightarrow a=1$
  $a$没有自由度, 所以消去了伸缩不确定性.

没有消去旋转不确定性, 会对新的表示产生负面的影响.

• $y_i$分量之间可能独立, 但由于没有消去旋转不确定, 实际得到$y_i$分量之间不独立.
• $Y$有稀疏解, 但由于没有消去旋转不确定性, 实际得到一个稠密解.
  实际上$Q^{T}Q=I$, $Q$的语义除了旋转, 还有反射和排列.
  从信号处理的角度, 消去旋转不确定性相当于盲信号分离(通常假设信号之间独立).
  从机器学习的角度, 我们可以添加适当的正则化项, 例如$\Vert Y{\Vert }_1$, 迫使尽量多的分量与坐标轴同向, 消除一部分旋转不确定性, 进而获得稀疏解.

d LLE 低维表示 优化
引理 迹(trace)的性质
tr(AB) = tr(BA)   tr(ABC) = tr(CAB) = tr(BCA)

$y_i\in {\mathbb{R}}^{d^{\prime }\times 1}$, $w_i\in {\mathbb{R}}^{1\times n}$, $Y=[y_1\dots y_n]$, $w_i=[w_{i1}\dots w_{in}]$.
$(e_i{)}_k=\{\begin{array}{ll}1& \mathrm{i}\mathrm{f}k=1\\ 0& \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}$, $I=[e_1\dots e_n]$.

注意$(I-W^T{)}^T$和$Y^{T}Y$和$(I-W^T)$都是$n\times n$的方形矩阵, 可以使用引理.

∑ i = 1 n ∥ y i − ∑ j = 1 n w i j y j ∥ 2 2 = ∑ i = 1 n ∥ Y e i − Y w i T ∥ 2 2 = ∥ Y I − Y W T ∥ F 2 = ∥ Y ( I − W T ) ∥ F 2 = t r { [ Y ( I − W T ) ] T [ Y ( I − W T ) ] } = t r [ ( I − W T ) T Y T Y ( I − W T ) ] = t r [ ( I − W T ) ( I − W T ) T Y T Y ] = t r [ ( I − W ) T ( I − W ) Y T Y ] = t r [ M Y T Y ] = ∑ k = 1 n ∑ i = 1 n M k i ( Y T Y ) i k = ∑ k = 1 n ∑ i = 1 n M k i y i T y k \begin{aligned} \sum_{i=1}^{n} \left\| y_i - \sum_{j=1}^{n} w_{ij} y_j \right\|_2^2 &= \sum_{i=1}^{n} \left\| Ye_i - Yw_i^T \right\|_2^2 \\ &= \left\| YI - YW^T \right\|_F^2 \\ &= \left\| Y(I-W^T) \right\|_F^2 \\ &= {\rm tr}\{[Y(I-W^T)]^T[Y(I-W^T)]\} \\ &= {\rm tr}[(I-W^T)^TY^TY(I-W^T)] \\ &= {\rm tr}[(I-W^T)(I-W^T)^TY^TY] \\ &= {\rm tr}[(I-W)^T(I-W)Y^TY] \\ &= {\rm tr}[MY^TY] \\ &= \sum_{k=1}^{n}\sum_{i=1}^{n} M_{ki} \left(Y^TY\right)_{ik} \\ &= \sum_{k=1}^{n}\sum_{i=1}^{n} M_{ki} y_i^Ty_k \\ \end{aligned} i=1∑n​∥∥∥∥∥​yi​−j=1∑n​wij​yj​∥∥∥∥∥​22​​=i=1∑n​∥∥​Yei​−YwiT​∥∥​22​=∥∥​YI−YWT∥∥​F2​=∥∥​Y(I−WT)∥∥​F2​=tr{[Y(I−WT)]T[Y(I−WT)]}=tr[(I−WT)TYTY(I−WT)]=tr[(I−WT)(I−WT)TYTY]=tr[(I−W)T(I−W)YTY]=tr[MYTY]=k=1∑n​i=1∑n​Mki​(YTY)ik​=k=1∑n​i=1∑n​Mki​yiT​yk​​

e LLE 低维表示 优化 求解

$$\begin{array}{rl}M& =(I-W{)}^T(I-W)\\ & =(I-W^T)(I-W)\\ & =I-W-W^T+W^{T}W\end{array}$$

e.1 $M$的半正定性

要证明$M$是半正定矩阵, 只需证明对任意$n$维向量$v\ne 0$都有$v^{T}Mv⩾0$

• $v^{T}Mv=v^T(I-W{)}^T(I-W)v=[(I-W)v{]}^T[(I-W)v]=\Vert (I-W)v{\Vert }_2^2⩾0$
  $v^{T}Xv=(L^{T}v{)}^{T}D(L^{T}v)v$
• 综上所述, 对任意$n$维向量$v\ne 0$都有$v^{T}Mv⩾0$, 即$M$是半正定矩阵.

e.2 $M$的特征向量$1$

注意$w_{i}1=1$, 所以$W1=1$.
(注意${\vec{w}}_i\vec{1}=1$, 所以$\mathbf{W}\vec{1}=\vec{1}$.)

$$\begin{array}{rl}M1& =(I-W-W^T+W^{T}W)1\\ & =I1-W1-W^{T}1+W^{T}W1\\ & =1-1-W^{T}1+W^{T}1\\ & =0\end{array}$$

注意$M1=01$, 所以$M$的特征值$0$有一个特征向量是$1$.
(注意$\mathbf{M}\vec{1}=0\cdot \vec{1}$, 所以$\mathbf{M}$的特征值$0$有一个特征向量是$\vec{1}$.)

f 实际求解过程中, 对$M$特征值分解, 特征值升序排列, 去除最小特征值的特征向量${\xi }_1$, $Y^T=[y_1\dots y_n{]}^T=[{\xi }_2\dots {\xi }_{d^{\prime }+1}]$就是最终的低维表示.

f.1 舍弃${\xi }_1$, 实际上是舍弃$1$

• 由于$M$是半正定矩阵, 所以$M$的特征值及其特征向量
  $$\begin{array}{r}0⩽{\sigma }_1⩽{\sigma }_2⩽\cdots ⩽{\sigma }_n\\ {\xi }_1{\xi }_2\dots {\xi }_n\end{array}$$
• 又$M$具有特征值$0$及其特征向量是$1$, 所以$M$的特征值及其特征向量
  $$\begin{array}{r}0={\sigma }_1⩽{\sigma }_2⩽\cdots ⩽{\sigma }_n\\ 1{\xi }_2\dots {\xi }_n\end{array}$$

f.2 舍弃$1$的原因

• 从直观的角度, 每个$y_i$的这个分量都是1, 因此这个分量不包含任何信息, 所以舍弃.
• 从优化的角度, 舍弃$1$能确保约束条件$\sum _{i=1}^n{y}_i=0$.
  • 如果不舍弃$1$, 由于每个$y_i$的这个分量都是1, 所以$\left(\sum _{i=1}^n{y}_i\right)$的这个分量都是$n$, 不等于0.
  • 如果舍弃$1$, 由于$M$矩阵是规正矩阵(实对称矩阵/Hermitian矩阵是规正矩阵的子集), 所以$M$矩阵的特征向量相互正交.
    首先注意到$Y^T=[y_1\dots y_n{]}^T=[{\xi }_2\dots {\xi }_{d^{\prime }+1}]$
    然后注意到${\xi }_1^T{\xi }_i=0,i\ne 1$
    即${\xi }_1^T[{\xi }_2\dots {\xi }_{d^{\prime }+1}]=0$
    即$1^T[{\xi }_2\dots {\xi }_{d^{\prime }+1}]=0$
    即$1^T[{\xi }_2\dots {\xi }_{d^{\prime }+1}]=0$
    即$1^T{Y}^T=0$
    即$Y1=0$
    即$\sum _{i=1}^n{y}_i=0$