统计学习（九）：神经网络

神经网络

感知器

$$f(x)=\sum _{j=0}^p{w}_j{x}_j=\sum _{j=1}^p{w}_j{x}_j+w_0={\mathbf{w}}^T\mathbf{x}$$

用感知器分类

$$f(x)=s(\sum _{j=1}^p{w}_j{x}_j+w_0)=s({\mathbf{w}}^T\mathbf{x})$$

$s$ 是一个阈值函数。分类：$f(x)>0$ 或 $f(x)<0$。

如果不是简单的决定正负（+、-），那么可以构造：
$$\begin{gathered} f(x)=\sigma (\sum _{j=1}^p{w}_j{x}_j+w_0)=\sigma ({\mathbf{w}}^T\mathbf{x}) \\ \sigma (u)=\frac{1}{1+\exp (-u)} \end{gathered}$$

一维的总结

回归：
$$f(x)=\sum _{j=1}^p{w}_j{x}_j+w_0$$
分类：
$$f(x)=1/(1+\exp [-(\sum _{j=1}^p{w}_j{x}_j+w_0)])$$

多类分类

选择 C k : f k ( x ) = max ⁡ l ∈ { 1 , ⋯   , K } f l ( x ) C_k:f_k(x)=\max_{l\in\{1,\cdots,K\}}f_l(x) Ck​:fk​(x)=maxl∈{1,⋯,K}​fl​(x)。

为得到概率，使用 softmax：
$$\begin{gathered} \sigma (u)=\frac{1}{1+e^{-u}}=\frac{e^u}{1+e^u} \\ {f}_k(x)=\frac{\exp (o_k)}{{\sum }_{l=1}^K\exp (o_l)},o_k={\mathbf{w}}^{kT}\mathbf{x} \end{gathered}$$
如果一个类的输出足够大于其他类，则其softmax将接近1（否则为0）。

训练感知器

随机梯度下降：从随机权重开始，在每个点，调整权重以最小化误差。

通用的Update rule：
$$\Delta w_j=-\eta \frac{\partial Error(f(x^i),y^i)}{\partial w_j}$$
在每个训练实例后，对于每个权重：
$$\begin{gathered} w_i^{(t+1)}=w_j^{(t)}+\Delta w_i^{(t)} \\ {w}_j\leftarrow w_j+\Delta w_j \end{gathered}$$

回归

$$Error(f(x^i),y^i)=\frac{1}{2}(y^i-f(x^i){)}^2=\frac{1}{2}(y^i-{\mathbf{w}}^T{\mathbf{x}}^i{)}^2$$

对于回归的Update rule为：
$$\Delta w_j=\eta (y^i-f(x^i))x_j^i$$
Sigmoid output:
$$f(x^i)=\sigma ({\mathbf{w}}^T{\mathbf{x}}^i),\sigma (u)=\frac{1}{1+e^{-u}},{\sigma }^{\prime }(u)=u^{\prime }\sigma (u)(1-\sigma (u))$$
Cross-entropy error:
$$Error(f(x^i),y^i)=-y^i\log f(x^i)-(1-y^i)\log (1-f(x^i))$$

K类

K>2的 softmax输出：
$$f_k(x^i)=\frac{\exp ({\mathbf{w}}^{kT}{\mathbf{x}}^i)}{{\sum }_{l=1}^K\exp ({\mathbf{w}}^{lT}{\mathbf{x}}^i)}$$
Cross-entropy error:
$$Error(f(x^i),y^i)=-\sum _{k=1}^K{y}_k^i\log f_k(x^i)$$
Update rule
$$\Delta w_j=\eta (y^i-f(x^i))x_j^i$$

布尔函数

AND

$$f(x)=s(w_0+w_1{x}_1+w_2{x}_2)$$
理想的分界线：

XOR

$$\begin{gathered} f(x)=s(w_0+w_1{x}_1+w_2{x}_2) \\ {w}_0\le 0 \\ {w}_0+w_2>0 \\ {w}_0+w_1>0 \\ {w}_0+w_1+w_2\le 0 \end{gathered}$$

多层感知器

隐藏层：

隐藏单元的输出：
$$z_h=\frac{1}{1+e^{-{\mathbf{w}}_h^T\mathbf{x}}}$$
神经网络输出：
$$\begin{gathered} f(x)={\mathbf{v}}^T\mathbf{z}=v_0+\sum _{h=1}^H\frac{v_h}{1+e^{-{\mathbf{w}}_h^T\mathbf{x}}} \\ \frac{\partial E}{\partial w_{hj}}=\frac{\partial E}{\partial f(x)}\frac{\partial f(x)}{\partial z_h}\frac{\partial z_h}{\partial w_{hj}} \\ \frac{\partial f(x)}{\partial z_h}\to v_h,\frac{\partial z_h}{\partial w_{hj}}\to z_h^i(1-z_h^i)x_j^i \end{gathered}$$

回归

$$\begin{gathered} Error(f(x^i),y^i)=\frac{1}{2}(y^i-f(x^i){)}^2 \\ f(x)=v_0+\sum _{h=1}^H{v}_h{z}_h,z_h=\sigma ({\mathbf{w}}_h^T\mathbf{x}) \\ \Delta v_h={\eta }_1(y^i-f(x^i))z_h^i \\ \Delta w_{hj}=-\eta \frac{\partial E^i}{\partial w_{hj}}=-\eta \frac{\partial E^i}{\partial f(x^i)}\frac{\partial f(x^i)}{\partial z_h^i}\frac{\partial z_h^i}{\partial w_{hj}} \\ =-\eta -(y^i-f(x^i))v_h{z}_h^i(1-z_h^i)x_j^i \end{gathered}$$

初始化所有的 $v_h,w_{hj}$ 到$(-0.01,0.01)$的随机数，重复，直到收敛：
$$\begin{gathered} fori=1,\cdots ,n \\ forh=1,\cdots ,H \\ {z}_h^i=\sigma ({\mathbf{w}}_h^T{\mathbf{x}}^i) \\ f(x^i)={\mathbf{v}}^T{\mathbf{z}}^i \\ forh=1,\cdots ,H \\ \Delta v_h={\eta }_1(y^i-f(x^i))z_h^i \\ forh=1,\cdots ,H \\ forj=1,\cdots ,p \\ \Delta w_{hj}=\eta (y_k^i-f_k(x^i))v_h{z}_h^i(1-z_h^i)x_j^i \\ forh=1,\cdots ,H \\ {v}_h\leftarrow v_h+\Delta v_h \\ forj=1,\cdots ,p \\ {w}_{hj}\leftarrow w_{hj}+\Delta w_{hj} \end{gathered}$$

分类

$$\begin{gathered} z_h=\sigma ({\mathbf{w}}_h^T\mathbf{x}) \\ f(x)=\sigma (v_0+\sum _{h=1}^H{v}_h{z}_h) \\ error=-\sum _{i=1}^n{y}^i\log (f(x^i))+(1-y^i)\log (1-f(x^i)) \\ \Delta v_h={\eta }_1\sum _{i=1}^n(y^i-f(x^i))z_h^i \\ \Delta w_{hj}=\eta \sum _{i=1}^n(y^i-f(x^i))v_h{z}_h^i(1-z_h^i)x_j^i \end{gathered}$$

K类

$$\begin{gathered} o_k^i=v_{k0}+\sum _{h=1}^H{v}_{kh}{z}_h^i \\ {f}_k(x)=\frac{\exp (o_k^i)}{{\sum }_{l=1}^K\exp (o_l^i)} \\ error=-\sum _{i=1}^n\sum _{k=1}^K{y}_k^i\log (f_k(x^i)) \\ \Delta v_{kh}={\eta }_1\sum _{i=1}^n(y_k^i-f_k(x^i))z_h^i \\ \Delta w_{hj}=\eta \sum _{i=1}^n(\sum _{k=1}^K(y_k^i-f_k(x^i))v_{kh})z_h^i(1-z_h^i)x_j^i \end{gathered}$$

多个隐藏层

$$\begin{gathered} f(x)=v_0+\sum _{l=1}^{H_2}{v}_l{z}_{2l} \\ {z}_{2l}=\sigma ({\mathbf{w}}_{2l}^T{\mathbf{z}}_1)=\sigma (w_{2l0}+\sum _{h=1}^{H_1}{w}_{2lh}{z}_{1h}) \\ {z}_{1h}=\sigma ({\mathbf{w}}_{1h}^T\mathbf{x})=\sigma (w_{1h0}+\sum _{j=1}^p{w}_{1hj}{x}_j) \\ f(x)=v_0+\sum _{l=1}^{H_2}{v}_l\cdot \sigma (w_{2l0}+\sum _{h=1}^{H_1}{w}_{2lh}\cdot \sigma (w_{1h0}+\sum _{j=1}^p{w}_{1hj}{x}_j)) \end{gathered}$$