机器学习-线性模型数学推导和实现-Linear Model

1.线性模型的数学推导

1.1.简单的线性模型的数学推导

我们先考虑一个简单的线性模型，表达式如下：
$$f(x)=w_{1}x+w_0$$
其中，$w_1,w_0\in {\mathbb{R}}^1$，我们的损失函数定义为
$$L(y^{\prime },y^{\ast })=\frac{1}{2N}(y^{\prime }-y^{\ast }{)}^2$$
其中，$y^{\prime }$是模型预测的标签，$y^{\ast }$是真实的标签，$N$是样本的个数。那么我们的$Empirical$ $Risk$为
$$\begin{array}{rl}\underset{f\in H}{\min }{R}_{emp}(f)& =\underset{w_0,w_1}{\min }\frac{1}{2N}\sum _{i=1}^N(f(x_i)-y_i)\\ & =\underset{w_0,w_1}{\min }\frac{1}{2N}\sum _{i=1}^N(w_0+w_{1}x-y_i)\end{array}$$
然后令$R_{emp}$对$w_0和w_1$的偏导等于0，得到
$$\begin{array}{rl}{\hat{w}}_1& =\frac{\overline{xy}-\bar{x}\bar{y}}{\overline{(x^2)}-(\bar{x}{)}^2}\\ {\hat{w}}_0& =\bar{y}-{\hat{w}}_1\bar{x}\end{array}$$
求得$w_0和w_1$后，我们就可以带入$f(x)$中，得到训练出来的模型为
$$\hat{f}(x)={\hat{w}}_0+{\hat{w}}_{1}x$$

1.2.一般化的推导

1.2.1.无加权线性模型的数学推导¹

一般化的线性模型的表达式为
$$\begin{array}{rl}f(x)& =\sum _{j=1}^M{w}_j{\varphi }_j(x)\\ & =w_1{\varphi }_1(x)+w_2{\varphi }_2(x)+\cdots +w_M{\varphi }_M(x)\end{array}$$
其中，$M$为数据集的特征的个数，也是$X\_train$矩阵的列数。${\varphi }_j(x)$为$basisfunction$。同样的，我们令损失函数为$meansquarelossfunction$，那么我们的$Empirical$ $Risk$为
min ⁡ f ∈ H R e m p ( f ) = min ⁡ w ∈ R M R e m p ( w ) = min ⁡ w ∈ R M 1 2 N ∑ i = 1 N [ f ( x i ) − y i ] 2 = min ⁡ w ∈ R M 1 2 N ∑ i = 1 N { ∑ j = 1 M w j ϕ j ( x i ) − y i } 2 = min ⁡ w ∈ R M 1 2 N ∑ i = 1 N { ( w 1 w 2 ⋯ w M ) ( ϕ 1 ( x i ) ϕ 2 ( x i ) ⋮ ϕ M ( x i ) ) − y i } 2 = min ⁡ w ∈ R M 1 2 N ∑ i = 1 N [ W T ϕ ( x i ) − y i ] 2 = min ⁡ w ∈ R M 1 2 N ( W T ϕ ( x 1 ) − y 1 W T ϕ ( x 2 ) − y 2 ⋯ W T ϕ ( x N ) − y N ) ( W T ϕ ( x 1 ) − y 1 W T ϕ ( x 2 ) − y 2 ⋮ W T ϕ ( x N ) − y N ) = min ⁡ w ∈ R M 1 2 N [ W T ( ϕ ( x 1 ) ϕ ( x 2 ) ⋯ ϕ ( x N ) ) − ( y 1 y 2 ⋯ y N ) ] ( W T ϕ ( x 1 ) − y 1 W T ϕ ( x 2 ) − y 2 ⋮ W T ϕ ( x N ) − y N ) = min ⁡ w ∈ R M 1 2 N ( W T Φ T − y T ) ( W T Φ T − y T ) T = min ⁡ w ∈ R M 1 2 N ( W T Φ T − y T ) ( Φ W − y ) = min ⁡ w ∈ R M 1 2 N ( W T Φ T Φ W − W T Φ T y − y T Φ W + y T y ) = min ⁡ w ∈ R M 1 2 N ( W T Φ T Φ W − 2 W T Φ T y + y T y ) \begin{aligned} \min_{f\in H}R_{emp}(f) &= \min_{w\in R^M}R_{emp}(w) \\ &= \min_{w\in R^M}\frac{1}{2N}\sum_{i=1}^{N}[f(x_i) - y_i]^2 \\ &= \min_{w\in R^M}\frac{1}{2N}\sum_{i=1}^{N}\Big \{\sum_{j=1}^{M}w_j\phi_j(x_i) -y_i \Big \}^2 \\ &= \min_{w\in R^M}\frac{1}{2N}\sum_{i=1}^{N}\bigg \{ \left( \begin{array}{ccc} w_1 & w_2 & \cdots & w_M \end{array} \right) \left( \begin{array}{c} \phi_1(x_i) \\ \phi_2(x_i) \\ \vdots \\ \phi_M(x_i) \end{array} \right) - y_i \bigg \}^2 \\ &= \min_{w\in R^M}\frac{1}{2N}\sum_{i=1}^{N}[ W^T\phi(x_i) - y_i]^2 \\ &= \min_{w\in R^M}\frac{1}{2N} \left( \begin{array}{ccc} W^T\phi(x_1) - y_1 & W^T\phi(x_2) - y_2 & \cdots & W^T\phi(x_N) - y_N \end{array} \right) \left( \begin{array}{c} W^T\phi(x_1) - y_1 \\ W^T\phi(x_2) - y_2 \\ \vdots \\ W^T\phi(x_N) - y_N \end{array} \right) \\ &= \min_{w\in R^M}\frac{1}{2N}\Big[W^T \left( \begin{array}{ccc} \phi(x_1)& \phi(x_2) & \cdots & \phi(x_N) \end{array} \right) - \left( \begin{array}{ccc} y_1& y_2 & \cdots & y_N \end{array} \right)\Big] \left( \begin{array}{c} W^T\phi(x_1) - y_1 \\ W^T\phi(x_2) - y_2 \\ \vdots \\ W^T\phi(x_N) - y_N \end{array} \right) \\ &= \min_{w\in R^M}\frac{1}{2N}(W^T\Phi^T - y^T) (W^T\Phi^T - y^T)^T \\ &= \min_{w\in R^M}\frac{1}{2N}(W^T\Phi^T - y^T) (\Phi W - y) \\ &= \min_{w\in R^M}\frac{1}{2N}(W^T\Phi^T\Phi W - W^T\Phi^Ty - y^T\Phi W + y^Ty) \\ &= \min_{w\in R^M}\frac{1}{2N}(W^T\Phi^T\Phi W - 2W^T\Phi^Ty + y^Ty) \end{aligned} f∈Hmin​Remp​(f)​=w∈RMmin​Remp​(w)=w∈RMmin​2N1​i=1∑N​[f(xi​)−yi​]2=w∈R