三、线性模型笔记
来源: <机器学习>
作者:周志华
3.1 线性回归
线性回归(linear model)是在假设特征满足线性关系,根据给定的训练数据训练一个模型,并用此模型进行预测:
函数形式:
向量表达式形式:
注意:
- 非线性模型可在其基础上引入层次结构和高维映射获得
- w w w表达了各属性( x 1 , x 2 , x 3 . . . x i x_{1}, x_{2}, x_{3}... x_{i} x1,x2,x3...xi)的 “权重” >>> 线性模型具有较好解释性
3.1.1 几个概念复习
(1) 欧式距离
欧氏距离是最常见的距离度量,衡量的是多维空间中各个点之间的绝对距离。公式如下:
dist ( X , Y ) = ∑ i = 1 n ( x i − y i ) 2 \operatorname{dist}(X, Y)=\sqrt{\sum_{i=1}^{n}\left(x_{i}-y_{i}\right)^{2}} dist(X,Y)=i=1∑n(xi−yi)2
欧氏距离和余弦相似度的区别: 余弦相似度更加注重两个向量在方向上的差异,而非距离或长度上
(2) 均方误差, 范数
均方误差 MSE即预测点与实际点之间距离之差平方和的均值,又被称为 L2范数损失:
MSE ( y , y ^ ) = 1 n samples ∑ i = 0 n samples − 1 ( y i − y ^ i ) 2 . \text{MSE}(y, \hat{y}) = \frac{1}{n_\text{samples}} \sum_{i=0}^{n_\text{samples} - 1} (y_i - \hat{y}_i)^2. MSE(y,y^)=nsamples1i=0∑nsamples−1(yi−y^i)2.
均方误差对应欧式距离 >>> 有较好几何意义
- 范数复习: https://zhuanlan.zhihu.com/p/85305655
如 w T x i − y i w^Tx_i-y_i wTxi−yi的 2-范 (欧式范数,计算向量长度) : ∥ w T x i − y i ∥ 2 = ∑ i = 1 n ( w T x i − y i ) 2 \|w^Tx_i-y_i\|_{2}=\sqrt{\sum_{i=1}^{n} (w^Tx_i-y_i)^2} ∥wTxi−yi∥2=∑i=1n(wTxi−yi)2
(3) 最小二乘估计
- 基于均方误差最小化来进行模型求解的方法 ---------- 最小二乘法
在线性回归中, 即找到一条直线, 使所有样本到直线上欧式距离最小
3.1.2 最小二乘求 f ( x i ) f(x_i) f(xi) , w w w, b b b
1.最简单一元线性情况
对于以上线性模型即, 确定 w w w和 b b b, 使得 f ( x ) f(x) f(x)与 y y y间均方误差最小:
( w ∗ , b ∗ ) = arg min ( w , b ) ∑ i = 1 m ( f ( x i ) − y i ) 2 = arg min ( w , b ) ∑ i = 1 m ( y i − w x i − b ) 2 . \begin{aligned}\left(w^{*}, b^{*}\right) &=\underset{(w, b)}{\arg \min } \sum_{i=1}^{m}\left(f\left(x_{i}\right)-y_{i}\right)^{2} \\ &=\underset{(w, b)}{\arg \min } \sum_{i=1}^{m}\left(y_{i}-w x_{i}-b\right)^{2} . \end{aligned} (w∗,b∗)=(w,b)argmini=1∑m(f(xi)−yi)2=(w,b)argmini=1∑m(yi−wxi−b)2.
即求 E ( w , b ) = ∑ i = 1 m ( y i − w x i − b ) 2 E_{(w, b)}=\sum_{i=1}^{m}\left(y_{i}-w x_{i}-b\right)^{2} E(w,b)=∑i=1m(yi−wxi−b)2最小化过程, E ( w , b ) E_{(w, b)} E(w,b)为关于w和b的凸函数, 即偏导为0过程:
∂ E ( w , b ) ∂ w = 2 ( w ∑ i = 1 m x i 2 − ∑ i = 1 m ( y i − b ) x i ) = 0 \frac{\partial E_{(w, b)}}{\partial w}=2\left(w \sum_{i=1}^{m} x_{i}^{2}-\sum_{i=1}^{m}\left(y_{i}-b\right) x_{i}\right) = 0 ∂w∂E(w,b)=2(wi=1∑mxi2−i=1∑m(yi−b)xi)=0
∂ E ( w , b ) ∂ b = 2 ( m b − ∑ i = 1 m ( y i − w x i ) ) = 0 \frac{\partial E_{(w, b)}}{\partial b}=2\left(m b-\sum_{i=1}^{m}\left(y_{i}-w x_{i}\right)\right) = 0 ∂b∂E(w,b)=2(mb−i=1∑m(yi−wxi))=0
→→→→→→→→→→→→→→→→→→→→→公式详解来源: 南瓜书
先求 b, 易得:
b = 1 m ∑ i = 1 m ( y i − w x i ) b=\cfrac{1}{m}\sum_{i=1}^{m}(y_i-wx_i) b=m1i=1∑m(yi−wxi)
因为 1 m ∑ i = 1 m y i = y ˉ , 1 m ∑ i = 1 m x i = x ˉ , 则 b = y ˉ − w x ˉ \cfrac{1}{m}\sum_{i=1}^{m}y_i=\bar{y} , \cfrac{1}{m}\sum_{i=1}^{m}x_i=\bar{x},则b=\bar{y}-w\bar{x} m1∑i=1myi=yˉ,m1∑i=1mxi=xˉ,则b=yˉ−wxˉ带入 w w w偏导求 w w w:
w ∑ i = 1 m x i 2 = ∑ i = 1 m y i x i − y ˉ ∑ i = 1 m x i + w x ˉ ∑ i = 1 m x i w \sum_{i=1}^{m} x_{i}^{2}=\sum_{i=1}^{m} y_{i} x_{i}-\bar{y} \sum_{i=1}^{m} x_{i}+w \bar{x} \sum_{i=1}^{m} x_{i} wi=1∑mxi2=i=1∑myixi−yˉi=1∑mxi+wxˉi=1∑mxi
化简得:
w = ∑ i = 1 m y i x i − y ˉ ∑ i = 1 m x i ∑ i = 1 m x i 2 − x ˉ ∑ i = 1 m x i w=\frac{\sum_{i=1}^{m} y_{i} x_{i}-\bar{y} \sum_{i=1}^{m} x_{i}}{\sum_{i=1}^{m} x_{i}^{2}-\bar{x} \sum_{i=1}^{m} x_{i}} w=∑i=1mxi2−xˉ∑i=1mxi∑i=1myixi−yˉ∑i=1mxi
由 1 m ∑ i = 1 m y i = y ˉ , 1 m ∑ i = 1 m x i = x ˉ \cfrac{1}{m}\sum_{i=1}^{m}y_i=\bar{y} ,\cfrac{1}{m}\sum_{i=1}^{m}x_i=\bar{x} m1∑i=1myi=yˉ,m1∑i=1mxi=xˉ化简得:
w = ∑ i = 1 m y i ( x i − x ˉ ) ∑ i = 1 m x i 2 − 1 m ( ∑ i = 1 m x i ) 2 w=\frac{\sum_{i=1}^{m} y_{i}\left(x_{i}-\bar{x}\right)}{\sum_{i=1}^{m} x_{i}^{2}-\frac{1}{m}\left(\sum_{i=1}^{m} x_{i}\right)^{2}} w=∑i=1mxi2−m1(∑i=1mxi)2∑i=1myi(xi−xˉ)
将上式向量化以便在numpy实现, 将上式向量化:
w w w分子中: ∑ i = 1 m x ˉ y i = x ˉ ⋅ m ⋅ 1 m ⋅ ∑ i = 1 m y i = m x ˉ y ˉ = ∑ i = 1 m x ˉ y ˉ \sum_{i=1}^{m}\bar{x}y_i =\bar{x} ⋅ m ⋅ \cfrac{1}{m}⋅ \sum_{i=1}^{m}y_i=m\bar{x}\bar{y}=\sum_{i=1}^{m}\bar{x}\bar{y} ∑i=1mxˉyi=xˉ⋅m⋅m1⋅∑i=1myi=mxˉyˉ=∑i=1mxˉyˉ
w w w分母中: ∑ i = 1 m x ˉ x i = x ˉ ⋅ m ⋅ 1 m ⋅ ∑ i = 1 m x i = m x ˉ 2 = ∑ i = 1 m x ˉ 2 \sum_{i=1}^{m}\bar{x}x_i =\bar{x} ⋅ m ⋅ \cfrac{1}{m}⋅ \sum_{i=1}^{m}x_i=m\bar{x}^2=\sum_{i=1}^{m}\bar{x}^2 ∑i=1mxˉxi=xˉ⋅m⋅m1⋅∑i=1mxi=mxˉ2=∑i=1mxˉ2
于是有:
w = ∑ i = 1 m ( y i x i − y i x ˉ − x i y ˉ + x ˉ y ˉ ) ∑ i = 1 m ( x i 2 − x i x ˉ − x i x ˉ + x ˉ 2 ) = ∑ i = 1 m ( x i − x ˉ ) ( y i − y ˉ ) ∑ i = 1 m ( x i − x ˉ ) 2 \begin{aligned} w &=\frac{\sum_{i=1}^{m}\left(y_{i} x_{i}-y_{i} \bar{x}-x_{i} \bar{y}+\bar{x} \bar{y}\right)}{\sum_{i=1}^{m}\left(x_{i}^{2}-x_{i} \bar{x}-x_{i} \bar{x}+\bar{x}^{2}\right)} \\ &=\frac{\sum_{i=1}^{m}\left(x_{i}-\bar{x}\right)\left(y_{i}-\bar{y}\right)}{\sum_{i=1}^{m}\left(x_{i}-\bar{x}\right)^{2}} \end{aligned} w=∑i=1m(xi2−xixˉ−xixˉ+xˉ2)∑i=1m(yixi−yixˉ−xiyˉ+xˉyˉ)=∑i=1m(xi−xˉ)2∑i=1m(xi−xˉ)(yi−yˉ)
若令 x = ( x 1 , x 2 , . . . , x m ) T \boldsymbol{x}=(x_1,x_2,...,x_m)^T x=(x1,x2,...,xm)T,