一元线性回归
推导思路
详细推导过程:
由最小二乘法导出 E ( w , b ) E(w,b) E(w,b)
E ( w , b ) = ∑ i = 1 m ( y i − f ( x i ) ) 2 = ∑ i = 1 m ( y i − ( w x i + b ) ) 2 = ∑ i = 1 m ( y i − w x i − b ) 2 \begin{aligned} E_{(w, b)} &=\sum_{i=1}^{m}\left(y_{i}-f\left(x_{i}\right)\right)^{2} \\ &=\sum_{i=1}^{m}\left(y_{i}-\left(w x_{i}+b\right)\right)^{2} \\ &=\sum_{i=1}^{m}\left(y_{i}-w x_{i}-b\right)^{2} \end{aligned} E(w,b)=i=1∑m(yi−f(xi))2=i=1∑m(yi−(wxi+b))2=i=1∑m(yi−wxi−b)2(对应公式3.4)
证明 E ( w , b ) E(w,b) E(w,b) 是关于 w w w 和 b b b 的凸函数
二元函数判断凹凸性:
设 f ( x , y ) f(x,y) f(x,y) 在区域 D D D 上具有二阶连续偏导数,记 A = f x x ′ ′ ( x , y ) A=f_{x x}^{\prime \prime}(x, y) A=fxx′′(x,y), B = f x y ′ ′ ( x , y ) B=f_{x y}^{\prime \prime}(x, y) B=fxy′′(x,y), C = f y y ′ ′ ( x , y ) C=f_{y y}^{\prime \prime}(x, y) C=fyy′′(x,y) 则:
( 1 ) (1) (1) 在 D D D 上恒有 A > 0 A>0 A>0,且 A C − B 2 ≥ 0 A C-B^{2} \geq 0 AC−B2≥0 时, f ( x , y ) f(x,y) f(x,y) 在区域 D D D 上是凸函数;
( 2 ) (2) (2) 在 D D D 上恒有 A < 0 A<0 A<0,且 A C − B 2 ≥ 0 A C-B^{2} \geq 0 AC−B2≥0 时, f ( x , y ) f(x,y) f(x,y) 在区域 D D D 上是凹函数。
二元凹凸函数求最值:
设 f ( x , y ) f(x,y) f(x,y) 在开区域 D D D 内具有连续偏导数的凸(或凹)函数, ( x 0 , y 0 ) ∈ D \left(x_{0}, y_{0}\right) \in D (x0,y0)∈D 且 f x ′ ( x 0 , y 0 ) = 0 f_{x}^{\prime}\left(x_{0}, y_{0}\right)=0 fx′(x0,y0)=0, f y ′ ( x 0 , y 0 ) = 0 f_{y}^{\prime}\left(x_{0}, y_{0}\right)=0 fy′(x0,y0)=0 ,则 f ( x 0 , y 0 ) f(x_{0},y_{0}) f(x0,y0) 为 f ( x , y ) f(x,y) f(x,y) 在 D D D 内的最小(或大)值。
- 求 A = f x x ′ ′ ( x , y ) A=f_{x x}^{\prime \prime}(x, y) A=fxx′′(x,y), B = f x y ′ ′ ( x , y ) B=f_{x y}^{\prime \prime}(x, y) B=fxy′′(x,y) 和 C = f y y ′ ′ ( x , y ) C=f_{y y}^{\prime \prime}(x, y) C=fyy′′(x,y)
求一阶偏导数 f x ′ ( x , y ) f_{x}^{\prime}(x, y) fx′(x,y)
∂ E ( w , b ) ∂ w = ∂ ∂ w [ ∑ i = 1 m ( y i − w x i − b ) 2 ] = ∑ i = 1 m ∂ ∂ w ( y i − w x i − b ) 2 = ∑ i = 1 m 2 ⋅ ( y i − w x i − b ) ⋅ ( − x i ) = 2 ( w ∑ i = 1 m x i 2 − ∑ i = 1 m ( y i − b ) x i ) \begin{aligned} \frac{\partial E_{(w, b)}}{\partial w} &=\frac{\partial}{\partial w}\left[ \sum_{i=1}^{m} \left(y_{i}-w x_{i}-b\right)^{2}\right] \\ &=\sum_{i=1}^{m} \frac{\partial}{\partial w}\left(y_{i}-w x_{i}-b\right)^{2} \\ &=\sum_{i=1}^{m} 2 \cdot\left(y_{i}-w x_{i}-b\right) \cdot\left(-x_{i}\right) \\ &=2\left(w \sum_{i=1}^{m} x_{i}^{2}-\sum_{i=1}^{m}\left(y_{i}-b\right) x_{i}\right) \end{aligned} ∂w∂E(w,b)=∂w∂[i=1∑m(yi−wxi−b)2]=i=1∑m∂w∂(yi−wxi−b)2=i=1∑m2⋅(yi−wxi−b)⋅(−xi)=2(wi=1∑mxi2−i=1∑m(yi−b)xi)(对应公式3.5)
求二阶偏导数 A = f x x ′ ′ ( x , y ) A=f_{x x}^{\prime \prime}(x, y) A=fxx′′(x,y)
∂ 2 E ( w , b ) ∂ w 2 = ∂ ∂ w ( ∂ E ( w , b ) ∂ w ) = ∂ ∂ w [ 2 ( w ∑ i = 1 m x i 2 − ∑ i = 1 m ( y i − b ) x i ) ] = ∂ ∂ w [ 2 w ∑ i = 1 m x i 2 ] = 2 ∑ i = 1 m x i 2 \begin{aligned} \frac{\partial^{2} E_{(w, b)}}{\partial w^{2}} &=\frac{\partial}{\partial w}\left(\frac{\partial E_{(w, b)}}{\partial w}\right) \\ &=\frac{\partial}{\partial w}\left[2\left(w \sum_{i=1}^{m} x_{i}^{2}-\sum_{i=1}^{m}\left(y_{i}-b\right) x_{i}\right)\right] \\ &=\frac{\partial}{\partial w}\left[2 w \sum_{i=1}^{m} x_{i}^{2}\right] \\ &=2 \sum_{i=1}^{m} x_{i}^{2} \quad \end{aligned} ∂w2∂2E(w,b)=∂w∂(∂w∂E(w,b))=∂w∂[2(wi=1∑mxi2−i=1∑m(yi−b)xi)]=∂w∂[2wi=1∑mxi2]=2i=1∑mxi2
求二阶偏导数 B = f x y ′ ′ ( x , y ) B=f_{x y}^{\prime \prime}(x, y) B=fxy′′(x,y)
∂ 2 E ( w , b ) ∂ w ∂ b = ∂ ∂ b ( ∂ E ( w , b ) ∂ w ) = ∂ ∂ b [ 2 ( w ∑ i = 1 m x i 2 − ∑ i = 1 m ( y i − b ) x i ) ] = ∂ ∂ b [ − 2 ∑ i = 1 m ( y i − b ) x i ] = ∂ ∂ b ( − 2 ∑ i = 1 m y i x i + 2 ∑ i = 1 m b x i )