机器学习 2014斯坦福大学课程: 3 正则化问题
刚开始学习机器学习,学习的视频教程是coursera网站上吴恩达(Andrew Ng)教授的机器学习课程。
在此梳理并记录总结笔记,供学习交流,欢迎批评指正!
机器学习 分类 回顾
- 监督学习(supervised learning )
1.1 回归问题(regression):线性回归问题(linear regression)
1.2 分类问题(classification):逻辑回归问题(logistic regression)
- 无监督学习(unsupervised learning)
2.1 聚集问题(clustering)
2.2. 非聚集问题(non-clustering)
正则化(Regularization)
正则化用于优化线性回归和逻辑回归问题,解决过拟合问题。
过拟合(overfitting)问题
从左至右分别为:欠拟合(underfitting),拟合,过拟合
过拟合:当特征太多时,将样本集拟合的特别好,但是预测时难以一般化。
解决方法:
减少特征数,手动筛选保留一些特征
保留特征数,引入正则化,将参数的权重降低
成本函数正则化
-
线性回归
特征: x 1 , x 2 , x 3 , x 4 . . . x n x_1, x_2,x_3,x_4...x_n x1,x2,x3,x4...xn
参数: θ 0 , θ 1 , θ 2 . . . θ n \theta_0,\theta_1,\theta_2...\theta_n θ0,θ1,θ2...θn
引入: λ \lambda λ
J ( θ 0 , θ 1 , θ 2 . . . θ n ) = 1 2 m [ ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) 2 + λ ∑ j = 1 n θ j 2 ] J(\theta_0,\theta_1,\theta_2...\theta_n)=\frac{1}{2m}[\sum_{i=1}^m (h_\theta(x^{(i)})-y^{(i)})^2+\lambda\sum_{j=1}^n\theta_j^2] J(θ0,θ1,θ2...θn)=2m1[i=1∑m(hθ(x(i))−y(i))2+λj=1∑nθj2]
梯度下降算法为:
r e p e a t { repeat\{ repeat{ θ 0 = θ 0 − α 1 m ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) ∗ x 0 ( i ) \theta_0= \theta_0 - \alpha\frac{1}{m}\sum_{i=1}^m (h_\theta(x^{(i)})-y^{(i)})*x_0^{(i)} θ0=θ0−αm1i=1∑m(hθ(x(i))−y(i))∗x0(i) θ j = θ j − α [ 1 m ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) ∗ x j ( i ) + λ m θ j ] \theta_j = \theta_j - \alpha[\frac{1}{m}\sum_{i=1}^m (h_\theta(x^{(i)})-y^{(i)})*x_j^{(i)}+\frac{\lambda}{m}\theta_j] θj=θj−α[m1i=1∑m(hθ(x(i))−y(i))∗xj(i)+mλθj] j = 1 , 2 , 3 , . . . n j=1,2,3,...n j=1,2,3,...n } \} }j = 1... n j=1...n j=1...n 注意 θ 0 \theta_0 θ0不引入参数
若 λ = 1000 \lambda=1000 λ=1000
J ( θ 0 , θ 1 , θ 2 , θ 3 , θ n ) = 1 2 m ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) 2 + J(\theta_0,\theta_1,\theta_2,\theta_3,\theta_n)=\frac{1}{2m}\sum_{i=1}^m (h_\theta(x^{(i)})-y^{(i)})^2+ J(θ0,θ1,θ2,θ3,θn)=2m1∑i=1m(hθ(x(i))−y(i))2+ 1000 θ 1 2 + 1000 θ 2 2 . . . 1000\theta_1^2+1000\theta_2^2... 1000θ12+1000θ22...
当 m i n J − > 0 minJ ->0 minJ−>0 1000 越大 θ 越 小 \theta越小 θ越小 θ \theta θ趋近于0
但是当 λ \lambda λ太大,所有参数 θ \theta θ都趋近0
那么将会变为一条 y = θ 0 y=\theta_0 y=θ0直线,出现欠拟合情况。 -
逻辑回归
成本函数:
J ( θ 0 , θ 1 ) = 1 m ∑ i = 1 m ( − y ( i ) ∗ l o g ( h θ ( x ( i ) ) ) + ( 1 − y ( i ) ) l o g ( 1 − h θ ( x ( i ) ) ) ) + λ 2 m ∑ j = 1 n θ j 2 J(\theta_0,\theta_1)=\frac{1}{m}\sum_{i=1}^m (-y^{(i)}*log(h_\theta(x^{(i)}))+(1-y^{(i))}log(1-h_\theta(x^{(i)}))) +\frac{\lambda}{2m}\sum_{j=1}^n\theta_j^2 J(θ0,θ1)=m1i=1∑m(−y(i)∗log(hθ(x(i)))+(1−y(i))log(1−hθ(x(i))))+2mλj=1∑nθj2
J ( θ ) = 1 m ∑ i = 1 m ( − y . ∗ l o g ( h ( θ ) ) + ( 1 − y ) . ∗ l o g ( 1 − h ( θ ) ) ) J(\theta)=\frac{1}{m}\sum_{i=1}^m(-y.*log(h(\theta))+(1-y).*log(1-h(\theta))) J(θ)=m1i=1∑m(−y.∗log(h(θ))+(1−y).∗log(1−h(θ)))梯度下降算法为:
r e p e a t { repeat\{ repeat{ θ 0 = θ 0 − α 1 m ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) ∗ x 0 ( i ) \theta_0= \theta_0 - \alpha\frac{1}{m}\sum_{i=1}^m (h_\theta(x^{(i)})-y^{(i)})*x_0^{(i)} θ0=θ0−αm1i=1∑m(hθ(x(i))−y(i))∗x0(i) θ j = θ j − α [ 1 m ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) ∗ x j ( i ) + λ m θ j ] \theta_j = \theta_j - \alpha[\frac{1}{m}\sum_{i=1}^m (h_\theta(x^{(i)})-y^{(i)})*x_j^{(i)}+\frac{\lambda}{m}\theta_j] θj=θj−α[m1i=1∑m(hθ(x(i))−y(i))∗xj(i)+mλθj] j = 1 , 2 , 3 , . . . n j=1,2,3,...n j=1,2,3,...n } \} } -
正则方程
θ = ( X T X + λ [ 0 1 ⋱ 1 1 ] ) − 1 X T y \theta=(X^TX+\lambda\begin{bmatrix} 0 & && & \\ & 1 && & \\ & &\ddots&&\\ & & &1 & \\ & & &&1 \\ \end{bmatrix})^{-1}X^Ty θ=(XTX+λ⎣⎢⎢⎢⎢⎡01⋱11⎦⎥⎥⎥⎥⎤)−1XTy
证明过程略
程序实现
python代码实现
线性回归成本函数和梯度下降函数
import numpy as np
def computeCost(X,y,theta,l):#成本函数
theta1=npconcatenate([np.array([0],theta[1:]))#将theta0换成0,不参与正则化
m=X.shape[0]
inner=X@theta-y #X@theta=X.dot(theta)
return inner.T@inner/(2*m)+l/(2*m)*(theta1.T@theta1)
def gradientDescent(X,y,theta,alpha,interation):#梯度下降
theta1=npconcatenate([np.array([0],theta[1:]))#将theta0换成0
m=X.shape()
J=[]
for _ in range(interation):
theta=theta-alpha/m*X.T@(X@theta-y)-alpha*l/m*theta1
J.append(computeCost(X,y,theta))
return theta,J
逻辑回归成本函数和偏导数函数
import numpy as np
def sigmoid(z):
return 1/(1+np.exp(-z))
def computeCost(X,y,theta,l):#成本函数
theta1=npconcatenate([np.array([0],theta[1:]))#将theta0换成0
m=X.shape[0]
h=sigmoid(X@theta)
J=np.mean(-y*log(h)+(1-y)*log(h))+l/(2*m)*(theta1.T@theta1)
return J
def gradient(X,y,theta,l):#导数
theta1=npconcatenate([np.array([0],theta[1:]))
m=X.shape[0]
h=sigmoid(X@theta)
grad=1/m*(X.T@(h-y))+l/m*theta1
return grad
总结
增加了 λ \lambda λ一项,注意 θ 0 \theta_0 θ0不考虑,从 θ 1 \theta_1 θ1开始
参考
[1] 来自本课程课件和作业题,以及黄海广github上的中文笔记资料等