2017/3/20
Regularization
正则化的目的是改善或者减少过拟合overfitting
什么是overfitting?
underfitting /high bias欠拟合,是指 not fitting training set very well
overfitting /high variance 过拟合,是指 too many features,fit the training set very well( J(θ)≈0 ),but fail to generalize to new examples.怎么去解决过拟合的问题
- reduce the num of features (但是可能会丢失一部分信息)
- regularization
penalize parameters
在原来的cost function上面增加一个regularize term
J(θ)=J(θ)+λ2m∑j=1mθ2j,于是我们将这一项分别应用到线性回归的Gradient descent 和Normal Equation 和逻辑回归里面去regularize linear regression
梯度下降
repeat{
θ0:=θ0−α1m∑i=1m(hθ(x(i))−y(i))2x(i)0
θj:=θj−α1m[∑i=1m(hθ(x(i))−y(i))2x(i)j+λθj]=θj(1−αλm)−α1m∑i=1m(hθ(x(i))−y(i))2x(i)j
j=1,2,3,...,n
}Normal Equation
θ=⎛⎝⎜⎜⎜⎜⎜XTX+λ⎡⎣⎢⎢⎢⎢⎢01⋱1⎤⎦⎥⎥⎥⎥⎥⎞⎠⎟⎟⎟⎟⎟−1XTy
regularized logistic regression
- 梯度下降,和线性回归的形式是一样的,但是并不是同一个函数,因为 hθ(x) 并不相同
repeat{
θ0:=θ0−α1m∑i=1m(hθ(x(i))−y(i))2x(i)0
θj:=θj−α1m[∑i=1m(hθ(x(i))−y(i))2x(i)j+λθj]=θj(1−αλm)−α1m∑i=1m(hθ(x(i))−y(i))2x(i)j
j=1,2,3,...,n
}