《吴恩达机器学习》笔记——8 正则化
1 过拟合问题
解决过拟合问题 | 具体 |
---|---|
减少选取变量的数量 | 人为选择保留的特征 模型选择算法 |
正则化 | 保留所有特征,但减小 θ j \theta_j θj的数量级或值 |
2 代价函数
正则化代价函数 | J ( θ ) = 1 2 m [ ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) 2 + λ ∑ j = 1 n θ j 2 ] J(\theta)=\frac{1}{2m}\left[\sum\limits^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})^2+\lambda\sum\limits^n_{j=1}\theta^2_j\right] J(θ)=2m1[i=1∑m(hθ(x(i))−y(i))2+λj=1∑nθj2] |
---|---|
正则化参数 | λ { 过 大 → 欠 拟 合 过 小 → 过 拟 合 \lambda\left\{\begin{matrix}过大\rightarrow欠拟合\\过小\rightarrow过拟合\end{matrix}\right. λ{过大→欠拟合过小→过拟合 |
3 线性回归的正则化
- | 梯度下降 |
---|---|
J ( θ ) J(\theta) J(θ) | 1 2 m [ ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) 2 + λ ∑ j = 1 n θ j 2 ] \frac{1}{2m}\left[\sum\limits^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})^2+\lambda\sum\limits^n_{j=1}\theta^2_j\right] 2m1[i=1∑m(hθ(x(i))−y(i))2+λj=1∑nθj2] |
min θ J ( θ ) \min\limits_\theta J(\theta) θminJ(θ) | 重 复 { θ 0 : = θ 0 − α 1 m ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) x 0 ( i ) θ j : = θ j − α 1 m [ ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) x j ( i ) + λ θ j ] } ( j = 1 , … , n ) 重复\{\\\theta_0:=\theta_0-\alpha \frac{1}{m}\sum\limits^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_0\\\theta_j:=\theta_j-\alpha\frac{1}{m}\left[\sum\limits^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_j+\lambda\theta_j\right]\\\}\qquad (j=1,\dots,n) 重复{θ0:=θ0−αm1i=1∑m(hθ(x(i))−y(i))x0(i)θj:=θj−αm1[i=1∑m(hθ(x(i))−y(i))xj(i)+λθj]}(j=1,…,n) |
θ j \theta_j θj | θ j : = θ j ( 1 − α λ m ) − α 1 m ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) x j ( i ) \theta_j:=\theta_j(1-\alpha\frac{\lambda}{m})-\alpha\frac{1}{m}\sum\limits^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_j θj:=θj(1−αmλ)−αm1i=1∑m(hθ(x(i))−y(i))xj(i) |
- | 正规方程 |
X X X | [ ( x ( 1 ) ) T ⋮ ( x ( m ) ) T ] m × ( n + 1 ) \left[\begin{matrix}(x^{(1)})^\mathrm{T}\\\vdots\\(x^{(m)})^\mathrm{T}\end{matrix}\right]_{m\times(n+1)} ⎣⎢⎡(x(1))T⋮(x(m))T⎦⎥⎤m×(n+1) |
y y y | [ y ( 1 ) ⋮ y ( m ) ] ∈ R m \left[\begin{matrix}y^{(1)}\\\vdots\\y^{(m)}\end{matrix}\right]\in\mathbb{R}^m ⎣⎢⎡y(1)⋮y(m)⎦⎥⎤∈Rm |
θ \theta θ | ( X T X + λ [ 0 1 1 ⋱ 1 ] ( n + 1 ) × ( n + 1 ) ) − 1 X T y \left(X^\mathrm{T}X+\lambda\left[\begin{matrix}0&&&&\\&1&&&\\&&1&&\\&&&\ddots&\\&&&&1\end{matrix}\right]_{(n+1)\times(n+1)}\right)^{-1}X^\mathrm{T}y ⎝⎜⎜⎜⎜⎛XTX+λ⎣⎢⎢⎢⎢⎡011⋱1⎦⎥⎥⎥⎥⎤(n+1)×(n+1)⎠⎟⎟⎟⎟⎞−1XTy |
4 Logistic回归的正则化
J ( θ ) J(\theta) J(θ) | − [ 1 m ∑ i = 1 m y ( i ) log h θ ( x ( i ) ) + ( 1 − y ( i ) ) log ( 1 − h θ ( x ( i ) ) ) ] + λ 2 m ∑ j = 1 n θ j 2 -\left[\frac{1}{m}\sum\limits^m_{i=1}y^{(i)}\log h_\theta(x^{(i)})+(1-y^{(i)})\log(1-h_\theta(x^{(i)}))\right]+\frac{\lambda}{2m}\sum\limits^n_{j=1}\theta_j^2 −[m1i=1∑my(i)loghθ(x(i))+(1−y(i))log(1−hθ(x(i)))]+2mλj=1∑nθj2 |
---|---|
min θ J ( θ ) \min\limits_\theta J(\theta) θminJ(θ) | 重 复 { θ 0 : = θ 0 − α 1 m ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) x 0 ( i ) θ j : = θ j − α 1 m [ ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) x j ( i ) + λ θ j ] } ( j = 1 , … , n ) 重复\{\\\theta_0:=\theta_0-\alpha \frac{1}{m}\sum\limits^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_0\\\theta_j:=\theta_j-\alpha\frac{1}{m}\left[\sum\limits^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_j+\lambda\theta_j\right]\\\}\qquad (j=1,\dots,n) 重复{θ0:=θ0−αm1i=1∑m(hθ(x(i))−y(i))x0(i)θj:=θj−αm1[i=1∑m(hθ(x(i))−y(i))xj(i)+λθj]}(j=1,…,n) |
θ j \theta_j θj | θ j : = θ j ( 1 − α λ m ) − α 1 m ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) x j ( i ) \theta_j:=\theta_j(1-\alpha\frac{\lambda}{m})-\alpha\frac{1}{m}\sum\limits^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_j θj:=θj(1−αmλ)−αm1i=1∑m(hθ(x(i))−y(i))xj(i) |