《吴恩达机器学习》笔记——8 正则化

《吴恩达机器学习》笔记——8 正则化

1 过拟合问题

解决过拟合问题	具体
减少选取变量的数量	人为选择保留的特征 模型选择算法
正则化	保留所有特征，但减小${\theta }_j$的数量级或值

2 代价函数

正则化代价函数	$J(\theta )=\frac{1}{2m}\left[\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)}{)}^2+\lambda \sum _{j=1}^n{\theta }_j^2\right]$
正则化参数	$\lambda \{\begin{array}{c}过大\to 欠拟合\\ 过小\to 过拟合\end{array}$

3 线性回归的正则化

-	梯度下降
$J(\theta )$	$\frac{1}{2m}\left[\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)}{)}^2+\lambda \sum _{j=1}^n{\theta }_j^2\right]$
$\underset{\theta }{\min }J(\theta )$	重 复 { θ 0 : = θ 0 − α 1 m ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) x 0 ( i ) θ j : = θ j − α 1 m [ ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) x j ( i ) + λ θ j ] } ( j = 1 , … , n ) 重复\{\\\theta_0:=\theta_0-\alpha \frac{1}{m}\sum\limits^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_0\\\theta_j:=\theta_j-\alpha\frac{1}{m}\left[\sum\limits^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_j+\lambda\theta_j\right]\\\}\qquad (j=1,\dots,n) 重复{θ0​:=θ0​−αm1​i=1∑m​(hθ​(x(i))−y(i))x0(i)​θj​:=θj​−αm1​[i=1∑m​(hθ​(x(i))−y(i))xj(i)​+λθj​]}(j=1,…,n)
${\theta }_j$	${\theta }_j:={\theta }_j(1-\alpha \frac{\lambda }{m})-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}$
-	正规方程
$X$	${\left[\begin{array}{c}(x^{(1)}{)}^{\mathrm{T}}\\ ⋮\\ (x^{(m)}{)}^{\mathrm{T}}\end{array}\right]}_{m\times (n+1)}$
$y$	$\left[\begin{array}{c}{y}^{(1)}\\ ⋮\\ y^{(m)}\end{array}\right]\in {\mathbb{R}}^m$
$\theta$	${\left(X^{\mathrm{T}}X+\lambda {\left[\begin{array}{ccccc}0& & & & \\ & 1& & & \\ & & 1& & \\ & & & \ddots & \\ & & & & 1\end{array}\right]}_{(n+1)\times (n+1)}\right)}^{-1}{X}^{\mathrm{T}}y$

4 Logistic回归的正则化

$J(\theta )$	$-\left[\frac{1}{m}\sum _{i=1}^m{y}^{(i)}\log h_{\theta }(x^{(i)})+(1-y^{(i)})\log (1-h_{\theta }(x^{(i)}))\right]+\frac{\lambda }{2m}\sum _{j=1}^n{\theta }_j^2$
$\underset{\theta }{\min }J(\theta )$	重 复 { θ 0 : = θ 0 − α 1 m ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) x 0 ( i ) θ j : = θ j − α 1 m [ ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) x j ( i ) + λ θ j ] } ( j = 1 , … , n ) 重复\{\\\theta_0:=\theta_0-\alpha \frac{1}{m}\sum\limits^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_0\\\theta_j:=\theta_j-\alpha\frac{1}{m}\left[\sum\limits^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_j+\lambda\theta_j\right]\\\}\qquad (j=1,\dots,n) 重复{θ0​:=θ0​−αm1​i=1∑m​(hθ​(x(i))−y(i))x0(i)​θj​:=θj​−αm1​[i=1∑m​(hθ​(x(i))−y(i))xj(i)​+λθj​]}(j=1,…,n)
${\theta }_j$	${\theta }_j:={\theta }_j(1-\alpha \frac{\lambda }{m})-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}$