线性回归和逻辑斯蒂回归正则化

返回目录
当特征量过少时，会出现欠拟合。即训练数据不能进行较好的拟合。
当特征量过多时，会出现过拟合。即训练数据能很好的拟合，但测试数据不能进行较好的拟合。
针对过拟合，需要在代价函数上加上惩罚项，模型复杂度越大，惩罚项越大。

1. 线性回归正则化

代价函数：
$$J(\vec{\theta })=\frac{1}{2m}(\sum _{i=1}^{i=m}(h({\vec{x}}^{(i)})-y^{(i)}{)}^2+\lambda \sum _{j=1}^{j=n}{\theta }_j^2)$$
其中：
$$\begin{array}{r}\vec{y}=[y^{(1)},y^{(2)},...,y^{(m)}{]}^T\in {\mathbb{R}}^{m\times 1}\\ （m为测试样本个数）\end{array}$$
则：
$$\frac{\partial J(\vec{\theta })}{\partial {\theta }_j}=\{\begin{array}{ll}\frac{1}{m}{\sum }_{i=1}^{i=m}(h({\vec{x}}^{(i)})-y^{(i)})& j=0\\ \frac{1}{m}({\sum }_{i=1}^{i=m}(h({\vec{x}}^{(i)})-y^{(i)})x_j^{(i)}+\lambda {\theta }_j)& j=1,2,...,n\end{array}$$
令$\frac{\partial J(\vec{\theta })}{\partial {\theta }_j}=0$，得：
$$\begin{array}{rl}& \left(\begin{array}{cccc}(h({\vec{x}}^{(1)})-y^{(1)})& (h({\vec{x}}^{(2)})-y^{(2)})& ...& (h({\vec{x}}^{(m)})-y^{(m)})\end{array}\right)\left(\begin{array}{c}{x}_j^{(1)}\\ x_j^{(2)}\\ ...\\ x_j^{(m)}\end{array}\right)=-\lambda {\theta }_j\\ & {\left(\begin{array}{c}(h({\vec{x}}^{(1)})-y^{(1)})\\ (h({\vec{x}}^{(2)})-y^{(2)})\\ ...\\ (h({\vec{x}}^{(m)})-y^{(m)})\end{array}\right)}^T\left(\begin{array}{c}{x}_j^{(1)}\\ x_j^{(2)}\\ ...\\ x_j^{(m)}\end{array}\right)=-\lambda {\theta }_j\\ & {\left(\begin{array}{c}(h({\vec{x}}^{(1)})-y^{(1)})\\ (h({\vec{x}}^{(2)})-y^{(2)})\\ ...\\ (h({\vec{x}}^{(m)})-y^{(m)})\end{array}\right)}^T\left(\begin{array}{cccc}{x}_0^{(1)}& x_1^{(1)}& ...& x_n^{(1)}\\ x_0^{(2)}& x_1^{(2)}& ...& x_n^{(2)}\\ ...& ...& ...& ...\\ x_0^{(m)}& x_1^{(m)}& ...& x_n^{(m)}\end{array}\right)=-\lambda \left(\begin{array}{ccccc}0& {\theta }_1& {\theta }_2& ...& {\theta }_n\end{array}\right)\\ & {\left(\begin{array}{c}{{\vec{x}}^{(1)}}^T\vec{\theta }-y^{(1)})\\ {{\vec{x}}^{(2)}}^T\vec{\theta }-y^{(2)})\\ ...\\ {{\vec{x}}^{(m)}}^T\vec{\theta }-y^{(m)})\end{array}\right)}^T\left(\begin{array}{cccc}{x}_0^{(1)}& x_1^{(1)}& ...& x_n^{(1)}\\ x_0^{(2)}& x_1^{(2)}& ...& x_n^{(2)}\\ ...& ...& ...& ...\\ x_0^{(m)}& x_1^{(m)}& ...& x_n^{(m)}\end{array}\right)=-\lambda \left(\begin{array}{ccccc}{\theta }_0& {\theta }_1& {\theta }_2& ...& {\theta }_n\end{array}\right)\left(\begin{array}{cccc}0& & & \\ & 1& & \\ & & \ddots & \\ & & & 1\end{array}\right)\end{array}$$
所以公式化求解得到：
$$\begin{array}{r}(X\vec{\theta }-\vec{y}{)}^{T}X=-\lambda {\vec{\theta }}^T\left(\begin{array}{cccc}0& & & \\ & 1& & \\ & & \ddots & \\ & & & 1\end{array}\right)\\ \vec{\theta }=(X^{T}X+\lambda \left(\begin{array}{cccc}0& & & \\ & 1& & \\ & & \ddots & \\ & & & 1\end{array}\right){)}^{-1}{X}^T\vec{y}\end{array}$$
梯度下降法得到：
$$\{\begin{array}{ll}{\theta }_0:={\theta }_0-\alpha {\sum }_{i=1}^{i=m}({\vec{\theta }}^T{\vec{x}}^{(i)}-y^{(i)})& j=0\\ {\theta }_j:=(1-\alpha \frac{\lambda }{m}){\theta }_j-\frac{\alpha }{m}{\sum }_{i=1}^{i=m}({\vec{\theta }}^T{\vec{x}}^{(i)}-y^{(i)})x_j^{(i)}& j=1,2,...,n\end{array}$$

2. 逻辑斯蒂回归正则化

代价函数：
$$J(\vec{\theta })=-\frac{1}{m}(\sum _{i=1}^{i=m}{y}^{(i)}ln(h({\vec{x}}^{(i)}))+(1-y^{(i)})ln(1-h({\vec{x}}^{(i)})))+\frac{\lambda }{2m}\sum _{j=1}^{j=n}{\theta }_j^2$$
梯度下降法得到：
$$\{\begin{array}{ll}{\theta }_0:={\theta }_0-\alpha {\sum }_{i=1}^{i=m}({\vec{\theta }}^T{\vec{x}}^{(i)}-y^{(i)})& \\ {\theta }_j:=(1-\alpha \frac{\lambda }{m}){\theta }_j-\frac{\alpha }{m}{\sum }_{i=1}^{i=m}({\vec{\theta }}^T{\vec{x}}^{(i)}-y^{(i)})x_j^{(i)}& j=1,2,...,n\end{array}$$

3. Lasso回归

即在代价函数后面添加L1正则化项。
$$\widehat{J(\vec{\theta })}=J(\vec{\theta })+\alpha \sum _{j=1}^{j=n}|{\theta }_j|$$

4. 领回归

即在代价函数后面添加L2正则化项。1,2小节介绍的线性回归和Logistic回归正则化就是领回归。
$$\widehat{J(\vec{\theta })}=J(\vec{\theta })+\alpha \sum _{j=1}^{j=n}{\theta }_j^2$$

5. ElasticNet回归

即在代价函数后面添加L1+L2正则化项。
$$\widehat{J(\vec{\theta })}=J(\vec{\theta })+\sum _{j=1}^{j=n}(\alpha \rho |{\theta }_j|+\frac{\alpha (1-\rho )}{2}{\theta }_j^2)$$
返回目录