线性最小二乘回归和核岭回归

最新推荐文章于 2024-08-05 20:22:54 发布

Maxwellhang

最新推荐文章于 2024-08-05 20:22:54 发布

阅读量5.4k

点赞数 2

分类专栏：机器学习文章标签：回归最小二乘核方法岭回归

本文链接：https://blog.csdn.net/u011529752/article/details/54025172

版权

机器学习专栏收录该内容

13 篇文章 0 订阅

订阅专栏

一般线性回归最小二乘法推导

原理,求平方误差最小
假设有k个观测值 $X_i$ , $Y_i$ ,有关系 $Y=f(X|\theta)$ ,其中 $\theta=(\theta_1,\theta_2,...,\theta_l)$
平方误差和为:

Λ = 1 2 \sum i = 1 k [f (X i | θ) - Y i] 2

$\begin{eqnarray}\Lambda&=&\dfrac{1}{2}\sum_{i=1}^{k}[f(X_i|\theta)-Y_i]^2\end{eqnarray}$
要求误差平方和关于

θ $\theta$ 的函数在极值点前后接近单调

\partial Λ \partial θ j = \sum i = 1 k [f (X i | θ) - Y i] \partial f ( X i | θ ) \partial θ j

$\begin{eqnarray}\dfrac{\partial{\Lambda}}{\partial{\theta_j}}&=& \sum_{i=1}^{k}[f(X_i|\theta)-Y_i]\dfrac{\partial{f(X_i|\theta)}}{\partial{\theta_j}} \end{eqnarray}$
一般情况各参数之间是线性关系,以二项式为例,

f (X i | θ) = = θ 1 + θ 2 X i + θ 3 X 2 i + . . . + θ l X l - 1 i \sum j = 1 l θ j X j - 1 i

$\begin{eqnarray}f(X_i|\theta)&=&\theta_1+\theta_2X_i+\theta_3X_i^2+...+\theta_lX_i^{l-1}\\ &=&\sum_{j=1}^{l}\theta_jX_i^{j-1} \end{eqnarray}$
带入上式,得

\partial Λ \partial θ j = \sum i = 1 k [\sum j = 1 l θ j X j - 1 i - Y i] X j - 1 i

$\begin{eqnarray}\dfrac{\partial{\Lambda}}{\partial{\theta_j}}&=& \sum_{i=1}^{k}[\sum_{j=1}^{l}\theta_jX_i^{j-1}-Y_i]X_i^{j-1} \end{eqnarray}$
显然多项式关于参数的函数是开口向上的二次函数,是满足极值条件的,令导数为零,则零点即为最小值点
多参数联立,有:

\partial Λ \partial θ 1 \partial Λ \partial θ 2 \partial Λ \partial θ l = = ⋮ = \sum i = 1 k [\sum j = 1 l θ j X j - 1 i - Y i] X 0 i \sum i = 1 k [\sum j = 1 l θ j X j - 1 i - Y i] X 1 i \sum i = 1 k [\sum j = 1 l θ j X j - 1 i - Y i] X l - 1 i = = ⋮ = 000

$\begin{eqnarray}\dfrac{\partial{\Lambda}}{\partial{\theta_1}}&=& \sum_{i=1}^{k}[\sum_{j=1}^{l}\theta_jX_i^{j-1}-Y_i]X_i^{0}&=&0\\ \dfrac{\partial{\Lambda}}{\partial{\theta_2}}&=& \sum_{i=1}^{k}[\sum_{j=1}^{l}\theta_jX_i^{j-1}-Y_i]X_i^{1}&=&0\\ &\vdots&&\vdots&\\ \dfrac{\partial{\Lambda}}{\partial{\theta_l}}&=& \sum_{i=1}^{k}[\sum_{j=1}^{l}\theta_jX_i^{j-1}-Y_i]X_i^{l-1}&=&0\\ \end{eqnarray}$
整理得:

θ 1 \sum i = 1 k X 0 i θ 1 \sum i = 1 k X 1 i θ 1 \sum i = 1 k X l - 1 i + + ⋮ + θ 2 \sum i = 1 k X 1 i θ 2 \sum i = 1 k X 2 i θ 2 \sum i = 1 k X l i + + ⋮ + . . . . . . . . . + + ⋮ + θ l \sum i = 1 k X l - 1 i θ l \sum i = 1 k X l i θ l \sum i = 1 k X 2 l - 2 i - - ⋮ - \sum i = 1 k Y i X 0 i \sum i = 1 k Y i X 1 i \sum i = 1 k Y i X l - 1 i = = ⋮ = 000

$\begin{eqnarray} \theta_1\sum_{i=1}^{k}X_i^{0}&+&\theta_2\sum_{i=1}^{k}X_i^{1}&+&...&+&\theta_l\sum_{i=1}^{k}X_i^{l-1}&-&\sum_{i=1}^{k}Y_iX_i^0&=&0\\ \theta_1\sum_{i=1}^{k}X_i^{1}&+&\theta_2\sum_{i=1}^{k}X_i^{2}&+&...&+&\theta_l\sum_{i=1}^{k}X_i^{l}&-&\sum_{i=1}^{k}Y_iX_i^1&=&0\\ &\vdots&&\vdots&&\vdots&&\vdots&&\vdots&\\ \theta_1\sum_{i=1}^{k}X_i^{l-1}&+&\theta_2\sum_{i=1}^{k}X_i^{l}&+&...&+&\theta_l\sum_{i=1}^{k}X_i^{2l-2}&-&\sum_{i=1}^{k}Y_iX_i^{l-1}&=&0\\ \end{eqnarray}$
写成矩阵形式:

⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ \sum i = 1 k X 0 i \sum i = 1 k X 1 i ⋮ \sum i = 1 k X l - 1 i \sum i = 1 k X 1 i \sum i = 1 k X 2 i ⋮ \sum i = 1 k X l i \dots \dots ⋱ \dots \sum i = 1 k X l - 1 i \sum i = 1 k X l i ⋮ \sum i = 1 k X 2 l - 2 i ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ θ 1 θ 2 ⋮ θ l ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ = ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ \sum i = 1 k Y i X 0 i \sum i = 1 k Y i X 1 i ⋮ \sum i = 1 k Y i X l - 1 i ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥

$\left[\begin{matrix} \sum_{i=1}^{k}X_i^{0}&\sum_{i=1}^{k}X_i^{1} &\dots&\sum_{i=1}^{k}X_i^{l-1}\\ \sum_{i=1}^{k}X_i^{1}&\sum_{i=1}^{k}X_i^{2} &\dots&\sum_{i=1}^{k}X_i^{l}\\ \vdots&\vdots&\ddots&\vdots\\ \sum_{i=1}^{k}X_i^{l-1}&\sum_{i=1}^{k}X_i^{l} &\dots&\sum_{i=1}^{k}X_i^{2l-2}\\ \end{matrix}\right]\left[\begin{matrix} \theta_1\\\theta_2\\\vdots\\\theta_l\\ \end{matrix}\right]=\left[\begin{matrix} \sum_{i=1}^{k}Y_iX_i^{0}\\\sum_{i=1}^{k}Y_iX_i^{1}\\\vdots\\\sum_{i=1}^{k}Y_iX_i^{l-1}\\ \end{matrix}\right]$
可以得:

\sum i = 1 k ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ X 0 i X 1 i ⋮ X l - 1 i X 1 i X 2 i ⋮ X l i \dots \dots ⋱ \dots X l - 1 i X l i ⋮ X 2 l - 2 i ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ θ 1 θ 2 ⋮ θ l ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ = \sum i = 1 k ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ Y i X 0 i Y i X 1 i ⋮ Y i X l - 1 i ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥

$\sum_{i=1}^{k}\left[\begin{matrix} X_i^{0}&X_i^{1} &\dots&X_i^{l-1}\\ X_i^{1}&X_i^{2} &\dots&X_i^{l}\\ \vdots&\vdots&\ddots&\vdots\\ X_i^{l-1}&X_i^{l} &\dots&X_i^{2l-2}\\ \end{matrix}\right]\left[\begin{matrix} \theta_1\\\theta_2\\\vdots\\\theta_l\\ \end{matrix}\right]=\sum_{i=1}^{k}\left[\begin{matrix} Y_iX_i^{0}\\Y_iX_i^{1}\\\vdots\\Y_iX_i^{l-1}\\ \end{matrix}\right]$
假设k=1,即只有一组观测值有:

⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ X 0 i X 1 i ⋮ X l - 1 i X 1 i X 2 i ⋮ X l i \dots \dots ⋱ \dots X l - 1 i X l i ⋮ X 2 l - 2 i ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ θ 1 θ 2 ⋮ θ l ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ = ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ Y i X 0 i Y i X 1 i ⋮ Y i X l - 1 i ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ = = > [X 0 i X 1 i \dots X l - 1 i] ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ θ 1 θ 2 ⋮ θ l ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ = [Y i]

$\left[\begin{matrix} X_i^{0}&X_i^{1} &\dots&X_i^{l-1}\\ X_i^{1}&X_i^{2} &\dots&X_i^{l}\\ \vdots&\vdots&\ddots&\vdots\\ X_i^{l-1}&X_i^{l} &\dots&X_i^{2l-2}\\ \end{matrix}\right]\left[\begin{matrix} \theta_1\\\theta_2\\\vdots\\\theta_l\\ \end{matrix}\right]=\left[\begin{matrix} Y_iX_i^{0}\\Y_iX_i^{1}\\\vdots\\Y_iX_i^{l-1}\\ \end{matrix}\right]\\ ==>\\\left[\begin{matrix} X_i^{0}&X_i^{1} &\dots&X_i^{l-1}\\ \end{matrix}\right]\left[\begin{matrix} \theta_1\\\theta_2\\\vdots\\\theta_l\\ \end{matrix}\right]=\left[\begin{matrix} Y_i\\ \end{matrix}\right]$ 对所有的观测值都成立,由每一组观测值得出的方程,有

⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ X 01 X 02 ⋮ X 0 k X 11 X 12 ⋮ X 1 k \dots \dots ⋱ \dots X l - 1 1 X l - 1 2 ⋮ X l - 1 k ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ θ 1 θ 2 ⋮ θ l ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ = ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ Y 1 Y 2 ⋮ Y k ⎤ ⎦ ⎥ ⎥ ⎥ ⎥

$\left[\begin{matrix} X_1^{0}&X_1^{1} &\dots&X_1^{l-1}\\ X_2^{0}&X_2^{1} &\dots&X_2^{l-1}\\ \vdots&\vdots&\ddots&\vdots\\ X_k^{0}&X_k^{1} &\dots&X_k^{l-1}\\ \end{matrix}\right]\left[\begin{matrix} \theta_1\\\theta_2\\\vdots\\\theta_l\\ \end{matrix}\right]=\left[\begin{matrix} Y_1\\Y_2\\\vdots\\Y_k\\ \end{matrix}\right]$
即:

X k \times l θ l \times 1 = Y k \times 1 X T l \times k X k \times l θ l \times 1 = X T l \times k Y k \times 1 θ l \times 1 = (X T l \times k X k \times l) - 1 X T l \times k Y k \times 1

$X_{k\times l}\theta_{l\times 1}=Y_{k\times 1}\\ X^T_{l\times k}X_{k\times l}\theta_{l\times 1}=X^T_{l\times k}Y_{k\times 1}\\ \theta_{l\times 1}=(X^T_{l\times k}X_{k\times l})^{-1}X^T_{l\times k}Y_{k\times 1}$

假设 $X$ 和 $Y$ 分别为m和n维的向量(一般 $m>=n$ ),模型采用一阶线性回归,则可直接由上面的方法得到:

⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ X 11 X 12 ⋮ X 1 k X 21 X 22 ⋮ X 2 k \dots \dots ⋱ \dots X m 1 X m 2 ⋮ X m k ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ θ 11 θ 12 ⋮ θ 1 m θ 21 θ 22 ⋮ θ 2 m \dots \dots ⋱ \dots θ n 1 θ n 2 ⋮ θ n m ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ = ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ Y 11 Y 12 ⋮ Y 1 k Y 21 Y 22 ⋮ Y 2 k \dots \dots ⋱ \dots Y n 1 Y n 2 ⋮ Y n k ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥

$\left[\begin{matrix} X_1^{1}&X_1^{2} &\dots&X_1^{m}\\ X_2^{1}&X_2^{2} &\dots&X_2^{m}\\ \vdots&\vdots&\ddots&\vdots\\ X_k^{1}&X_k^{2} &\dots&X_k^{m}\\ \end{matrix}\right]\left[\begin{matrix} \theta_1^1&\theta_1^2&\ldots&\theta_1^n\\ \theta_2^1&\theta_2^2&\ldots&\theta_2^n\\ \vdots&\vdots&\ddots&\vdots\\ \theta_m^1&\theta_m^2&\ldots&\theta_m^n\\ \end{matrix}\right]=\left[\begin{matrix} Y_1^{1}&Y_1^{2} &\dots&Y_1^{n}\\ Y_2^{1}&Y_2^{2} &\dots&Y_2^{n}\\\vdots&\vdots&\ddots&\vdots\\Y_k^{1}&Y_k^{2} &\dots&Y_k^{n}\\ \end{matrix}\right]$
即:

X k \times m θ m \times n = Y k \times n X T m \times k X k \times m θ m \times n = X T m \times k Y k \times n θ m \times n = (X T m \times k X k \times m) - 1 X T m \times k Y k \times n

$X_{k\times m}\theta_{m\times n}=Y_{k\times n}\\ X^T_{m\times k}X_{k\times m}\theta_{m\times n}=X^T_{m\times k}Y_{k\times n}\\ \theta_{m\times n}=(X^T_{m\times k}X_{k\times m})^{-1}X^T_{m\times k}Y_{k\times n}$
对偶表达

θ m \times n = X T m \times k X k \times m (X T m \times k X k \times m) - 2 X T m \times k Y k \times n = X T m \times k α k \times n

$\theta_{m\times n}=X^T_{m\times k}X_{k\times m}(X^T_{m\times k}X_{k\times m})^{-2}X^T_{m\times k}Y_{k\times n}\\=X^T_{m\times k}\alpha_{k\times n}$

岭回归和核技巧

在上面的基础上,当 $k<m$ 的时候,即样本数不足的情况下,在求逆的时候会发生问题,此时在求逆时减去一个 $\lambda I$ 来使得求逆可行
得

θ m \times n = (X T m \times k X k \times m - λ I m \times m) - 1 X T m \times k Y k \times n (X T m \times k X k \times m - λ I m \times m) θ m \times n = X T m \times k Y k \times n X T m \times k X k \times m θ m \times n - λ I m \times m θ m \times n = X T m \times k Y k \times n λ θ m \times n = X T m \times k X k \times m θ m \times n - X T m \times k Y k \times n θ m \times n = λ - 1 X T m \times k (X k \times m θ m \times n - Y k \times n) = X T m \times k α k \times n

$\theta_{m\times n}=(X^T_{m\times k}X_{k\times m}-\lambda I_{m\times m})^{-1}X^T_{m\times k}Y_{k\times n}\\ (X^T_{m\times k}X_{k\times m}-\lambda I_{m\times m})\theta_{m\times n}=X^T_{m\times k}Y_{k\times n}\\ X^T_{m\times k}X_{k\times m}\theta_{m\times n}-\lambda I_{m\times m}\theta_{m\times n}=X^T_{m\times k}Y_{k\times n}\\ \lambda\theta_{m\times n}=X^T_{m\times k}X_{k\times m}\theta_{m\times n}-X^T_{m\times k}Y_{k\times n}\\ \theta_{m\times n}=\lambda^{-1}X^T_{m\times k}(X_{k\times m}\theta_{m\times n}-Y_{k\times n})=X^T_{m\times k}\alpha_{k\times n}$
得

α k \times n = λ - 1 (X k \times m θ m \times n - Y k \times n) = = > λ α k \times n = X k \times m X T m \times k α k \times n - Y k \times n = = > (X k \times m X T m \times k - λ I k \times k) α k \times n = Y k \times n = = > α k \times n = (X k \times m X T m \times k - λ I k \times k) - 1 Y k \times n

$\alpha_{k\times n}=\lambda^{-1}(X_{k\times m}\theta_{m\times n}-Y_{k\times n})\\ ==>\lambda\alpha_{k\times n}=X_{k\times m}X^T_{m\times k}\alpha_{k\times n}-Y_{k\times n} ^{} \\= => (X_{k\times m}X^T_{m\times k}-\lambda I_{k\times k})\alpha_{k\times n}=Y_{k\times n}\\ = =>\alpha_{k\times n}=(X_{k\times m}X^T_{m\times k}-\lambda I_{k\times k})^{-1}Y_{k\times n}$

遇到不可线性回归的情况,考虑核方法,由于预测时并不用求出具体的值,假设预测时有q组观测值,则预测值 $Y^p_{q\times n}$ ,在线性条件下有

Y p q \times n = X p q \times m θ m \times n = X p q \times m X T m \times k (X k \times m X T m \times k - λ I k \times k) - 1 Y k \times n

$Y^p_{q\times n}=X^p_{q\times m}\theta_{m\times n}\\ =X^p_{q\times m}X^T_{m\times k}(X_{k\times m}X^T_{m\times k}-\lambda I_{k\times k})^{-1}Y_{k\times n}$

变换到这个等式,形成了训练集的內积,就可以使用核技巧了.
定义核函数 $K(X_1,X_2)=<\Phi(X1),\Phi(X2)>$ ,其中 $\Phi$ 为空间变换函数,不需要指明.
则

Y p q \times n = ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ K (X p 1, X 1) K (X p 2, X 1) ⋮ K (X p q, X 1) K (X p 1, X 2) K (X p 2, X 2) ⋮ K (X p q, X 2) \dots \dots ⋱ \dots K (X p 1, X k) K (X p 2, X k) ⋮ K (X p q, X k) ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ ⎜ ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ K (X 1, X 1) K (X 2, X 1) ⋮ K (X k, X 1) K (X 1, X 2) K (X 2, X 2) ⋮ K (X k, X 2) \dots \dots ⋱ \dots K (X 1, X k) K (X 2, X k) ⋮ K (X k, X k) ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ - λ I k \times k ⎞ ⎠ ⎟ ⎟ ⎟ ⎟ ⎟ Y k \times n

$Y^p_{q\times n}=\left[\begin{matrix}K(X_1^p,X_1)&K(X_1^p,X_2)&\dots&K(X_1^p,X_k)\\ K(X_2^p,X_1)&K(X_2^p,X_2)&\dots&K(X_2^p,X_k)\\ \vdots&\vdots&\ddots&\vdots\\ K(X_q^p,X_1)&K(X_q^p,X_2)&\dots&K(X_q^p,X_k)\\ \end{matrix}\right]\\\left(\left[\begin{matrix}K(X_1,X_1)&K(X_1,X_2)&\dots&K(X_1,X_k)\\ K(X_2,X_1)&K(X_2,X_2)&\dots&K(X_2,X_k)\\ \vdots&\vdots&\ddots&\vdots\\ K(X_k,X_1)&K(X_k,X_2)&\dots&K(X_k,X_k)\\ \end{matrix}\right]-\lambda I_{k\times k}\right)Y_{k\times n}$
核岭回归适用于解决小样本的非线性回归问题,随着问题规模的扩大不再适用,而且抗噪声能力不强,多用于手机定位的计算