SVR简介

Support vector machine (SVM) analysis is a popular machine learning tool for classification and regression, first identified by Vladimir Vapnik and his colleagues in 1992. Linear epsilon-insensitive SVM (ε-SVM) regression is also known as $L_1$ loss. In ε-SVM regression, the set of training data includes predictor variables and observed response values. The goal is to find a function f(x) that deviates from $y_n$ by a value no greater than ε for each training point x, and at the same time is as flat as possible.

Linear SVM Regression: Primal Formula

原SVR问题：
$$\begin{array}{l}{\min }_{\omega ,b}\frac{1}{2}\Vert \omega {\Vert }^2+C{\sum }_{k=1}^N{L}_{ϵ}(f(x_k)-y_k)\\ L_{ϵ}(z)=\{\begin{array}{l}0,if|z|<ϵ\\ |z|-ϵ,otherwise\end{array}\end{array}$$
引入松弛变量 ${\xi }_k,{\xi }_k^{\ast }$ , 将上式重写
$$\begin{array}{l}\min J(\omega )=\frac{1}{2}{\omega }^T\omega +C{\sum }_{k=1}^N({\xi }_k+{\xi }_k^{\ast })\\ s.t.\\ \begin{array}{l}{y}_k-(x_k^T\omega +b)\le ϵ+{\xi }_k\\ (x_k^T\omega +b)-y_k\le ϵ+{\xi }_k^{\ast }\\ {\xi }_k\ge 0\\ {\xi }_k^{\ast }\ge 0\end{array}\end{array}$$

Linear SVM Regression: Dual Formula

$$\begin{array}{l}L(\omega ,b,\xi ,{\xi }^{\ast },\alpha ,{\alpha }^{\ast },\mu ,{\mu }^{\ast })=\frac{1}{2}||\omega |{|}^2+C\sum _{i=1}^N({\xi }_i+{{\xi }_i}^{\ast })\\ -\sum _{i=1}^N{\mu }_i{\xi }_i-\sum _{i=1}^N{{\mu }_i}^{\ast }{{\xi }_i}^{\ast }+\sum _{i=1}^N{\alpha }_i(y_i-f(x_i)-ϵ-{\xi }_i)+\sum _{i=1}^N{{\alpha }_i}^{\ast }(f(x_i)-y_i-ϵ-{{\xi }_i}^{\ast })\\ s.t.\begin{array}{l}{\alpha }_i\ge 0,{\alpha }_i^{\ast }\ge 0\\ {\mu }_i\ge 0,{\mu }_i^{\ast }\ge 0\end{array}\end{array}\text{\hspace{1em}(3)}$$

KKT 偏导为0条件

$$\begin{array}{l}\frac{\partial }{\partial \omega }L(\omega ,b,\xi ,{\xi }^{\ast },\alpha ,{\alpha }^{\ast },\mu ,{\mu }^{\ast })=\omega -\sum _{i=1}^N{\alpha }_i{x}_i+\sum _{i=1}^N{{\alpha }_i}^{\ast }{x}_i=0\\ \frac{\partial }{\partial b}L(\omega ,b,\xi ,{\xi }^{\ast },\alpha ,{\alpha }^{\ast },\mu ,{\mu }^{\ast })=-\sum _{i=1}^N{\alpha }_i+\sum _{i=1}^N{{\alpha }_i}^{\ast }=0\\ \frac{\partial }{\partial {\xi }_i}L(\omega ,b,\xi ,{\xi }^{\ast },\alpha ,{\alpha }^{\ast },\mu ,{\mu }^{\ast })=C-{\mu }_i-{\alpha }_i=0\\ \frac{\partial }{\partial {\xi }_i^{\ast }}L(\omega ,b,\xi ,{\xi }^{\ast },\alpha ,{\alpha }^{\ast },\mu ,{\mu }^{\ast })=C-{\mu }_i^{\ast }-{\alpha }_i^{\ast }=0\end{array}\text{\hspace{1em}(4)}$$
代入式(3)得

$$\begin{array}{l}L(\omega ,b,\xi ,{\xi }^{\ast },\alpha ,{\alpha }^{\ast },\mu ,{\mu }^{\ast })=\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N({\alpha }_i-{{\alpha }_i}^{\ast })({\alpha }_j-{{\alpha }_j}^{\ast }){x_i}^T{x}_j+C\sum _{i=1}^N({\xi }_i+{{\xi }_i}^{\ast })-\sum _{i=1}^N{\mu }_i{\xi }_i-\sum _{i=1}^N{{\mu }_i}^{\ast }{{\xi }_i}^{\ast }\\ +\sum _{i=1}^N{y}_i({\alpha }_i-{{\alpha }_i}^{\ast })-\sum _{i=1}^Nϵ({\alpha }_i+{{\alpha }_i}^{\ast })-\sum _{i=1}^N{\alpha }_i{\xi }_i-\sum _{i=1}^N{{\alpha }_i}^{\ast }{{\xi }_i}^{\ast }-\sum _{i=1}^N({\alpha }_i-{{\alpha }_i}^{\ast }){x_i}^T\omega -\sum _{i=1}^N({\alpha }_i-{{\alpha }_i}^{\ast })b\\ =-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N({\alpha }_i-{{\alpha }_i}^{\ast })({\alpha }_j-{{\alpha }_j}^{\ast }){x_i}^T{x}_j+\sum _{i=1}^N{y}_i({\alpha }_i-{{\alpha }_i}^{\ast })-\sum _{i=1}^Nϵ({\alpha }_i+{{\alpha }_i}^{\ast })\end{array}$$

其余KKT条件
$$\begin{array}{l}{\alpha }_i(y_i-f(x_i)-ϵ-{\xi }_i)=0\\ {{\alpha }_i}^{\ast }(f(x_i)-y_i-ϵ-{{\xi }_i}^{\ast })=0\\ {\mu }_i{\xi }_i=0⟹(C-{\alpha }_i){\xi }_i=0\\ {{\mu }_i}^{\ast }{{\xi }_i}^{\ast }=0⟹(C-{\alpha }_i^{\ast }){\xi }_i^{\ast }=0\\ {\alpha }_i\ge 0,{{\alpha }_i}^{\ast }\ge 0\\ {\mu }_i\ge 0,{{\mu }_i}^{\ast }\ge 0\end{array}$$
解KKT条件，可得SVR解
$$f(x)={\omega }^{T}x+b=\sum _{i=1}^N({\alpha }_i-{\alpha }_i^{\ast })x_i^{T}x+b$$
仅当样本不落入 $ϵ$ 间的隔带中，相应的${\alpha }_i$和${\alpha }_i^{\ast }$才能取非零值。
使上式中的 ${\alpha }_i-{\alpha }_i^{\ast }\ne 0$ 的样本即为SVR的支持向量，它们落在 $ϵ$ 间的隔带之外。

在得到 ${\alpha }_i$ 后，若 $0<{\alpha }_i<C$ ，则必有 ${\xi }_i=0$ ，进而有
$$b=y_i-ϵ-{\omega }^T{x}_i$$
实践中采用更鲁棒的办法：选取多个或所有满足条件 $0<{\alpha }_i<C$ 的样本求解 b 后取平均值。

引入核函数，则有
$$f(x)=\sum _{i=1}^N({\alpha }_i-{\alpha }_i^{\ast })\varphi (x_i{)}^T\varphi (x)+b$$

参考资料

https://www.mathworks.com/help/stats/understanding-support-vector-machine-regression.html