SVR推导

个人笔记，非教程

目标函数：

$$\begin{gathered} \underset{w,b}{\min }\frac{1}{2}||w|{|}^2+C\sum _{i=1}^l({\xi }_i+{\xi }_i^{\ast }) \\ s.t.\{\begin{array}{ll}{y}_i-<w,x_i>-b& \le \epsilon +{\xi }_i\\ <w,x_i>+b-y_i& \le \epsilon +{\xi }_i^{\ast }\\ {\xi }_i,{\xi }_i^{\ast }& \ge 0\end{array} \end{gathered}$$

拉格朗日函数

$$\begin{gathered} L=\frac{1}{2}||w|{|}^2+C\sum _{i=1}^l({\xi }_i+{\xi }_i^{\ast })-\sum _{i=1}^l({\eta }_i{\xi }_i+{\eta }_i^{\ast }{\xi }_i^{\ast }) \\ -\sum _{i=1}^l{\alpha }_i(\epsilon +{\xi }_i-y_i+<w,x_i>+b) \\ -\sum _{i=1}^l{\alpha }_i^{\ast }(\epsilon +{\xi }_i^{\ast }+y_i-<w,x_i>-b) \\ s.t.{\alpha }_i^{(\ast )},{\eta }_i^{(\ast )}\ge 0 \end{gathered}$$

原问题化为

$$\underset{w,b}{\min }\underset{{\alpha }_i^{(\ast )},{\eta }_i^{(\ast )}}{\max }L(w,b,{\xi }_i,{\xi }_i^{\ast },{\alpha }_i,{\alpha }_i^{\ast })$$

对偶问题

$$\underset{{\alpha }_i^{(\ast )},{\eta }_i^{(\ast )}}{\max }\underset{w,b}{\min }L(w,b,{\xi }_i,{\xi }_i^{\ast },{\alpha }_i,{\alpha }_i^{\ast })$$
$$KKT条件\{\begin{array}{l}{\alpha }_i(\epsilon +{\xi }_i-y_i+<w,x_i>+b)=0\\ {\alpha }_i^{\ast }(\epsilon +{\xi }_i^{\ast }+y_i-<w,x_i>-b)=0\\ (C-{\alpha }_i){\xi }_i=0\\ (C-{\alpha }_i^{\ast }){\xi }_i^{\ast }=0\end{array}$$

求导令其为零

$$\frac{\partial L}{\partial b}=\sum _{i=1}^l({\alpha }_i^{\ast }-{\alpha }_i)=0$$
$$\frac{\partial L}{\partial w}=w-\sum _{i=1}^l({\alpha }_i-{\alpha }_i^{\ast })x_i=0$$
$$\frac{\partial L}{\partial {\xi }_i^{(\ast )}}=C-{\alpha }_i^{(\ast )}-{\eta }_i^{(\ast )}=0$$

回归方程

$$w=\sum _{i=1}^l({\alpha }_i-{\alpha }_i^{\ast })x_i$$
$$f(x)=\sum _{i=1}^l({\alpha }_i-{\alpha }_i^{\ast })<x_i,x>+b$$

SMO完事