机器学习-白板推导 P6_2 (SVM 模型求解 对偶)

SVM 模型求解 对偶问题

primal problem 原问题

$$带约束：\{\begin{array}{l}{\min }_{w,b}\frac{1}{2}{w}^{T}w\\ s.t.y_i(w^T{x}_i+b)\ge 1,for\forall i=1,2..N\end{array}$$

拉格朗日函数：
$L(w,b,\lambda )=\frac{1}{2}{w}^{T}w+{\sum }_{i=1}^N{\lambda }_i(1-y_i(w^T{x}_i+b))$
$\lambda \ge 01-y_i(w^T{x}_i+b)\le 0$

$$无约束：\{\begin{array}{l}{\min }_{w,b}{\max }_{\lambda }L(w,b,\lambda )\\ s.t.\lambda \ge 0\end{array}$$

$$\{\begin{array}{l}if:1-y_i(w^T{x}_i+b)>0，{\max }_{\lambda }L(w,b,\lambda )=\frac{1}{2}{w}^{T}w+\infty =\infty \\ if:1-y_i(w^T{x}_i+b)\le 0，{\max }_{\lambda }L(w,b,\lambda )=\frac{1}{2}{w}^{T}w+0=\frac{1}{2}{w}^{T}w,{\min }_{w,b}{\max }_{\lambda }L(w,b,\lambda )={\min }_{w,b}\frac{1}{2}{w}^{T}w\end{array}$$
$\Rightarrow$
${\min }_{w,b}{\max }_{\lambda }L(w,b,\lambda )={\min }_{w,b}(\infty ,\frac{1}{2}{w}^{T}w)={\min }_{w,b}\frac{1}{2}{w}^{T}w$

dual problem 对偶问题

$$强对偶：\{\begin{array}{l}{\max }_{\lambda }{\min }_{w,b}L(w,b,\lambda )\\ s.t.\lambda \ge 0\end{array}$$
若对偶关系：$\min \max L\ge \max \min L$

强对偶关系：$\min \max L=\max \min L$

凸优化二次型问题，满足强对偶。

${\min }_{w,b}L(w,b,\lambda )$

$$\begin{array}{rl}\frac{\partial L}{\partial b}& =\frac{\partial }{\partial b}[\sum _{i=1}^N{\lambda }_i-\sum _{i=1}^N{\lambda }_i{y}_i(w^T{x}_i+b)]\\ & =\frac{\partial }{\partial b}[-\sum _{i=1}^N{\lambda }_i{y}_{i}b]\\ & =-\sum _{i=1}^N{\lambda }_i{y}_i=0\end{array}$$
带入$L(w,b,\lambda )$
$$\begin{array}{rl}L(w,b,\lambda )& =\frac{1}{2}{w}^{T}w+\sum _{i=1}^N{\lambda }_i(1-y_i(w^T{x}_i+b))\\ & =\frac{1}{2}{w}^{T}w+\sum _{i=1}^N{\lambda }_i-\sum _{i=1}^N{\lambda }_i{y}_i{w}^T{x}_i-\sum _{i=1}^N{\lambda }_i{y}_{i}b\\ & =\frac{1}{2}{w}^{T}w+\sum _{i=1}^N{\lambda }_i-\sum _{i=1}^N{\lambda }_i{y}_i{w}^T{x}_i\end{array}$$

$$\begin{array}{rl}& \frac{\partial L}{\partial w}=\frac{1}{2}w-\sum _{i=1}^N{\lambda }_i{y}_i{x}_i=0\\ & \Rightarrow w=\sum _{i=1}^N{\lambda }_i{y}_i{x}_i\end{array}$$
$$\begin{array}{rl}L(w,b,\lambda )& =\frac{1}{2}(\sum _{i=1}^N{\lambda }_i{y}_i{x}_i{)}^T(\sum _{i=1}^N{\lambda }_i{y}_i{x}_i)-\sum _{i=1}^N{\lambda }_i{y}_i(\sum _{j=1}^N{\lambda }_j{y}_j{x}_j{)}^T{x}_i+\sum _{j=1}^N{\lambda }_i\\ & =\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\lambda }_i{\lambda }_j{y}_i{y}_j{x}_i^T{x}_j-\sum _{i=1}^N\sum _{j=1}^N{\lambda }_i{\lambda }_j{y}_i{y}_j{x}_j^T{x}_i+\sum _{j=1}^N{\lambda }_i\\ & =-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\lambda }_i{\lambda }_j{y}_i{y}_j{x}_i^T{x}_j+\sum _{j=1}^N{\lambda }_i\end{array}$$

$$等价代换：\{\begin{array}{l}{\max }_{\lambda }-\frac{1}{2}{\sum }_{i=1}^N{\sum }_{j=1}^N{\lambda }_i{\lambda }_j{y}_i{y}_j{x}_i^T{x}_j+{\sum }_{j=1}^N{\lambda }_i\\ s.t.{\lambda }_i\ge 0\end{array}$$