Problem Set 3

1Lagrange Duality Formulate the Lagrange dual problem of the following
linear programming prob-lem min cT rs.t.Ax 二b where a ∈R is variable,c
∈ R"，A ∈Rk×n, b ∈ Rk.

解：设拉格朗日函数为$\mathcal{L}(x,\lambda )=c^{T}x+{\lambda }^T(Ax-b)$,

对应的对偶函数为$\mathcal{G}(\lambda )=in{f}_{\lambda }\mathcal{L}(x,\lambda )$，

而LP问题与对偶问题强对偶，KTT 条件成立，满足 stationarity

${\nabla }_x{c}^T{x}^{\ast }+{{\lambda }^{\ast }}^T(Ax-b)=0$

$⟹$ $c^T+{{\lambda }^{\ast }}^{T}A=0$

以及$A{x}^{\ast }-b=0$，因此该点处拉格朗日函数可以表达为

$\mathcal{L}(x^{\ast },{\lambda }^{\ast })=(-{\lambda }^{T}A)(A^{-1}b)+{\lambda }^T(A{x}^{\ast }-b)$

$\mathcal{L}(x^{\ast },{\lambda }^{\ast })=-{\lambda }^{T}b$

根据 Dual feasibility 得 ${\lambda }_i\ge 0$

LP问题的对偶问题标准形式为
$$\begin{gathered} ma{x}_{\lambda }-{\lambda }^{T}b \\ s.t.\lambda \ge 0,c^T+{\lambda }^{T}A=0 \end{gathered}$$
这里补充一种做法：
将拉格朗日对偶函数变换为 $\mathcal{G}(\lambda )=inf\mathcal{L}(x,\lambda )=inf(c^T+{\lambda }^{T}A)x-{\lambda }^{T}b$，
当 $c^T+{\lambda }^{T}A=0$ 时，$\mathcal{G}(\lambda )=-{\lambda }^{T}b$；
否则，$\mathcal{G}(\lambda )=\infty$，不存在极值。

sVM
2.1Convex Functions Prove f(w) = w" . (where w ∈ R") is a convex function.2.2Soft-Margin for Separable Data Consider training a
soft-margin SVM with C set to some positive constant.Suppose the
training data is linearly separable. Since increasing the 6; can
onlyincrease the objective of the primal problem (which we are trying
to minimize),at the optimal solution to the primal problem，all the
training examples willhave functional margin at least 1 and all the i
will be equal to zero. True orfalse? Explain! Given a linearly
separable dataset, is it necessarily better to usea a hard margin SVM
over a soft-margin SVM?
2.3In-bound Support Vectors in Soft-Margin sVMs Examples ar() with a > 0 are called support vectors (SVs). For soft-marginsVM we distinguish
between in-bound SVs，for which 0 <Qi<C, and boundsVs for which a; = C.
Show that in-bound SVs lie exactly on the margin.Argue that bound SVs
can lie both on or in the margin，and that they will“usually” lie in
the margin. Hint: use the KKT conditions.

2.1证：${\omega }^T\omega$是凸函数

$⟺$ $||\lambda x+(1-\lambda )y|{|}^2\le \lambda ||x|{|}^2+(1-\lambda )||y||$

$⟺$$\lambda ||x|{|}^2+(1-\lambda )||y||-(\lambda x+(1-\lambda )y{)}^T(\lambda x+(1-\lambda )y)\ge 0$

$⟺$$\lambda ||x|{|}^2+(1-\lambda )||y||-(\lambda x^T+(1-\lambda )y^T)(\lambda x+(1-\lambda )y)\ge 0$

$⟺$$\lambda ||x|{|}^2+(1-\lambda )||y||-({\lambda }^2{x}^{T}x+\lambda (1-\lambda )(y^{T}x+y^{T}x)+(1-\lambda {)}^2{y}^{T}y)\lambda (1-\lambda )(y^{T}x+y^{T}x)\ge 0$

$⟺$$(\lambda -{\lambda }^2)x^{T}x+(\lambda -{\lambda }^2)y^{T}y-\lambda (1-\lambda )(y^{T}x+y^{T}x)\ge 0$

$⟺$$(\lambda -{\lambda }^2)x^{T}x+(\lambda -{\lambda }^2)y^{T}y-\lambda (1-\lambda )(y^{T}x+y^{T}x)\ge 0$

而$\lambda \in [0,1]$，因此$\lambda \ge {\lambda }^2$，

$⟺$$x^{T}x+y^{T}y-(y^{T}x+y^{T}x)\ge 0$

$⟺$$(x^T-y^T)(x-y)\ge 0$

$⟺$$||x-y|{|}^2\ge 0$

而$||x-y|{|}^2\ge 0$成立，故${\omega }^T\omega$是凸函数，证毕。

2.2不一定，软间隔SVM模型表达为
$$\begin{gathered} mi{n}_{\omega ,b,\xi }\frac{1}{2}||\omega |{|}^2+C\sum _{i=1}^m{\xi }_i \\ s.t.y^{(i)}({\omega }^T{x}^{(i)}+b)\ge 1-{\xi }_i \\ {\xi }_i\ge 0,\forall i=1,2,...,m \end{gathered}$$
考虑一维情形如下

令$\forall {\xi }_i=0$，即退化为硬间隔SVM，求得决策边界为${\omega }_1$；

令${\xi }_j=0,j\ne i$，求得决策边界为${\omega }_2$；

目标函数设为$f$，$f({\omega }_1)=\frac{1}{2}{\omega }_1^2$，$f({\omega }_2)=\frac{1}{2}{\omega }_2^2+C{\xi }_i$，

当$\frac{1}{2}{\omega }_1^2>\frac{1}{2}{\omega }_2^2+C{\xi }_i$时，${\xi }_i$可以不为0，${\omega }_2$优于${\omega }_1$，因而最优解一定不是${\omega }_1$.

软间隔SVM可以避免过拟合，正如上面的例子，右侧橙色点可能是噪声，用硬间隔SVM会拟合噪声；

相反，前者通过松弛变量，泛化模型，提高鲁棒性，因此某些情况下有必要使用软间隔SVM。

2.3①当$0<{\alpha }_i^{\ast }<C$时，

根据KTT条件${\alpha }_i^{\ast }+r_i^{\ast }=C$得$0<r_i^{\ast }<C$，

又因为$r_i^{\ast }{\xi }_i^{\ast }=0$，所以${\xi }_i^{\ast }=0$，

因为${\alpha }_i^{\ast }(y^{(i)}({{\omega }^{\ast }}^T{x}^{(i)}+b^{\ast })+{\xi }_i^{\ast }-1)=0$，

所以$y^{(i)}({{\omega }^{\ast }}^T{x}^{(i)}+b^{\ast })+{\xi }_i^{\ast }-1=0$，

所以$y^{(i)}({{\omega }^{\ast }}^T{x}^{(i)}+b^{\ast })=1$，

即 in-bound SVs 在支撑平面上。

②当${\alpha }_i^{\ast }=C$时，类似的可以得到$y^{(i)}({{\omega }^{\ast }}^T{x}^{(i)}+b^{\ast })+{\xi }_i^{\ast }-1=0$，

而${\xi }_i^{\ast }\ge 0$，因此$y^{(i)}({{\omega }^{\ast }}^T{x}^{(i)}+b^{\ast })\le 1$，

即 bound SVs 在支撑平面上或者在间隔内。

而往往少数的点就能确定支撑平面（n 维空间 n 个点确定一个 boundary），因此大部分的点在间隔内。