《统计学习方法》第7章习题答案（持续更新）

注：前面的习题待完善，稍后补充。

7.3

$$\begin{gathered} \underset{w,b,\xi }{\min }\frac{1}{2}{\parallel w\parallel }^2+C\sum _{i=1}^N{{\xi }_i}^2 \\ s.t.y_i(w\cdot x_i+b)\ge 1-{\xi }_i,i=1,2,\dots ,N \\ {\xi }_i\ge 0,i=1,2,\dots ,N \end{gathered}$$

对应的拉格朗日函数是$L(w,b,\xi ,\alpha ,\gamma )=\frac{1}{2}{\parallel w\parallel }^2+C{\sum }_{i=1}^N{{\xi }_i}^2+{\sum }_{i=1}^N{\alpha }_i(1-{\xi }_i-y_i(w\cdot x_i+b))-{\sum }_{i=1}^N{\gamma }_i{\xi }_i$

使用KKT条件得到

$$\begin{gathered} \frac{\partial L}{\partial w}=w-\sum _{i=1}^N{\alpha }_i{y}_i{x}_i=0 \\ \frac{\partial L}{\partial b}=-\sum _{i=1}^N{\alpha }_i{y}_i=0 \\ \frac{\partial L}{\partial {\xi }_i}=2C{\xi }_i-{\alpha }_i-{\gamma }_i=0 \end{gathered}$$

因此

$$\begin{gathered} w=\sum _{i=1}^N{\alpha }_i{y}_i{x}_i \\ \sum _{i=1}^N{\alpha }_i{y}_i=0 \\ 2C{\xi }_i={\alpha }_i+{\gamma }_i \end{gathered}$$

代入拉格朗日函数可得

$$\begin{gathered} \underset{w,b,\xi }{\min }L(w,b,\xi ,\alpha ,\gamma )=\frac{1}{2}{\parallel w\parallel }^2+C\sum _{i=1}^N{{\xi }_i}^2+\sum _{i=1}^N{\alpha }_i-\sum _{i=1}^N({\alpha }_i+{\gamma }_i){\xi }_i-\sum _{i=1}^N{\alpha }_i{y}_{i}w\cdot x_i-\sum _{i=1}^N{\alpha }_i{y}_{i}b \\ =-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i{\alpha }_j{y}_i{y}_j{x}_i{x}_j+\sum _{i=1}^N{\alpha }_i-\frac{1}{2}\sum _{i=1}^N({\alpha }_i+{\gamma }_i){\xi }_i \\ =-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i{\alpha }_j{y}_i{y}_j{x}_i{x}_j+\sum _{i=1}^N{\alpha }_i-\frac{1}{2}\sum _{i=1}^N({\alpha }_i+{\gamma }_i)\frac{{\alpha }_i+{\gamma }_i}{2C} \\ =-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i{\alpha }_j{y}_i{y}_j{x}_i{x}_j+\sum _{i=1}^N{\alpha }_i-\frac{1}{4C}\sum _{i=1}^N({\alpha }_i+{\gamma }_i{)}^2 \end{gathered}$$

对偶问题为
$$\begin{gathered} \underset{\alpha }{\max }W(\alpha )=-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i{\alpha }_j{y}_i{y}_j{x}_i{x}_j+\sum _{i=1}^N{\alpha }_i-\frac{1}{4C}\sum _{i=1}^N({\alpha }_i+{\gamma }_i{)}^2 \\ s.t.\sum _{i=1}^N{\alpha }_i{y}_i=0 \\ {\alpha }_i\ge 0,{\gamma }_i\ge 0,i=1,2,\dots ,N \end{gathered}$$

7.4

对$p$进行数学归纳。

当$p=1$时，$K(x,z)=x\cdot z$， 则$\varphi (x)=x$

假设$p=k$时，$K(x,z)=(x\cdot z{)}^k={\varphi }_k(x)\cdot {\varphi }_k(z)$

当$p=k$时，$K(x,z)=(x\cdot z{)}^{k+1}=(x\cdot z{)}^k(x\cdot z)={\varphi }_k(x)\cdot {\varphi }_k(z)(x\cdot z)$

不妨设${\varphi }_k(x)=(f_1(x),f_2(x),\dots ,f_m(x){)}^T,x=(x_1,x_2,\dots ,x_n{)}^T$

则$$\begin{gathered} K(x,z)=(f_1(x)f_1(z)+f_2(x)f_2(z)+\dots +f_m(x)f_m(z))(x_1{z}_1+x_2{z}_2+\dots +x_n{z}_n) \\ =f_1(x)f_1(z)(x_1{z}_1+x_2{z}_2+\dots +x_n{z}_n)+f_2(x)f_2(z)(x_1{z}_1+x_2{z}_2+\dots +x_n{z}_n)+\dots +f_m(x)f_m(z)(x_1{z}_1+x_2{z}_2+\dots +x_n{z}_n) \\ =(f_1(x)x_1)(f_1(z)z_1)+(f_1(x)x_2)(f_1(z)z_2)+\dots +(f_1(x)x_n)(f_1(z)z_n)+(f_2(x)x_1)(f_2(z)z_1)+\dots \\ +(f_2(x)x_n)(f_2(z)z_n)+(f_m(x)x_1)(f_m(z)z_1)+\dots +(f_m(x)x_n)(f_m(z)z_n) \\ :={\varphi }_{k+1}(x)\cdot {\varphi }_{k+1}(z) \end{gathered}$$

其中${\varphi }_{k+1}(x)=(f_1(x)x_1,f_1(x)x_2,\dots ,f_1(x)x_n,f_2(x)x_1,\dots ,f_2(x)x_n,\dots ,f_m(x)x_1,\dots ,f_m(x)x_n{)}^T$

因此$K(x,z)=(x\cdot z{)}^p$是正定核。