机器学习技法2019作业一

机器学习技法作业链接
我做的是Machine Learning, Spring 2019。目前Bonus（17、18）没做。已做内容有一些存疑，格式和内容待优化。欢迎指出问题。

Transforms: Explicit versus Implicit

1）

import numpy as np
import matplotlib.pyplot as plt

x = np.asarray([[1, 0], [0, 1], [0, -1], [-1, 0], [0, 2], [0, -2], [-2, 0]])
y = [-1, -1, -1, 1, 1, 1, 1]

z1 = 2 * (x[:, 1] ** 2) - 4 * x[:, 0] + 2 
z2 = x[:, 0] ** 2 -2 * x[:, 1] - 3
z = [[z1[i], z2[i]] for i in range(len(x))]

plt.scatter(z1[:3], z2[:3], label="-1")
plt.scatter(z1[3:], z2[3:], label="+1")
plt.xlabel("z1")
plt.ylabel("z2")
plt.legend()
plt.plot([5, 5], [-7, 1])

2)

from sklearn import svm

x = np.asarray([[1, 0], [0, 1], [0, -1], [-1, 0], [0, 2], [0, -2], [-2, 0]])
y = [-1, -1, -1, 1, 1, 1, 1]

clf = svm.SVC(C=1e100, kernel='poly', degree=3, gamma=1, coef0=1)
clf.fit(x, y)

alpha = clf.dual_coef_
sv = clf.support_vectors_

$$\begin{gathered} g_{svm}(x_1,x_2)=sign(-0.29096704(1+x_2{)}^2)-0.29094205(1-x_2{)}^2+0.38089797(1-x_1{)}^2 \\ +0.10050786(1+2x_2{)}^2+0.10050326(1+2x_2{)}^2)-1.66606947 \end{gathered}$$

3)

def get_weight(alpha, sv):
    w = np.zeros(6)
    for i in range(len(sv)):
        wi = np.asarray([1, 
                         2 * sv[i][0],
                         2 * sv[i][1],
                         2 * sv[i][0] * sv[i][1],
                         sv[i][0] ** 2,
                         sv[i][1] ** 2])
        
        w = w + alpha[0][i] * wi
    return w

def z(w, b, x1, x2):
    return w[0] + w[1]*x1 + w[2]*x2 + w[3]*x1*x2 + w[4]*(x1**2) + w[5]*(x2**2) + b

w = get_weight(alpha, sv)
b = clf.intercept_

x1 = np.linspace(-2, 2, 2000)
x2 = np.linspace(-2, 2, 2000)
X1, X2 = np.meshgrid(x1, x2)

plt.scatter(x[:3, 0], x[:3, 1], label="-1")
plt.scatter(x[3:, 0], x[3:, 1], label="+1")
plt.contour(X1, X2, z(w, b, X1, X2))
plt.xlabel("x1")
plt.ylabel("x2")
plt.legend()

4)

不一样，1)相当于把二维数据转换到另一个二维空间中，3）是隐射到6维空间

Kernels and Transforms

5)

要证$$exp(-x^2)=\frac{1}{||\widetilde{\varphi (x)}||}$$
只需证$$exp(x^2)=||\tilde{\varphi }(x)||$$
只需证$$exp(2x^2)=||\tilde{\varphi }(x)|{|}^2$$
$$exp(2x^2)=\sum _{i=0}^{+\infty }\frac{2^i}{i!}\cdot x^i=||\tilde{\varphi }(x)|{|}^2$$
$$\therefore 证毕$$
这题告诉我们$\varphi (x)=\frac{\tilde{\varphi }(x)}{|\tilde{\varphi }(x)|}$

6)

$$\cos (x,x^{’})=\frac{x\cdot x^{{}^{\prime }}}{|x||x^{{}^{\prime }}|}=\frac{x}{|x|}\cdot \frac{x^{{}^{\prime }}}{|x^{{}^{\prime }}|}=\varphi (x{)}^T\varphi (x^{{}^{\prime }})$$
其中$\varphi (x)=\frac{x}{|x|}$, $x$不为0向量
$$K=\left[\begin{array}{cccc}\frac{x_1\cdot x_1}{|x_1||x_2|}& \frac{x_1\cdot x_2}{|x_1||x_2|}& \cdots & \frac{x_1\cdot x_n}{|x_1||x_n|}\\ \frac{x_2\cdot x_1}{|x_2||x_1|}& \frac{x_2\cdot x_2}{|x_2||x_2|}& \cdots & \frac{x_2\cdot x_n}{|x_1||x_n|}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{x_n\cdot x_1}{|x_n||x_1|}& \frac{x_n\cdot x_2}{|x_n||x_2|}& \cdots & \frac{x_n\cdot x_n}{|x_n||x_n|}\end{array}\right]=\left[\begin{array}{c}\frac{x_1}{|x_1|}\\ \frac{x_2}{|x_2|}\\ ⋮\\ \frac{x_n}{|x_n|}\end{array}\right]\left[\begin{array}{c}\frac{x_1}{|x_1|}，\frac{x_2}{|x_2|},\cdots ,\frac{x_n}{|x_n|}\end{array}\right]=X{X}^T$$
对于任意不为0的实列向量$M=[a_1,a_2,\cdots ,a_n{]}^T$，
$M^{T}KM=(M^{T}X)(X^{T}M)=(\frac{a_1{x}_1}{|x_1|}+\frac{a_2{x}_2}{|x_2|}\cdots \frac{a_n{x}_n}{|x_n|}{)}^2\ge 0$
$\therefore K半是正定矩阵$
又因为K是对称矩阵，所以当x不为0向量时，$cos(x,x^{’})$是一个有效的核函数。

Radius of Transformed Vectors via the Kernel

7）

$L(R,c,\lambda )=R^2+\sum _{n=1}^N{\lambda }_n(||z_n-c|{|}^2-R^2)$

8）

primal feasible: $||z_n-c|{|}^2\le R^2$
dual feasible: ${\lambda }_n\ge 0$
dual-inner optimal:
令$\frac{\partial L(R,c,\lambda )}{\partial R}=2R-2R\sum _{n=1}^N{\lambda }_n=0⟹\sum _{n=1}^N{\lambda }_n=1$
$\frac{\partial L(R,c,\lambda )}{\partial c_i}=\sum _{n=1}^{N}2{\lambda }_n(-z_{n,i}+c_i)⟹\frac{\partial L(R,c,\lambda )}{\partial c}=\sum _{n=1}^{N}2{\lambda }_n(-z_n+c)$
令$\frac{\partial L(R,c,\lambda )}{\partial c}=\sum _{n=1}^{N}2{\lambda }_n(-z_n+c)=0⟹c\sum _{n=1}^N{\lambda }_n=\sum _{n=1}^N{\lambda }_n{z}_n$
primal-inner optimal:${\lambda }_n(||z_n-c|{|}^2-R^2)=0$
由$c\sum _{n=1}^N{\lambda }_n=\sum _{n=1}^N{\lambda }_n{z}_n$和$\sum _{n=1}^N{\lambda }_n=1⟹c=\sum _{n=1}^N{\lambda }_n{z}_n$

9）

$$\begin{array}{rl}& \underset{R\in \mathbb{R},c\in {\mathbb{R}}^d}{\min }\underset{{\lambda }_n\ge 0}{\max }{R}^2+\sum _{n=1}^N{\lambda }_n(||z_n-c|{|}^2-R^2)\\ & =\underset{{\lambda }_n\ge 0}{\max }\underset{R\in \mathbb{R},c\in {\mathbb{R}}^d}{\min }{R}^2+\sum _{n=1}^N{\lambda }_n(||z_n-c|{|}^2-R^2)\\ & =\underset{{\lambda }_n\ge 0}{\max }\sum _{n=1}^N{\lambda }_n(||z_n-c|{|}^2)\\ & =\underset{{\lambda }_n\ge 0}{\max }\sum _{n=1}^N{\lambda }_n(||z_n-\sum _{m=1}^N{\lambda }_m{z}_m|{|}^2)\\ & s.t\sum _{n=1}^N{\lambda }_n=1\end{array}$$

10)

$$\begin{array}{rl}& \underset{{\lambda }_n\ge 0}{\max }\sum _{n=1}^N{\lambda }_n(||z_n-\sum _{m=1}^N{\lambda }_m{z}_m|{|}^2)\\ & =-\underset{{\lambda }_n\ge 0}{\min }\sum _{n=1}^N{\lambda }_n(||z_n-\sum _{m=1}^N{\lambda }_m{z}_m|{|}^2)\\ & =\underset{{\lambda }_n\ge 0}{\min }\sum _{n=1}^N(\sum _{m=1}^N{\lambda }_n{\lambda }_m{z}_n^T{z}_m-{\lambda }_n{z}_n^T{z}_n)\\ & =\underset{{\lambda }_n\ge 0}{\min }\sum _{n=1}^N\sum _{m=1}^N{\lambda }_n{\lambda }_{m}K(x_n,x_m)-\sum _{n=1}^N{\lambda }_{n}K(x_n,x_n)\\ & s.t\sum _{n=1}^N{\lambda }_n=1,{\lambda }_n\ge 0\end{array}$$
然后用二次规划求$\lambda$.
二次项系数$q_{n,m}=K(x_n,x_m)$
一次项系数$p_n=K(x_n,x_n)$
$a_{\ge }=1,a_{\le }=-1,a_n^T是长度为N,第n个位置为1，其余位置为0的向量$
$c_{\ge }=1,c_{\le }=-1,c_n=0$
可以根据上面得到$Q,p,A,c$四个矩阵：
$$Q=\left[\begin{array}{ccc}K(x_1,x_1)& \cdots & K(x_1,x_N)\\ K(x_2,x_1)& \cdots & K(x_2,x_N)\\ ⋮& ⋮& ⋮\\ K(x_N,x_1)& \cdots & K(x_N,x_N)\end{array}\right]$$
$$p=\left[\begin{array}{c}K(x_1,x_1)\\ K(x_2,x_2)\\ ⋮\\ K(x_N,x_N)\end{array}\right]$$
$$A=\left[\begin{array}{c}(1_N{)}^T\\ (-1_N{)}^T\\ I_{N\times N}\end{array}\right]$$
$$c=\left[\begin{array}{c}1\\ -1\\ 0_N\end{array}\right]$$
$((1_N{)}^T=[1,1,\cdots ,1],N个1，其他类似的同理)$
用$Q,p,A,c$解QP问题可得$\lambda$的值。
${\lambda }_i>0时，由{\lambda }_i(||z_i-c|{|}^2-R^2)=0知，||z_i-c|{|}^2-R^2=0，$
则$$\begin{array}{rl}R& =||z_i-c||=||z_i-\sum _{n=1}^N{\lambda }_n{z}_n||\\ & =\sqrt{z_i^T{z}_i-2\sum _{n=1}^N{\lambda }_n{z}_i^T{z}_n+\sum _{n=1}^N\sum _{m=1}^N{\lambda }_n{\lambda }_m{z}_n^T{z}_m}\\ & =\sqrt{K(x_i,x_i)-2\sum _{n=1}^N{\lambda }_{n}K(x_i,x_n)+\sum _{n=1}^N\sum _{m=1}^N{\lambda }_n{\lambda }_{m}K(x_n,x_m)}\end{array}$$

Hard-Margin versus Soft-Margin

11)

$C\ge \underset{1\le n\le N}{\max }{\alpha }_n^{\ast }⟹C\ge {\alpha }_n^{\ast }$ for all $ n=1,\cdots,N$
由hard-margin和soft-margin svm对偶形式可知，若hard-margin SVM的一些最优解${\alpha }^{\ast }$，若满足${\alpha }^{\ast }\le C$,则这些${\alpha }^{\ast }$同样为soft-margin svm的最优解。

Operation of Kernels

12)

$\tilde{C}$时，最后的对偶形式只有约束条件发生变化$0\le {\alpha }_n\le \tilde{C}$
$\tilde{C}=\frac{C}{p},p>0⟹0\le p{\alpha }_n\le C$
$$\begin{array}{rl}& \underset{\alpha }{\min }\frac{1}{2}\sum _{n=1}^N\sum _{m=1}^N{\alpha }_n{\alpha }_m{y}_n{y}_m{z}_n^T{z}_m-\sum _{n=1}^N{\alpha }_n\\ & \underset{\alpha }{\min }\frac{1}{2}\sum _{n=1}^N\sum _{m=1}^N{\alpha }_n{\alpha }_m{y}_n{y}_m\tilde{K}(x,x{}^{\prime })-\sum _{n=1}^N{\alpha }_n\\ & \underset{\alpha }{\min }\frac{1}{2}\sum _{n=1}^N\sum _{m=1}^N{\alpha }_n{\alpha }_m{y}_n{y}_{m}pK(x,x{}^{\prime })-\sum _{n=1}^N{\alpha }_n\\ & ⟹\underset{\alpha }{\min }\frac{1}{2}\sum _{n=1}^N\sum _{m=1}^N(p{\alpha }_n)(p{\alpha }_m)y_n{y}_{m}K(x,x{}^{\prime })-\sum _{n=1}^N(p{\alpha }_n)\end{array}$$
通过解QP问题得到的$p\alpha$就与为$C$时的$\alpha$是一样的，接下来求$g_{svm}$。

$b=y_s-\sum _{SVindicesn}{\alpha }_n{y}_n\tilde{K}(x_n,x_s)=y_s-\sum _{SVindicesn}(p{\alpha }_n)y_{n}K(x_n,x_s)$
$g_{svm}(x)=sign(\sum _{SVindicesn}{\alpha }_n{y}_n\tilde{K}(x_n,x)+b)=sign(\sum _{SVindicesn}(p{\alpha }_n)y_{n}K(x_n,x)+b)$
所以是等价的。

Experiments with Soft-Margin Support Vector Machine

13)

import numpy as np
import matplotlib.pyplot as plt
from sklearn.svm import SVC

def get_dataset(file_name, n):
    data_set = []
    x = []
    y = []
    with open(file_name, "r") as f:
        for line in f.readlines():
            line = line.strip().split()
            data_set.append(line)
    f.close()
    data_set = np.asarray(data_set, dtype=np.float64)
    x = data_set[:, 1:]
    y = data_set[:, 0]
    for i in range(len(y)):
        if (y[i] == n): y[i] = 1
        else: y[i] = -1
    return x, y

def save_parameter(content, filename):
    file = open(filename, 'a')
    for i in range(len(content)):
        file.write(str(content[i])+'\n')
    file.close()

def read_parameter(filename):
    content = []
    with open(filename, 'r') as f:
        for line in f.readlines():
            line = line.strip().strip('[').strip(']')
            words = line.split(',')
            content.append(words)
    f.close()
    #for i in range(len(content)):
    return np.asarray(content, dtype=np.float64)   

def err_rate(y_pred, y):
    e_cnt = 0
    n = len(y)
    for i in range(n):
        if (y_pred[i] != y[i]):
            e_cnt += 1
    return e_cnt / n

train_x, train_y = get_dataset("features.train.txt", 2)
test_x, test_y = get_dataset("features.test.txt", 2)

C_list = [1e-5, 1e-3, 1e-1, 1e1, 1e3]

# 前两列为w，最后一列为b
wb_list = [] 
for C in C_list:
    clf = SVC(C=C, kernel='linear')
    clf.fit(train_x, train_y)
    w = clf.coef_[0].tolist()
    b = clf.intercept_[0]
    w.append(b)
    wb_list.append(w)

log10_C = [-5, -3, -1, 1, 3]
w = wb[:, :2]
w_L2 = np.sqrt((w**2).sum(axis=1))
print(w_L2)

plt.plot(log10_C, w_L2)
plt.xlabel('$log_{10}C$')
plt.ylabel('$|w|$')

C越大w越大

14)

train_x, train_y = get_dataset('features.train.txt', 4)
test_x, test_y = get_dataset('features.test.txt', 4)

C_list = [1e-5, 1e-3, 1e-1, 1e1, 1e3]
e_in_list= []
for C in C_list:
    clf = SVC(C=C, kernel='poly', degree=2, gamma=1, coef0=1)
    clf.fit(train_x, train_y)
    y_pred = clf.predict(train_x)
    e_in_list.append(err_rate(y_pred, train_y))

log10_C = [-5, -3, -1, 1, 3]

plt.plot(log10_C, e_in_list)
plt.xlabel('$log_{10}C$')
plt.ylabel('$E_{in}$')

不同的C，$E_{in}$是一样的

15)

train_x, train_y = get_dataset('features.train.txt', 0)
test_x, test_y = get_dataset('features.test.txt', 0)

C_list = [1e-2, 1e-1, 1, 1e1, 1e2]
w_list = []
for C in C_list:
    clf = SVC(C=C, kernel='rbf', gamma=80)
    clf.fit(train_x, train_y)
    y_alpha = clf.dual_coef_
    m, n = y_alpha.shape
    y_alpha = y_alpha.reshape(n, m)
    sv = clf.support_vectors_
    w = (y_alpha * sv).sum(axis=0)
    w_list.append(w)

distance = 1 / np.sqrt((np.array(w_list) ** 2).sum(axis=1))

log10_C = [-2, -1, -0, 1, 2]

plt.plot(log10_C, distance)
plt.xlabel('$log_{10}C$')
plt.ylabel('distance')

C越大，支持向量离超平面越近，意味着对误差的容忍越小，就越‘硬’。

16)

def train_val_set():
    train_x, train_y = get_dataset('features.train.txt', 0)
    
    train_y = train_y.reshape(train_y.shape[0], 1)
    data_set = np.concatenate((train_x, train_y), axis=1)
    np.random.shuffle(data_set)
    
    val_data = data_set[:1000]
    train_data = data_set[1000:]
    
    val_x = val_data[:, :2]
    val_y = val_data[:, -1]

    train_x = train_data[:, :2]
    train_y = train_data[:, -1]
    return train_x, train_y, val_x, val_y

gamma_list = [1e-2, 1e-1, 1, 1e1, 1e2]
number = []

for i in range(100):
    train_x, train_y, val_x, val_y = train_val_set()
    err = []
    for gamma in gamma_list:
        clf = SVC(C=0.1, kernel='rbf', gamma=gamma)
        clf.fit(train_x, train_y)

        pred_y = clf.predict(val_x)
        err.append(err_rate(pred_y, val_y))
    
    index = err.index(min(err))
    number.append(gamma_list[index])

plt.hist(number)