机器学习基石笔记（二）：学会说 是/否

Lecture 2: Learning to Answer Yes/No

Perceptron Hypothesis Set

For $x=(x_1,x_2,\cdot \cdot \cdot ,x_d)$ ‘features of sample’, compute a weighted ‘score’(${\sum }_{i=1}^d{w}_i{x}_i>threshold$)
and approve credit if score>threshold, deny credit if score>threshold and ignore the equals.
Y : { + 1 , − 1 } Y:\{+1, -1\} Y:{+1,−1}:
$$\begin{gathered} h(x)=sign((\sum _{i=1}^d{w}_i{x}_i)-threshold)\stackrel{w_0=-threshold,x_0=+1}{=}sign((\sum _{i=1}^d{w}_i{x}_i)+w_0\ast x_0) \\ =sign((\sum _{i=0}^d{w}_i{x}_i))=sign(w^{T}x) \end{gathered}$$
called ‘perceptron’ hypothesis historically

Perceptrons in $R^2$

perceptrons（感知器） ⇔ linear (binary) classifiers

Fun Time

Consider using a perceptron to detect spam messages.
Assume that each email is represented by the frequency of keyword
occurrence, and output +1 indicates a spam. Which keywords below
shall have large positive weights in a good perceptron for the task?
1. coffee, tea, hamburger, steak
2. free, drug, fantastic, deal  $✓$
3. machine, learning, statistics, textbook
4. national, Taiwan, university, coursera

Explanation
垃圾邮件常含"免费",’'打折","好消息"等词。

Perceptron Learning Algorithm (PLA)

PLA算法就是要在Perceptrons(Perceptron Hypothesis Set)找到合适的Perceptron，而Perceptron由$w$决定，即找到合适$w$，使样本完全分开。

Cyclic PLA

Cyclic PLA算法如下所示
（1）初始化$w$->$w_0$
（2）更新$w$
  For t = 0,1,…
    find a mistake of $w_t$ called ($x_{n(t)}$,$y_{n(t)}$)
      $sign(w_t^T{x}_{n(t)})̸=y_{n(t)}$

    correct the mistake by
      $w_{(t+1)}\leftarrow w_t+y_{n(t)}{x}_{n(t)}$

  … until no more mistakes
  return last w (called $w_{PLA}$ ) as g

PLA算法的精髓在于找到与感知器分类不正确的点，然后通过不正确的点修正感知器，直到所有的样本点分类正确。
修正公式：$w_{(t+1)}\leftarrow w_t+y_{n(t)}{x}_{n(t)}$
修正图示：

Fun Time

Let’s try to think about why PLA may work.
Let n = n(t), according to the rule of PLA below, which formula is true?
$$sign(w_t^T{x}_n)̸=y_n，w_{(t+1)}\leftarrow w_t+y_n{x}_n$$
1. $w_{t+1}^T{x}_n=y_n$
2. $sign(w_{t+1}^T{x}_n)=y_n$
3. $y_n{w}_{t+1}^T{x}_n\ge y_n{w}_t^T{x}_n$  $✓$
4. $y_n{w}_{t+1}^T{x}_n<y_n{w}_t^T{x}_n$

Explanation
经过($x_n$, $y_n$)更新后，$w_t$到$w_{t+1}$的变化
由于感知器是相对修正($\Delta w=y_n{x}_n$)，修正后不一定就能将($x_n$, $y_n$)分类正确。

$$y_n{w}_{t+1}^T{x}_n-y_n{w}_t^T{x}_n=(w_{t+1}^T-w_t^T)y_n{x}_n=x_n^T{y}_n^T{y}_n{x}_n\stackrel{y_n^T{y}_n=+1}{=}{x}_n^T{x}_n\ge 0$$

Guarantee of PLA

Linear Separability

if PLA halts (i.e. no more mistakes), (necessary condition) D allows some w to make no mistake call such D linear separable.

linear separable $D$ ⇔ exists perfect ${\mathbf{w}}_{\mathbf{f}}$ such that $sign(w_f^T{x}_n)=y_n$

感知器的修正

Fun Time中我们推导出感知器每次修正，$y_n{w}_{t+1}^T{x}_n\ge y_n{w}_t^T{x}_n$, $w_t$就会向$w_f$逼近。
我们计算$\Delta w_f^T{w}_{t+1}$再次确认下
$$\begin{gathered} \because w_{f}isperfectfor{x}_n \\ \therefore y_{n(t)}{w}_t^T{x}_{n(t)}\ge \underset{n}{\min }{y}_{n(t)}{w}_t^T{x}_{n(t)}>0 \\ {\mathbf{w}}_{\mathbf{t}}^{\mathbf{T}}{\mathbf{w}}_{\mathbf{n}(\mathbf{t})}\uparrow byupdatingwithany(x_{n(t)},y_{n(t)}) \\ \begin{array}{rl}{\mathbf{w}}_f^T{\mathbf{w}}_{t+1}& ={\mathbf{w}}_f^T\left({\mathbf{w}}_t+y_{n(t)}{\mathbf{x}}_{n(t)}\right)\\ & \ge {\mathbf{w}}_f^T{\mathbf{w}}_t+\underset{n}{\min }{y}_n{\mathbf{w}}_f^T{\mathbf{x}}_n\\ & >{\mathbf{w}}_f^T{\mathbf{w}}_t+0\end{array} \end{gathered}$$

感知器每次修正，${\mathbf{w}}_f^T{\mathbf{w}}_{t+1}>{\mathbf{w}}_f^T{\mathbf{w}}_t$，$w_t$是在向$w_f$逼近。

但是我们希望的逼近是夹角的逼近，而不是数值的增大。计算$\Delta ||w_t|{|}^2$。

$$\begin{array}{rl}{\Vert {\mathbf{w}}_{t+1}\Vert }^2& ={\Vert {\mathbf{w}}_t+y_{n(t)}{\mathbf{x}}_{n(t)}\Vert }^2\\ & ={\Vert {\mathbf{w}}_t\Vert }^2+2y_{n(t)}{\mathbf{w}}_t^T{\mathbf{x}}_{n(t)}+{\Vert y_{n(t)}{\mathbf{x}}_{n(t)}\Vert }^2\\ 又\mathrm{sign}& \left({\mathbf{w}}_t^T{\mathbf{x}}_{n(t)}\right)̸=y_{n(t)}\iff y_{n(t)}{\mathbf{w}}_t^T{\mathbf{x}}_{n(t)}\le 0\\ & \le {\Vert {\mathbf{w}}_t\Vert }^2+0+{\Vert y_{n(t)}{\mathbf{x}}_{n(t)}\Vert }^2\\ & \le {\Vert {\mathbf{w}}_t\Vert }^2+\underset{n}{\max }{\Vert y_n{\mathbf{x}}_n\Vert }^2\\ & \le {\Vert {\mathbf{w}}_t\Vert }^2+\underset{n}{\max }{\Vert {\mathbf{x}}_n\Vert }^2\end{array}$$
最大增长不会超过$\underset{n}{\max }{\Vert {\mathbf{x}}_n\Vert }^2$

实际上$w_0=0$，在T次修正后$\cos \theta =\frac{{\mathbf{w}}_f^T}{\Vert {\mathbf{w}}_f\Vert }\frac{{\mathbf{w}}_T}{\Vert {\mathbf{w}}_T\Vert }\ge \sqrt{T}\cdot constant$
证明:
$$\begin{gathered} R^2=\underset{n}{\max }\Vert {\mathbf{x}}_n{\Vert }^2\rho =\frac{\underset{n}{\min }{y}_n{\mathbf{w}}_f^T{x}_n}{\Vert {\mathbf{w}}_{\mathbf{f}}\Vert } \\ \because {\mathbf{w}}_0=0{\mathbf{w}}_f^T{\mathbf{w}}_T\ge T\underset{n}{\min }{y}_n{\mathbf{w}}_f^T{\mathbf{x}}_n{\Vert {\mathbf{w}}_T\Vert }^2\le T\underset{n}{\max }{\Vert {\mathbf{x}}_n\Vert }^2 \\ cos\theta =\frac{{\mathbf{w}}_f^T}{\Vert {\mathbf{w}}_f\Vert }\frac{{\mathbf{w}}_T}{\Vert {\mathbf{w}}_T\Vert }=\frac{1}{\Vert {\mathbf{w}}_f\Vert }\frac{{\mathbf{w}}_f^T{\mathbf{w}}_T}{\Vert {\mathbf{w}}_T\Vert }=\frac{1}{\Vert {\mathbf{w}}_f\Vert }\frac{T\underset{n}{\min }{y}_n{\mathbf{w}}_f^T{\mathbf{x}}_n}{\sqrt{T}\sqrt{\underset{n}{\max }{\Vert {\mathbf{x}}_n\Vert }^2}}\ge \sqrt{T}\frac{\rho }{R} \\ \therefore \cos \theta =\frac{{\mathbf{w}}_f^T}{\Vert {\mathbf{w}}_f\Vert }\frac{{\mathbf{w}}_T}{\Vert {\mathbf{w}}_T\Vert }\ge \sqrt{T}\cdot constant,其中constant=\frac{\rho }{R} \end{gathered}$$

Fun Time

Let’s upper-bound T, the number of mistakes that PLA ‘corrects’.
$$Define{R}^2=\underset{n}{\max }\Vert {\mathbf{x}}_n{\Vert }^2\rho =\frac{\underset{n}{\min }{y}_n{\mathbf{w}}_f^T{x}_n}{\Vert {\mathbf{w}}_{\mathbf{f}}\Vert }$$

We want to show that $T\le \square$. Express the upper bound $\square$ by the two terms above.
1 $\frac{R}{\rho }$
2 $\frac{R^2}{{\rho }^2}$  $✓$
3 $\frac{R}{{\rho }^2}$
4 $\frac{{\rho }^2}{R^2}$


Explanation
$$\begin{gathered} \sqrt{T}\frac{\rho }{R}\le \frac{{\mathbf{w}}_f^T}{\Vert {\mathbf{w}}_f\Vert }\frac{{\mathbf{w}}_T}{\Vert {\mathbf{w}}_T\Vert }=cos\theta \le 1 \\ \therefore T\ge \frac{{\rho }^2}{R^2} \end{gathered}$$

Non-Separable Data

More about PLA

保证
数据线性可分，存在一个感知器能使数据完全分开。

过程
通过一次次修正，更新感知器的参数。

结果
最终会达到完全可分，不知道会更新多少次。

Pocket PLA

假设数据存在噪音(noise)，线性不可分，此时我们想得到一个大致完全分类的感知器。
Pocket PLA：modify PLA algorithm (black lines) by keeping best weights in pocket
（1）初始化$w$->$w_0$
（2）更新$w$
  For t = 0,1,…
    find a (random) mistake of $w_t$ called ($x_{n(t)}$,$y_{n(t)}$)
      $sign(w_t^T{x}_{n(t)})̸=y_{n(t)}$

    correct the mistake by
      $w_{(t+1)}\leftarrow w_t+y_{n(t)}{x}_{n(t)}$

    if $w_{t+1}$ makes fewer mistakes than $\hat{\mathbf{w}}$, replace $\hat{\mathbf{w}}$ by $w_{t+1}$
      $w_{(t+1)}\leftarrow w_t+y_{n(t)}{x}_{n(t)}$

  … until engough iterations
  return $\hat{\mathbf{w}}$ (called $w_{Pocket}$ ) as g

Pocket PLA算法：
  按照PLA一样更新$w_t$的值，但是用一个Pocket(​$\hat{\mathbf{w}}$)存储当前犯错误最少的感知器。
  当执行一定时间或者更新次数，返回当前​$\hat{\mathbf{w}}$。

Fun Time

Should we use pocket or PLA?
Since we do not know whether D is linear separable in advance, we may decide to just go with pocket instead of PLA. If D is actually linear separable, what’s the difference between the two?
1 pocket on D is slower than PLA  $✓$
2 pocket on D is faster than PLA
3 pocket on D returns a better g in approximating f than PLA
4 pocket on D returns a worse g in approximating f than PLA

Explanation
如果数据线性可分，PLA和Pocket PLA结果一样，获得一个将数据完全分类（no mistakes）的感知器；
但是Pocket PLA会慢些，主要原因是每一轮，需要检查犯错的个数；其次与当前最优感知器比较，替换也会耗费一些时间。

Summary

讲义总结

Perceptron Hypothesis Set
  感知器假设集是多维空间的超平面

Perceptron Learning Algorithm (PLA)
  $sign(w_t^T{x}_{n(t)})̸=y_{n(t)}$，$w_{(t+1)}\leftarrow w_t+y_{n(t)}{x}_{n(t)}$
  $g\leftarrow w_T$

Guarantee of PLA
  若数据集线性可分，则使用PLA算法，找一个完全分类的感知器。

Non-Separable Data
  若数据集线性不可分，则使用Pocket PLA算法，找一个较好的感知器。
  Pocket PLA算法按照PLA一样更新$w_t$的值，但是用一个Pocket($\hat{\mathbf{w}}$)存储当前犯错误最少的感知器。
  $g\leftarrow \hat{\mathbf{w}}$

PLA $A$ takes linear separable $D$ and perceptrons H to get hypothesis g

参考文献

《Machine Learning Foundations》(机器学习基石)—— Hsuan-Tien Lin (林轩田)