Lecture2-3Guarantee of PLA

Linear Separable

• If PLA halts(no mistakes),(necessary condition) $\mathcal{D}$ allows some $\mathbf{w}$ to make no mistake
• Call $\mathcal{D}$ linear separable

Linear Separable $\mathcal{D} \Longleftrightarrow$ exists perfect $\mathbf{w_f}$ such that $y_n = sign(\mathbf{w_f^Tx_n})$

$\mathbf{w_f}$ perfect hence every $\mathbf{x_n}$ correctly away from the line:

$$y_{n(t)}\mathbf{w_f^T x_{n(t)}} \geq \min_n y_{n}\mathbf{w_f^T x_{n}} > 0$$

Fact:$\mathbf{w_t}$ gets closer to $\mathbf{w_f}$ by updating.

$$\begin{align} \mathbf{w^T_fw_{t+1}} &= \mathbf{w^T_f}(\mathbf{w_t} + y_{n(t)}\mathbf{x_{n(t)}})\\ &\geq \mathbf{w^T_f}\mathbf{w_t} + y_{n(t)}\mathbf{w^T_f}\mathbf{x_{n(t)}}\\ &\geq \mathbf{w^T_f}\mathbf{w_t} + \min_n y_{n}\mathbf{w_f^T x_{n}}\\ &>\mathbf{w^T_f}\mathbf{w_t} \end{align}$$

Fact: $\mathbf{w_t}$ doesn’t grow too fast.

Derivation

After T derivations, we can have:

Remark1

$$\begin{align} \because \mathbf{w^T_fw_{T}} &= \mathbf{w^T_f}(\mathbf{w_{T-1}} + y_{n(T)}\mathbf{x_{n(T)}})\\\\ \therefore \frac{\mathbf{w^T_fw_{T}}}{\parallel \mathbf{w^T_f}\parallel}&= \frac{\mathbf{w^T_f}\mathbf{w_{T-1}}}{\parallel \mathbf{w^T_f}\parallel}+ \frac{\mathbf{w^T_f}y_{n(T)}\mathbf{x_{n(T)}}}{\parallel \mathbf{w^T_f}\parallel}\\ &\geq \frac{\mathbf{w^T_f}\mathbf{w_{T-1}}}{\parallel \mathbf{w^T_f}\parallel}+ \frac{\min_n\mathbf{w^T_f}y_{n}\mathbf{x_{n}}}{\parallel \mathbf{w^T_f}\parallel}\\ &\geq \frac{\mathbf{w^T_f}\mathbf{w_{T-1}}}{\parallel \mathbf{w^T_f}\parallel}+ \rho\\ &\geq \frac{\mathbf{w^T_f}\mathbf{w_{T-2}}}{\parallel \mathbf{w^T_f}\parallel}+2\rho\\ &...\\ &\geq T\rho \end{align}$$

Remark2

$$\begin{align} \parallel \mathbf{w_T}\parallel^2 &=\parallel\mathbf{w_{T-1}} + y_{n(T)}\mathbf{x_{n(T)}}\parallel^2\\ &= \parallel \mathbf{w_{T-1}}\parallel^2 + \parallel y_{n(T)}\mathbf{x_{n(T)}}\parallel^2 + 2\mathbf{w_{T-1}^T}y_{n(T)}\mathbf{x_{n(T)}}\\ &\leq \parallel \mathbf{w_{T-1}}\parallel^2 + \parallel\mathbf{x_{n(T)}}\parallel^2\\ &\leq\parallel \mathbf{w_{T-1}}\parallel^2 + \max_n\parallel\mathbf{x_n}\parallel^2\\ &\leq\parallel \mathbf{w_{T-1}}\parallel^2 + R^2\\ &\leq \parallel \mathbf{w_{T-2}}\parallel^2 + 2R^2\\ &...\\ &\leq TR^2 \end{align}$$

$\therefore$ the perceptron iterates at most $T$ iterations before it stops, if $\mathcal{D}$ is linear separable

$$\therefore 1 \geq \frac{\mathbf{w_f^T w_T}}{\parallel \mathbf{w_f}\parallel \parallel \mathbf{w_T}\parallel} \geq \frac{T\rho}{\sqrt{T}R} = \sqrt{T}\frac{\rho}{R}\\ \therefore T \leq \frac{R^2}{\rho^2}$$
where $\rho = \frac{\min_n\mathbf{w^T_f}y_{n}\mathbf{x_{n}}}{\parallel \mathbf{w^T_f}\parallel}$ and $R = \max_n\parallel\mathbf{x_n}\parallel$.