Linear Separable
- If PLA halts(no mistakes),(necessary condition) allows some w to make no mistake
- Call linear separable
Linear Separable ⟺ exists perfect wf such that yn=sign(wTfxn)
wf
perfect hence every
xn
correctly away from the line:
yn(t)wTfxn(t)≥minnynwTfxn>0
Fact: wt gets closer to wf by updating.
wTfwt+1=wTf(wt+yn(t)xn(t))≥wTfwt+yn(t)wTfxn(t)≥wTfwt+minnynwTfxn>wTfwt
Fact: wt doesn’t grow too fast.
Derivation
After T derivations, we can have:
Remark1
∵wTfwT∴wTfwT∥wTf∥=wTf(wT−1+yn(T)xn(T))=wTfwT−1∥wTf∥+wTfyn(T)xn(T)∥wTf∥≥wTfwT−1∥wTf∥+minnwTfynxn∥wTf∥≥wTfwT−1∥wTf∥+ρ≥wTfwT−2∥wTf∥+2ρ...≥Tρ
Remark2
∥wT∥2=∥wT−1+yn(T)xn(T)∥2=∥wT−1∥2+∥yn(T)xn(T)∥2+2wTT−1yn(T)xn(T)≤∥wT−1∥2+∥xn(T)∥2≤∥wT−1∥2+maxn∥xn∥2≤∥wT−1∥2+R2≤∥wT−2∥2+2R2...≤TR2
∴
the perceptron iterates at most
T
iterations before it stops, if
∴1≥wTfwT∥wf∥∥wT∥≥TρT‾‾√R=T‾‾√ρR∴T≤R2ρ2
where ρ=minnwTfynxn∥wTf∥ and R=maxn∥xn∥ .