【Learning from data】Lecture 2_Learning to answer yes or no

The core idea of PLA

In perceptron learning algorithm, the core idea is to find an initial division line randomly to divide the two categories points labeled with ‘+1’ and ‘-1’, then correct the bias with one of the mistaken points iteratively. 

Assume that the equation of the green division line is ω_1x+ω_2y+ω_0 = 0, which divides the points into two areas, including area A that vector W points to and area B that the reverse of vector W points to, and vector W is the normal vector of the line. It is easily to know that the points in the area A that vector W points to is satisfied with ω_1x+ω_2y+ω_0 > 0. Otherwise, ω_1x+ω_2y+ω_0 < 0. 

In the above figure, the point labeled with ‘+’ is misclassified into area B. The angle between vector W and X is larger than 90 degrees, then we try to reduce the angle by adding vector W with yX. Here, y is equal to ‘+1’ along the forward direction of vector X to reduce the angle.

In the below figure, it is the same correcting process when a ‘-’ point is misclassified into area A. The angle between vector W and X is smaller than 90 degrees, then we try to increase the angle by adding vector W with yX. Here, y is equal to ‘-1’ along the opposite direction of vector X to increase the angle.

As for the reason we correct W with the bias yX other than other factor is that the only thing we can refer is the mistaken point. According to the Ockham's Razor, it may be more efficient to use yX to avoid some unnecessary problems.
 
 

Proof of PLA