Machine Learning - Logistic Regression - Multi-class Classification

This series of articles are the study notes of " Machine Learning ", by Prof. Andrew Ng., Stanford University. This article is the notes of week 3, Logistic Regression part II. This article contains Logistic Regression for Multi-class Classification (one-vs-all classification).

Logistic Regression

Multi-class Classification

In this section we'll talk about how to get logistic regression to work for multi-class classification problems. And in particular I want to tell you about an algorithm called one-versus-all classification.

Some examples of multi-class classification

• Email foldering/tagging: Work, Friends,Family, Hobby

y=1, 2, 3, 4

• Medical diagrams: Not ill, Cold, Flu

y=1, 2, 3

• Weather: Sunny, Cloudy, Rain, Snow

y=1, 2, 3, 4

Whereas previously for a binary classification problem, our data sets look like this. For a multi-class classification problem our data sets may look like this where here I'm using three different symbols to represent our three classes. 

So the question is given the data set with three classes where this is an example of one class, that's an example of a different class, and that's an example of yet a third class. How do we get a learning algorithm to work for the setting?

One-vs-all (one-vs-rest) classification

We know how to you know maybe fit a straight line to set for the positive and negative classes. You see an idea called one-vs-all classification. We can then take this and make it work for multi-class classification as well. Here's how a one-vs-all classification works. And this is also sometimes called one-vs-rest. 

Let's say we have a training set like that shown on the left, where we have three classes of y = 1, we denote that with a triangle, if y = 2, the square, and if y = 3, then the cross. What we're going to do is take our training set and turn this into three separate binary classification problems.

Start with class triangle △

So let's start with class one which is the triangle. We're going to essentially create a new sort of fake training set where classes two and three get assigned to the negative class. And class one gets assigned to the positive class.You want to create a new training set like that shown on the right, and we're going to fit a classifier which I'm going to callh_θ⁽¹⁾(x) where here the triangles are the positive examples and the circles are the negative examples. So think of the triangles being assigned the value of one and the circles assigned the value of zero. And we're just going to train a standard logistic regression classifier and maybe that will give us a position boundary that looks like that. This superscript one here stands for class one, so we're doing this for the triangles of class one.

Second class square □

Next we do the same thing for class two. going to take the squares and assign the squares as the positive class, and assign everything else, the triangles and the crosses, as a negative class. And then we fit a second logistic regression classifier and call this h_θ⁽²⁾(x), where the superscript two denotes that we're now doing this, treating the square class as the positive class.

Finally class cross ×

finally, we do the same thing for the third class and fit a third classifier h_θ⁽³⁾(x), and maybe this will give us a decision bounty of the visible cross fire. This separates the positive and negative examples like that.

So to summarize, what we've done is, we've fit three classifiers. So, for i = 1, 2, 3, we'll fit a classifier x super script i subscript theta of x. Thus trying to estimate what is the probability that y is equal to class i, given x and parametrized by theta.

One-vs-all 

Train a logistic regression classifier h_θ⁽ⁱ⁾(x) for each class i to predict the probability that y=i.

On a new input x , to make a prediction, pick the classi that maximizes