Notes on Machine Learning By Andrew Ng (3)

Note on Machine Learning By Andrew Ng (3)

Click here for previous note.

Logistic Regression

—A classification algorithm.

Classificaton

binary classification: y ∈ { 0 , 1 } y \in \{0 ,1\} y∈{0,1}

0: “Negative Class”

1: “Positive Class”

If we want to use a line to fit or predict something (like the tumor is malignant ), we could set threshold classifier output $h_{\theta }(x)$ at 0.5.

If $h_{\theta }(x)\ge 0.5$, predict $y=1$,

if $h_{\theta }(x)<0.5$, predict $y=0$.

You may think linear regression does work in classification, but if the data changes a little bit, it would give us a really bad prediction.

Also, the classification requires $y=0$ or 1, but $h_{\theta }(x)$ by linear regression can be $>1$ or $<0$.

So, by using Logistic Regression, we will generate an $h_{\theta }(x)$ that around $[0,1]$.

Hypothesis Representation

Fix $h_{\theta }(x)$ that fit outr need, which is map $\mathbb{R}$ to $[0,1]$.
$$\begin{gathered} h_{\theta }(x)=g({\theta }^{T}x) \\ g(z)=\frac{1}{1+e^{-z}} \\ \to h_{\theta }(x)=\frac{1}{1+e^{-{\theta }^{T}x}} \end{gathered}$$
We call that Sigmoid function or Logistic function.

$h_{\theta }(x)=P(y=1|x;\theta )$

Probability that y=1, given x, parameterized by $\theta$.

$P(y=1|x;\theta )+P(y=0|x;\theta )=1$

Decision Boundary

The property of the hypothesis.

Separate two different classes by a line odr curve.

If $h_{\theta }(x)\ge 0.5$, predict $y=1$,

if $h_{\theta }(x)<0.5$, predict $y=0$.

When $z\ge 0$, $g(z)\ge 0.5$. So, $h_{\theta }(x)=g({\theta }^{T}x)\ge 0.5$, when ${\theta }^{T}x\ge 0$.

Non-linear decision boundaries

USE POLYNOMIAL TERMS

Cost Function

How do we choose parameters $\theta$ ?

For linear regression, $J(\theta )=\frac{1}{m}{\sum }_{i=1}^m\frac{1}{2}{(h_{\theta }(x^{(i)})-y^{(i)})}^2$, and we modify it to be like $J(\theta )=\frac{1}{m}{\sum }_{i=1}^{m}cost(h_{\theta }(x),y)$,and the $cost(h_{\theta }(x),y)=\frac{1}{2}{(h_{\theta }(x^{(i)})-y^{(i)})}^2$.

But it is a non-convex function, so we have to find a new cost function, which is convex and has a global minimum.

Logistic regression cost function

$$cost=0ify=1,h_{\theta }(x)=1$$

But as
$$\begin{gathered} h_{\theta }(x)\to 0 \\ cost\to \infty \end{gathered}$$

Capture intuition that if $h_{\theta }(x)=0$, (predict $P(y=1|x;\theta )=0$), but $y=1$, we’ll penalize learning algorithm by a very large cost.

Simplified cost function and gradient descent

$Cost(h_{\theta }(x),y)=-y\ast log(h_{\theta }(x))-(1-y)log(1-h_{\theta }(x))$
$$\begin{gathered} J(\theta )=\frac{1}{m}\sum _{i=1}^{m}Cost(h_{\theta }(x),y) \\ =-\frac{1}{m}[\sum _{i=1}^m{y}^{(i)}\ast log(h_{\theta }(x^{(i)}))+(1-y^{(i)})log(1-h_{\theta }(x^{(i)}))] \end{gathered}$$
To find parameters $\theta$:

$minJ(\theta )$

To make a prediction given new $x$:

Output $h_{\theta }(x)=\frac{1}{1+e^{-{\theta }^{T}x}}$

Gradient Descent

Repeat{

${\theta }_j:={\theta }_j-\alpha {\sum }_{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})\ast x_j^{(i)}$

Just like linear regression!

}

Advanced optimization

Multi-class classification（One vs. all）

Multiclass classification

Email tagging: Work, Friends, Family, Hobby

Weather: Sunny, Cloudy, Rain, Snow

One-vs-all

By sparating to lots of 2-class classification problems.

Train a logistic regression classifier $h_{\theta }^{(i)}(x)$ for each class $i$ to predict the probability that $y=i$.

On a new input $x$, to make a prediction, pick the class $i$ that maximizes $\underset{i}{\max }{h}_{\theta }^{(i)}(x)$.

Please click here to see the next note.