Algorithm Learning：Supervised Learning - Logistic Regression - two-class Classification

Introduction

This report focuses on two-class Classification and introduces the Logistic Regression , including theories, explanations, formula derivations, potential applications and so on.
.
Note：This report is a study note of “CS229 Lecture Notes”, by Andrew Ng, Stanford University.

---

Theories of Logistic Regression

Classification and regression are two major problems that machine learning can solve. From the perspective of the types of predicted values, the quantitative output of continuous variable prediction is called regression.The qualitative output of discrete variable prediction is called classification.

Logistic Regression, although with the word “Regression” in its name, is actually a classification method for two classification problems (i.e., only two kinds of outputs). In this method the variable that we’re trying to predict is a variable y that we can think of as taking on two values either 0 or 1. Class 0 is called “Negative Class” and Class 1 is called “Positive Class” .
y ∈ { 0 , 1 } y\in\lbrace {0},{1}\rbrace y∈{0,1}

Note：The formula represents the value space of the label y .

---

Procedure Explanation

The theories of logistic regression and linear regression are similar and can be simply described as follows:

（1）Find an appropriate prediction function (called Hypothesis in Andrew Ng’s class), which is generally expressed as $h$ function.

（2）Define a decision boundary. Its meaning is to find out exactly when logistic regression determines the data to be of class 1 or class 0.

（3）Construct a cost function. In this article, the cross entropy function based on maximum likelihood estimation is used as the cost function to analyze.

（4）Use gradient descent to minimize the cost function.

1. Hypothesis Representation

In this section, Let’s introduce the logistic regression hypothesis first. According to the above steps, we need to find a suitable hypothesis representation $h$_θ$(x)$. And here the hypothesis of logistic regression is $$g(h(\theta ))=\frac{1}{1+e^{-{\theta }^{T}x+ϵ}}$$where
$$g(z)=\frac{1}{1+e^{-z}}$$is called the logistic function or the sigmoid function, it looks like this:

Notice that it starts off near 0 and then rises until it crosses 0.5 and the origin, and then it flattens out again like so. It has the following characteristics: if $z>>0$, $y\to 1$，else if $z<<0$，$y\to 0$. This has strong practical implications: $y$ represents the probability that the sample belongs to class 1.

Based on this, the probabilities of classification results of input X as class 1 and class 0 are respectively:
$$\begin{gathered} P(y=1|x;\theta )=h_{\theta }(x) \\ P(y=0|x;\theta )=1-h_{\theta }(x) \end{gathered}$$

2. Decision Boundary

The decision boundary is a property of the hypothesis under the parameters. Once we have the parameters θ, we can determine the decision boundaries. The decision boundary is: $${\theta }^{T}x=0$$If ${\theta }^{T}x\ge 0$, then predict $y=1$;
If ${\theta }^{T}x<0$, then predict $y=0$;

Let’s start with an example:

example:
The figure above depicts a linear decision boundary，decision boundary form is as follows：$${\theta }^{T}x={\theta }_0{x}_0+{\theta }_1{x}_1+{\theta }_2{x}_2,x_0=1$$$$if\theta =[-3,1,1{]}^T$$The prediction function is constructed as:$$h_{\theta }(x)=g({\theta }^{T}x)=\frac{1}{1+e^{3-x_1-x_2}}$$

Predict $y=1$, if $-3+x_1+x_2\ge 0$, that is, if $x_1+x_2\ge 3$, then $y=1$,
Predict $y=0$, if $-3+x_1+x_2<0$, that is, if $x_1+x_2<3$, then $y=0$.

3. Cost Function

Assuming that the $n$ training examples were generated independently, the loss function and the J function are as follows, which are derived from the maximum likelihood estimation: $$Loss(h_{\theta }(x),y)=\{\begin{array}{ll}-log(h_{\theta }(x)),& \text{if y=1}\\ -log(1-h_{\theta }(x)),& \text{if y=0}\end{array}$$$$J(\theta )=\frac{1}{m}\sum _{i=1}^{m}Loss(h_{\theta }(x^{(i)}),y^{(i)})=-\frac{1}{m}\sum _{i=1}^m(y^{(i)}log(h_{\theta }(x^{(i)}))+(1-y^{(i)})log(1-h_{\theta }(x^{(i)})))$$

And then we’re going to do some formula derivations:
$$\begin{gathered} P(y=1|x;\theta )=h_{\theta }(x) \\ P(y=0|x;\theta )=1-h_{\theta }(x) \end{gathered}$$In fact, this can be combined into the following formula:$$P(y|x;\theta )=(h_{\theta }(x){)}^y(1-h_{\theta }(x){)}^{1-y}$$Take the likelihood function:$$L(\theta )=\prod _{i=1}^{m}P(y|x^{(i)};\theta )=\prod _{i=1}^m(h_{\theta }(x^{(i)}){)}^{y^{(i)}}(1-h_{\theta }(x^{(i)}){)}^{1-y^{(i)}}$$After taking the logarithm:$$l(\theta )=logL(\theta )=\sum _{i=1}^m(y^{(i)}log(h_{\theta }(x^{(i)}))+(1-y^{(i)})log(1-h_{\theta }(x^{(i)})))$$The above equation is to gain θ when x is maximized by using the maximum likelihood estimation. If gradient descent method is used, a negative coefficient should be added to obtain the final J function：$$J(\theta )=-\frac{1}{m}l(\theta )$$In this case, we just have to minimize $J$ to get $\theta$.

Note：The likelihood function of parameter θ given output x is equal to the probability of taking x given θ：$L(\theta |x)=P(x|\theta )$.

4. Gradient Descent

Then we use gradient descent to train this model. The $\theta$ update process is derived as follows:

${\theta }_j:={\theta }_j-\alpha \frac{\delta }{\delta {\theta }_j}J(\theta )$

$\frac{\delta }{\delta {\theta }_j}J(\theta )=-\frac{1}{m}{\sum }_{i=1}^m(y^{(i)}\frac{1}{h_{\theta }(x^{(i)})}\frac{\delta }{\delta {\theta }_j}{h}_{\theta }(x^{(i)})-(1-y^{(i)})\frac{1}{1-h_{\theta }(x^{(i)})}\frac{\delta }{\delta {\theta }_j}{h}_{\theta }(x^{(i)}))$

$\frac{\delta }{\delta {\theta }_j}J(\theta )=-\frac{1}{m}{\sum }_{i=1}^m(y^{(i)}\frac{1}{h_{\theta }(x^{(i)})}-(1-y^{(i)})\frac{1}{1-h_{\theta }(x^{(i)})})\frac{\delta }{\delta {\theta }_j}{h}_{\theta }(x^{(i)})$

for $g(z)$:

$g^{\prime }(z)=\frac{d}{dz}\frac{1}{1+e^{-z}}=\frac{1}{(1+e^{-z}{)}^2}{e}^{-z}=\frac{1}{1+e^{-z}}(1-\frac{1}{1+e^{-z}})=g(z)(1-g(z))$

$\frac{\delta }{\delta {\theta }_j}g({\theta }^{T}x)=g({\theta }^{T}x)(1-g({\theta }^{T}x))\frac{\delta }{\delta {\theta }_j}{\theta }^T{x}^{(i)}$

To the original:

$\frac{\delta }{\delta {\theta }_j}J(\theta )=-\frac{1}{m}{\sum }_{i=1}^m(y^{(i)}\frac{1}{g({\theta }^T{x}^{(i)})}-(1-y^{(i)})\frac{1}{1-g({\theta }^T{x}^{(i)})})g({\theta }^T{x}^{(i)})(1-g({\theta }^T{x}^{(i)}))\frac{\delta }{\delta {\theta }_j}{\theta }^T{x}^{(i)}$

$=-\frac{1}{m}{\sum }_{i=1}^m(y^{(i)}(1-g({\theta }^T{x}^{(i)}))-(1-y^{(i)})g({\theta }^T{x}^{(i)}))x_j^{(i)}$

$=-\frac{1}{m}{\sum }_{i=1}^m(y^{(i)}-g({\theta }^T{x}^{(i)}))x_j^{(i)}$

$=\frac{1}{m}{\sum }_{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}$

Finally, we get Parameter $\theta$ update process:

${\theta }_j:={\theta }_j-\alpha \frac{1}{m}{\sum }_{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)},j=0,1,...,n$

This has a lot of similarities to the linear regression gradient descent process.

5. Vectorization

To facilitate code implementation, the parameter update process needs to be vectorized further, and the following is the mathematical derivation of the vectorization process:

1. The matrix form of the training data is specified as follows, where each row of $x$ is a unit sample and each column is a different characteristic value.

$$x=\left[\begin{array}{c}{x}^{(1)}\\ x^{(2)}\\ ...\\ x^{(m)}\end{array}\right]=\left[\begin{array}{cccc}{x}_0^{(1)}& x_1^{(1)}& ...& x_n^{(1)}\\ x_0^{(2)}& x_1^{(2)}& ...& x_n^{(2)}\\ ...& ...& ...& ...\\ x_0^{(m)}& x_1^{(m)}& ...& x_n^{(m)}\end{array}\right],y=\left[\begin{array}{c}{y}^{(1)}\\ y^{(2)}\\ ...\\ y^{(m)}\end{array}\right](1)$$

2. The matrix form of the parameter to be solved is as follows:
   $$\theta =\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ ...\\ {\theta }_n\end{array}\right](2)$$

3. $x\cdot \theta$ write for $A$:
   $$A=x\cdot \theta =\left[\begin{array}{cccc}{x}_0^{(1)}& x_1^{(1)}& ...& x_n^{(1)}\\ x_0^{(2)}& x_1^{(2)}& ...& x_n^{(2)}\\ ...& ...& ...& ...\\ x_0^{(m)}& x_1^{(m)}& ...& x_n^{(m)}\end{array}\right]\cdot \left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ ...\\ {\theta }_n\end{array}\right]=\left[\begin{array}{c}{\theta }_0{x}_0^{(1)}+{\theta }_1{x}_1^{(1)}+...+{\theta }_n{x}_n^{(1)}\\ {\theta }_0{x}_0^{(2)}+{\theta }_1{x}_1^{(2)}+...+{\theta }_n{x}_n^{(2)}\\ ...\\ {\theta }_0{x}_0^{(m)}+{\theta }_1{x}_1^{(m)}+...+{\theta }_n{x}_n^{(m)}\end{array}\right]=\left[\begin{array}{c}{A}^{(1)}\\ A^{(2)}\\ ...\\ A^{(m)}\end{array}\right](3)$$

4. $h_{\theta }(x)-y$ write for $E$:
   $$E=h_{\theta }(x)-y=\left[\begin{array}{c}g(A^{(1)})-y^{(1)}\\ g(A^{(2)})-y^{(2)}\\ ...\\ g(A^{(m)})-y^{(m)}\end{array}\right]=\left[\begin{array}{c}{E}^{(1)}\\ E^{(2)}\\ ...\\ E^{(m)}\end{array}\right]=g(A)-y(4)$$

5. Put the formula (4) results into the Parameter $\theta$ update process:
   ${\theta }_j:={\theta }_j-\alpha \frac{1}{m}{\sum }_{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}$
   
   $={\theta }_j-\alpha \frac{1}{m}{\sum }_{i=1}^m{E}^{(i)}{x}_j^{(i)}(5)$
   
   $={\theta }_j-\alpha \frac{1}{m}(x_j^{(1)}{x}_j^{(2)},...,x_j^{(m)})E$

6. The sum of (5) in terms of parameters is:
   $\theta =\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ ...\\ {\theta }_n\end{array}\right]:=\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ ...\\ {\theta }_n\end{array}\right]-\frac{1}{m}\alpha \left[\begin{array}{cccc}{x}_0^{(1)}& x_0^{(2)}& ...& x_0^{(m)}\\ x_1^{(1)}& x_1^{(2)}& ...& x_1^{(m)}\\ ...& ...& ...& ...\\ x_n^{(1)}& x_n^{(2)}& ...& x_n^{(m)}\end{array}\right]\cdot E$
   
   $=\theta -\frac{1}{m}\alpha x^{T}E(6)$

In summary, the parameterization steps are as follows:
$$A=x\cdot \theta \to E=g(A)-y\to \theta :=\theta -\frac{1}{m}\alpha x^{T}E$$
The composite representation is:
$$\theta :=\theta -\frac{1}{m}\alpha x^T(g(x\cdot \theta )-y)$$

Potential Applications

There are many application scenarios of logistic regression algorithm. At present, this algorithm has been widely used in virtual application recommendation and guess the buyer’s choice.Extract as many features as possible to accurately judge and cater to user preferences.

Logistic regression has its potential application value in many fields including finance. For instance, in the application of credit risk: PD modeling prediction: As a measure of default rate, PD’s modeling needs to rely on historical data, that is, the record of default of the credit portfolio of financial institutions within a certain period of time in the past.Default and non-default can be represented by 0 and 1 respectively, and default data can be marked by a binary data.With features and markers, modeling can be done, and since the output is a binary variable, logistic regression is the natural choice, and the model parameters are clearly defined and easily accepted by the industry and regulatory authorities.

Logistic regression can also play a role in the field of circuits. By extracting features and judging the input and output of integrated circuits, machines can automatically tell us the damage information of the circuits without having to disassemble them.