Algorithm: Logistic Regression and Gradient Descent

The most classic model in machine learning : Logistic Regression.

Some problems for two class classify

Logistic Regression is a base line for classify problem

When we design a model for machine learning system，at first we create a simple model which is used for the base line and then we optimize the model which will compare with the base line.

A classify problem:

for condition probabiliy: define P(y|x) we need to calculate p(y = 1 |x) and p(y = 0 |x).

For a sample data if p(y = 1 |x) > p(y = 0 |x), we can classify the sample to category '1' ,otherwise '0'.

Mapping the linear regression model W.Tx + b into the domain(0,1), sigmoid function:

 

An example:

the condition probability is :

Summary:

Proof for the logistic regression model is an linear model.

decision boundary

 

The loss function for logistic regression:

Maximum Likelihood : for the logistic regression model, there are two parameter as w and b(intercept)

We investigate the input sample data and estimate the maximum of the conditon probability related to the parameters , and maxize this probability to derive the best parameters for the model.

Maximum likelihood estimation: