Basics of Neural Network Programming - Logistic Regression cost function

For logistic regression, to train the parameters w and b, we need to define a cost function.

Loss function

The loss function measures the discrepancy between the prediction  and the desired output . In other words, the loss function computes the error for a single training example. It's defined as:

Let's take a look how the loss function is derived:

Given following training examples:

For specific training example,

And we want:

If , then . It means given , what's the probability of

If , then . It means given , what's the probability of

Above two can be merged into one as below:

So, just means the probability of (=0 OR =1) for given . We want this probability as high as possible.

Next, we define following equation since log function is a strictly monotonically increasing function, and maximizing gives you the same result as maximizing :

Because , .  So, we define the loss function as the following negavie of . And minimizing the loss corresponds to maximizing the log of the probability:

And

Let's try to understand this loss function:

• When , then
  • If , then
  • If , then
• When , then
  • If , then
  • If , then

Cost function

It's defined as:

It's the average of loss function over all m training examples.

Actually, this definition can also be derived as below based on Maximum Likelihood Estimation (最大似然估计):

If assuming that the training examples are IID (Identically Independently Distributed, 独立同分布), the probability of all the training examples is the product of probabilities:

To maximize the above that is the chance of your observations for all examples in the training set, is the same as miximizing the following:

Per Maximum Likelihood Estimation, we need choose the parameters w and b that maximizes above equation. This justifies the cost we had for logistic regression which is . But because now we want to minimize the cost instead of maximize the likelihood, we'll get rid of the minus sign. And also for convenience, we make sure that our quantities are better scale, we just add a 1/m scaling factor:

So, by minimizing the cost function , we're really carrying out maximum likelihood estimation with the logistic regression model under the assumption that our training examples are IID.

<end>