Coursera | Andrew Ng (01-week-2-2.3)—Logistic 回归成本函数

该系列仅在原课程基础上部分知识点添加个人学习笔记，或相关推导补充等。如有错误，还请批评指教。在学习了 Andrew Ng 课程的基础上，为了更方便的查阅复习，将其整理成文字。因本人一直在学习英语，所以该系列以英文为主，同时也建议读者以英文为主，中文辅助，以便后期进阶时，为学习相关领域的学术论文做铺垫。- ZJ

Coursera 课程 |deeplearning.ai |网易云课堂

---

转载请注明作者和出处：ZJ 微信公众号-「SelfImprovementLab」

知乎：https://zhuanlan.zhihu.com/c_147249273

CSDN：http://blog.csdn.net/JUNJUN_ZHAO/article/details/78855589

---

Logistic 回归 损失函数

(字幕来源：网易云课堂)

In the previous video,you saw the logistic regression model.To train the parameters W and B of the logistic regression model,you need to define a cost function.Let’s take a look at the cost function,you can use to train logistic regression.To recap, this is what we had to find from the previous slide.So your output y-hat is sigmoid of w transpose x plus b,where a sigmoid of Z is as defined here.

在上一个视频中，你看到的是 logistic 回归的模型，为了训练 logistic 回归模型的参数 w 以及 b，需要定义一个成本函数，让我们来看一下，用 logistic 回归来训练的成本函数，回忆一下 这是上一张幻灯片的函数 ，这里定义为$sigmoid(z)$ 。

So to learn parameters for your model you’re given a training set of m training examples,and it seems natural that you want to find parameters W and B,so that at least on the training set, the outputs you have. The predictions you have on the training set,which we only write as y-hat (i) that that will be close to the ground truth labels $y^{(i)}$ that you got in the training set.

为了让模型来通过学习调整参数，要给一个 m 个样本的训练集，很自然地你想通过在训练集 找到参数 w 和 b，来得到你的输出，对训练集中的预测值，将它写成 $\hat y^{(i)}$我们希望它会接近于在训练集中的 $y^{(i)}$ 值。

So to throw in a little bit more detail for the equation on top,we had said that y-hat is as defined at the top for a training example x and of course for each training example,we’re using these superscripts with round brackets with parentheses to index and to differentiate examples,Your prediction on training sample (i) which is y-hat (i) is going to be obtained by taking the sigmoid function,and applying it to w transpose $x^{(i)}$, the input that the training example plus B ,and you can also define $z^{(i)}$ as follows,$z^{(i)}$ is equal to the w transpose $x^{(i)}$ plus b ($w^{T} x^{(i)}+b$) .

为了让上面的方程更详细一些，需要说明上面这里定义的 $\hat y$，是对一个训练样本 x 来说的，对于每个训练样本，使用这些带有圆括号的上标，方便引用说明，还有区分样本，你的训练样本 (i) 对应的预测值是 $\hat y^{(i)}$，是用训练样本，通过 sigmoid 函数作用到 $w^{T} x^{(i)}+b$ 得到的。你也可以将$z^{(i)}$定义成这样，$z^{(i)}$ 等于$w^{T} x^{(i)}+b$。

So throughout this course ,we’re going to use this notational convention （符号约定）,that the superscript parentheses i refers to data,x or y or z or something else associated with the i-th training example,associated with the i-th example,That’s what the superscript i in parentheses means .

在这门课里，我们将使用这个符号约定，就是这个上标 (i) 来指明数据，表示 x 或者 y 或者 y 和第 i 个训练样本有关，和第 i 个样本有关，这就是上标(i)的含义。

Now, let’s see what loss function or error function,we can use to measure how well our algorithm is doing.One thing you could do is define the loss,when your algorithm outputs y-hat and the true label as Y to be maybe the square error or one half a square error. It turns out that you could do this,but in logistic regression people don’t usually do this,because when you come to learn the parameters ,you find that the optimization problem,which we talk about later becomes non-convex,So you end up with optimization problem with multiple local optima.

现在，我们来看看损失函数，或叫做误差函数，它们可以用来衡量算法的运行情况，你可以定义损失为，$\hat y$和 y 的差的平方，或者它们差的平方的 1/2 ，结果表明你可以这样做，但通常在 logistic 回归中大家都不这么做，因为当你学习这些参数的时候，你会发现之后讨论的优化问题，会变成非凸的，最后会得到很多个局部最优解。

So gradient descent may not find the global optimum,If you didn’t understand the last couple of comments,Don’t worry about it, we’ll get to it in later video,But the intuition （直观理解） to take away is that,this function L called the loss function is a function you’ll need to define to measure how good our output y-hat is when the true label is y.As square error seems like it might be a reasonable choice,except that it makes gradient descent not work well.

梯度下降法 可能找不到全局最优值，如果你不理解这几句话，别担心，我们会在后面的视频中讲到它，但是这个的直观理解就是，我们通过定义这个损失函数 L 来衡量你的预测输出值$\hat y$和 y 的实际值有多接近。误差平方看起来似乎是一个合理的选择，但用这个的话，梯度下降法就不太好用。

So in logistic regression, we will actually define a different loss function that plays a similar role as squared error, that will give us an optimization problem that is convex,and so we’ll see in that later video,becomes much easier to optimize. So, what we use in logistic regression is actually the following loss function which I’m just like right up here is negative y log y-hat plus one minus y log, one minus y-hat $-(y*log(\hat y) + (1-y)log(1-\hat y))$.

在 logistic 回归中，我们会定义一个不同的损失函数，它起着与误差平方相似的作用，这些会给我们一个凸的优化问题，在后面的视频能看到，它很容易去做优化，在 logistic 回归中我们用的，会是这里写的损失函，它是$-(y*log(\hat y) + (1-y)log(1-\hat y))$。

Here’s some intuition for why this loss function makes sense.Keep in mind that if we’re using squared error then you want the squared error to be as small as possible.And with this logistic regression loss function,we’ll also want this to be as small as possible.

直观地看看为何这个损失函数能起作用，记得如果我们使用误差平方越小越好。对于这个 logistic 回归的损失函数，同样地我们也想让它尽可能地小。

To understand why this makes sense,let’s look at the two cases.In the first case,let’s say Y is equal to one then the loss function y-hat comma y is justice this first term, $L(\hat y,y)$ right, with this negative sign.So this negative log y-hat （$-log(\hat y)$ ）. If y is equal to one.Because if y equals one then the second term one minus Y is equal to zero.So this says if y equals one,you want negative log y-hat to be as big(little) as possible.

为了更好地理解为什么它能够起作用，让我们来看两个例子，在第一个例子中，我们说 y=1 时，就是这第一项 $L(\hat y,y)$ 带个负号，就是$-log(\hat y)$ 如果 y=1，因为如果 y=1 那么第二项 1-y 就等于0，这就是说当 y=1时，你想让 $-log(\hat y)$ 尽可能小。

So that means you want log y-hat to be large,to be as big as possible and that means,you want y-hat to be large,But because y-hat is you know,the sigmoid function, it can never be bigger than one,So this is saying that if y is equal to one,you want y-hat to be as big as possible.But it can’t ever be bigger than one,so saying you want y-hat to be close to one as well.

这意味着想让 $log(\hat y)$够大，尽可能地大，这样就意味，你想要$\hat y$够大，但是因为$\hat y$是，$simoid$ 函数得出的，永远不会比 1 大，也就是说 如果 y=1 时，你会想让 $\hat y$ 尽可能地大，但它永远不会大于 1，所以 你要让$\hat y$接近 1。

The other case is if y equals zero,If y equals zero,then this first term in the loss function is equal to zero,because y zero and then the second term defines the loss function,So the loss becomes negative log one minus y-hat,And so if in your learning procedure,you try to make the loss function small,what this means is that you want log one minus y-hat to be large,And because it’s a negative sign there,and then through a similar piece of reason you can conclude that,this loss function is trying to make y-hat as small as possible,

补充 log 函数图:

另一个情况就是，如果 y=0，如果 y=0，损失函数的第一项等于 0，因为 y 是 0 然后第二项就是这个损失函数，这损失函数变成 $-log(1-\hat y)$，在学习过程中，想让损失函数小一些，也就意味着 你想要 $log(1-\hat y)$够大，因为这里有一个负号，通过这一系列推理 你可以得出，损失函数让$\hat y$尽可能地小，

And again because y-hat has to be between zero and one,This is saying that if y is equal to zero,then your loss function will push the parameters to make y-hat as close to zero as possible.Now, there are a lot of functions with roughly this effect that,if y is equal to one we try to make y-hat large,and if Y is equal to zero we try to make y-hat small.We just gave here in green,a somewhat informal justification for this loss function,

再次，因为$\hat y$只能介于 0 到 1 之间，这就是说，当 y=0 时，损失函数会让这些参数，让 $\hat y$ 尽可能地接近 0，有很多函数都能达到这个效果，如果 y=1 我们尽可能让 $\hat y$ 很大，如果 y=0 尽可能让 $\hat y$ 足够小，绿色字体这里，稍微解释了为什么用这个作为损失函数，

we’ll provide an optional video later,to give a more formal justification for,why in logistic regression we like to use the loss function with this particular form.Finally, the loss function,was defined with respect to a single training example.It measures how well you’re doing on a single training example.I’m now going to define something called the cost function,which measures how well you’re doing an entire training set.

后面我们会提供选修视频，给出更正式的这样做的原因，解释为什么在logistic 回归中要用这个形式的损失函数，最后说一下 损失函数是，在单个训练样本中定义的，它衡量了在单个训练样本上的表现，下面我要定义一个成本函数， 它衡量的是在全体训练样本上的表现。

So the cost function J which is applied to your parameters W and B is going to be the average with one over m of the sum of the loss function ,applied to each of the training examples and turn,While here y-hat is of course the prediction output,by your logistic regression algorithm using you know,a particular set of parameters W and B,

这个成本函数 J，根据之前得到的两个参数 w 和 b $J(w, b)$等于，1/m 乘以求和 $L(\hat y^(i), y^(i))$，即所有训练样本的损失函数和，而$\hat y$是 用一组特定的参数 w 和 b，通过 logistic 回归算法，得出的预测输出值，

And so just to expand this out,this is equal to negative one over m,sum from i equals one through m of the definition of the loss function,So this is y (i) Log y-hat(i),plus 1 minus y (i) log one minus y-hat (i),I guess I could put square brackets here.So the minus sign is outside everything else.So the terminology I’m going to use is that,the loss function is applied to ,just a single training example like so.And the cost function is the cost of your parameters.

所以把这个展开，这等于 $-\frac{1}{m}$ ，从 i=1 到 m 对损失函数求和，这是$y^{(i)}*log(\hat y^(i))$，加上 $(1-y^{(i)})*log(1-\hat y^{(i)})$，我在这里画上方括号，负号在这一堆式子的外面，术语这样来用，损失函数只适用于，像这样的单个训练样本，这个成本函数基于参数的总成本。

So in training your logistic regression model ,we’re going to try to find parameters W and B,that minimize the overall cost function J written at the bottom.So, you’ve just seen the set up for the logistic regression algorithm,the loss function for training example,and the overall cost function for the parameters of your algorithm.It turns out that logistic regression,can be viewed as a very very small neural network,

所以在训练 logistic 回归模型时，我们要找到合适的参数 w 和 b，让下面这里的成本函数 J 尽可能地小，你刚刚看到了 logistic 回归算法的过程，以及训练样本的损失函数，还有和参数相关的总体成本函数，结果表明 logistic回归，可以被看作是 一个非常小的神经网络，

In the next video we’ll go over that,so you can start gaining intuition about what neural networks do,So that let’s go on to the next video,about how to view logistic regression,as a very small neural network.

在下一个视频里我们将会学到，直观地去理解神经网络能做什么，继续看下一段视频，看看如何将 logistic 回归,看作一个非常小的神经网络。

---

重点总结：

2.3 logistic回归 损失函数

Loss function

一般经验来说，使用平方错误（squared error）来衡量 Loss Function：

$L(\hat y, y)=\dfrac{1}{2}(\hat y-y)^{2}$

但是，对于 logistic regression 来说，一般不适用平方错误来作为Loss Function，这是因为上面的平方错误损失函数一般是非凸函数（non-convex），其在使用梯度下降算法的时候，容易得到局部最优解，而不是全局最优解。因此要选择凸函数。

逻辑回归的 Loss Function：

$L(\hat y, y)=-(y\log\hat y+(1-y)\log(1-\hat y))$

• 当 $y=1$ 时，$L(\hat y, y)=-\log \hat y$。如果 $\hat y$ 越接近 1，$L(\hat y, y) \approx 0$，表示预测效果越好；如果$\hat y$越接近 0，$L(\hat y, y) \approx +\infty$，表示预测效果越差；

• 当 $y=0$ 时，$L(\hat y, y)=-\log (1-\hat y)$。如果$\hat y$越接近0，$L(\hat y, y) \approx 0$，表示预测效果越好；如果$\hat y$越接近1，$L(\hat y, y) \approx +\infty$，表示预测效果越差；

• 我们的目标是最小化样本点的损失 Loss Function，损失函数是针对单个样本点的。

Cost function

全部训练数据集的 Loss function 总和的平均值即为训练集的代价函数（Cost function）。

$$J(w,b)=\dfrac{1}{m}\sum_{i=1}^{m}L(\hat y^{(i)}, y^{(i)})=-\dfrac{1}{m}\sum_{i=1}^{m}\left[y^{(i)}\log\hat y^{(i)}+(1-y^{(i)})\log(1-\hat y^{(i)})\right]$$

• Cost function 是待求系数 w 和 b的函数；
• 我们的目标就是迭代计算出最佳的 w 和 b 的值，最小化 Cost function，让其尽可能地接近于0。

参考文献：

[1]. 大树先生.吴恩达Coursera深度学习课程 DeepLearning.ai 提炼笔记（1-2）– 神经网络基础

---

PS: 欢迎扫码关注公众号：「SelfImprovementLab」！专注「深度学习」，「机器学习」，「人工智能」。以及 「早起」，「阅读」，「运动」，「英语 」「其他」不定期建群 打卡互助活动。