摘要: 本文是吴恩达 (Andrew Ng)老师《机器学习》课程,第十三章《支持向量机》中第103课时《大间隔分类器的数学原理》的视频原文字幕。为本人在视频学习过程中记录下来并加以修正,使其更加简洁,方便阅读,以便日后查阅使用。现分享给大家。如有错误,欢迎大家批评指正,在此表示诚挚地感谢!同时希望对大家的学习能有所帮助.
————————————————
In this video, I'd like to tell you a bit about the math behind large margin classification. This video is optional, so please feel free to skip it. But it may also give you better intuition about how the optimization problem of the SVM, how that leads to large margin classifier.
In order to get started, let me first remind you a couple of properties of what vector inner product look like. Let's say I have two vectors and
. Both 2 dimensional vectors. Let's see what
looks like. And
is also called the inner product between the vectors
and
.
is a two dimensional vector, so I can plot it on this figure. So, let's say that is the vector
. And what I mean by that is on the horizontal axis, that value takes whatever value
is; and on the vertical axis, the height of that is whatever
is. Now, one quantity that will be nice to have is the norm (范数) of the vector
. So, these are double bars on the left and right, that denotes the norm or length of
. So, this just means really the euclidean length of the vector
. And this is by Pythagoras theorem is just equal to
. And this is the length of the vector
. That is a real number. Now, let's go back and look at the vector
because we want to compute the inner product. So
will be some other vector with some other value
. And so the vector
will look like that. Now, let's go back and look at how to compute the inner product between
and
. Let me take vector
and project it down onto the vector
. So, I'm going to take a orthogonal projection or a 90 degree projection, so project it down on
like so. And I'm going to do is measure length of this red line I just draw here. So, I'm going to call the length of that red line
. So
is the length or is the magnitude of the projection of the vector
onto the vector
. And it's possible to show that inner product
is going to equal to
. So, this is one way to compute the inner product. And if you actually do the geometry and figure out what
is and figure out what the norm
is. This should give you the same answer as the other way of computing the inner product which is if you take
. And so the theorem of linear algebra that these two formulas give you the same answer. And by the way,
is also equal to
. So, if you were to do the same process in reverse, instead of projecting
onto
, you could project
onto
. Then do the same process, but with the rows of
and
reversed. And you should actually get the same number whatever that number is. And just to clarify what's going on in this equation: the norm of
is a real number, and
is also a real number. So
is the regular multiplication of two real numbers, of the length of
and the norm of
. Just one last detail, which is if you look at the norm of
,
is actually signed, and it can either positive or negative. So, let me say what I mean by that, if
is a vector like this, And if
is a vector that looks like this. So, if the angle between
and
is greater than
, then if I project
onto
, what I get is a projection it looks like this and so you have that length
. And in this case, I will still have that
. Except that in this example,
will be negative. So, inner product if the angle between
and
is less than
, then
is the positive length for that red line. Whereas if the angle is greater than
, then
will be negative of the length of the little line. So the inner product between two vectors can be also negative. So, that's how vector inner products work. We're going to use these properties of vector inner product to try to understand the SVM optimization objective over there.
Here's the optimization objective for SVM. Just for the purpose of this side, I am going to make one simplification or once just make the objective easier to analyze. What I'm going to do is ignore the intercept term. Just set . To make things easier to plot, I'm also going to set
, the number of features, to be equal to 2. So, we have only 2 features
and
. Now, let's look at the objective function. The optimization objective of SVM. This can be written as
. What I'm going to do is rewrite this a bit, as
. And the reason I can do that is because for any number
,
. Now what you may notice is that this term inside the parenthesis, that's equal to the norm or the length of the vector
. And what I mean by that is that if we write out the vector
, then this term that I've just underlined in red, that's exactly the length or the norm of the vector
. And in fact, this is actually equal to the length of vector
, whether you write this as
, if
, or just the length of
. But for this side, I'm going to igore
. So finally, this means that my optimization objective is
. So all the SVM is doing in the optimization objective is it's minimizing the squared norm or the squared length of the parameter vector
. Now, what I'd like to do is look at these terms
and understand better what they're doing. So, given the parameter vector
, and given an example
, what is this equal to? On the previous slide, we've figured out what
looks like with different
and
. So, we're going to take those definitions, with
and
playing the roles of
and
. And let's see what that picture looks like. Let's say I look at a single training example. let's say I have a positive example that was drawing across there and let's say that's my example
. So what that really means is that I plotted on the horizontal axis some value
and on the vertical axis
. That's how I plot my training examples. And although we haven't been really thinking of this as a vector, what this really is, this is a vector from the origin
, out to the location of the training example. And now, let's say we have a parameter
, and I'm goging to plot that as a vector as well. So what is the inner product
? Well, using the early method, the way we compute that is we take my example and project it onto my parameter vector
. And then I'm goging to look at the length of this segment I'm coloring it in red. And I'm going to call that
to denote that this is a projection of the
training example onto the parameter vector
. And so what we have is that
. So, each of this is an equally valid way of computing the inner product between
and
. So, what does this leave us? This contraints that
or
, what does this means is that it can replace the use of constrains that
or
.
So, writing that into our optimization objective, this is what we get. Just to remind you we worked out earlier too that this optimization objective is . So, now let's consider the training example that we have at the bottom. And for now, continuing to use the simplification that
. Let's see what decision boundary the SVM will choose. Here's one option, let's say the SVM were to choose this decision boundary (green line). This is not a very good choise because it has very small margin. This decision boundary comes very close to the training examples. Let's see why the support vector machine will not do this. For this choice of parameters, it's possible to show that the parameter vector
(blud line) is actually at
to the decision boundary. So that green decision boundary corresponds to a parameter vector
that points in that direction. And by the way, the simplification that
that just means that the decision boundary has to pass through the origin
over there. So now, let's look at what this implies for the optimization objective. Let's say that this example here is my first example
. If we look at the projection of this example onto my parameter
, that little red segment. That is equal to
. That is going to be pretty small, right? And similarly, if this example here happens to be
. Then, if I look at the projection, this little magenta line segment is going to be
. It will actually be a negative number because it is in the opposite direction. This vector
has greater than
angle with my parameter vector
. So
is going to be less than
. And so what we're finding is that these terms
are going to be pretty small numbers. And so if we look at the optimization objective, and see for positive examples, we need
. But if
or
over here is pretty small, that means that we need the
to be pretty large. And similarly for our negative example, we need
. And we saw that in this example,
is going to be pretty small negative number, and so the only way for that to happen as well is for the norm of
to be large. But what we're doing in the optimization objective is we're trying to find a setting of parameters where the
is small, so this doesn't seem like a good direction for the parameter vector
. In contrast, just look at a different decision boundary. Here, let' say, this SVM choose that decision boundary. Now the picture is going to be very different. If that is the decision boundary, here is the corresponding direction for
. So, with the decision boundary, being that vertical line, it is possible to show using linear algebra, that the way to get green decision bounary is having the vector of
be at
to it. And now, if you look at the projection of your data onto the vector
, let's say as befor that this example is my example of
. So, when I project this onto
, what I find is that this is
. And for the other example
, I do the same projection. This length here (magenta) is
which is going to be less than
. And you notice that now
and
, these length of projections are going to be much bigger. So, if we still need to enforce these constraints,
, because
is so much bigger now, The
can be small. And so, what this means is that by choosing the decision boundary shown on the right, the SVM can make the norm of the parameter
much smaller and therefore make the
smaller. Which is why the SVM would choose this hypothesis on the right instead. And this is how the SVM gives rise to this large margin certification effect. Mainly, if you look at this green line, if you look at this green hypothesis, we want the projections of my positive and negative examples onto
to be large. And the only way for that to hold true is if surrounding the green line. This is a large margin that separates the positive and negative examples. And this is really the magnitude of this gap. The magnitude of this margin is exactly the values of
,
,
and so on. And so by making the margin large, the SVM can end up with a smaller value for the norm of
which is it is trying to do in the objective. And this is why the SVM ends up with large margin classifiers because it is trying to maximize the norm of these
which is the distance from the training examples to the decision boundary. Finally, we did this whole derivation using the simplification that
. The effect of that as I mentioned briefly is that if
, we're entertaining only decision boundaries that pass through the origin. If you allow
to be non zero, then what that means is that you entertain the decision boundaries that didn't pass through the origin like that one I just drew. I'm not going to do the full derivation. But it turns out that the same large margin works in pretty much exactly the same way. And there's a generalization of this argument that we just went through and not to go through. It shows that even when
is non 0, what the SVM is trying to do, when you have this optimization objective, is still finding large margin separator between the positive and negative examples. Which again corresponds to the case when
is very large. So, that explains how the SVM is a large margin classifier. In the next video, we'll start to talk about how to take some of these SVM ideas and start to apply them to build a complex nonlinear classifier as well.
<end>