Support Vector Machine - The mathematics behind large margin classification

摘要: 本文是吴恩达 (Andrew Ng)老师《机器学习》课程，第十三章《支持向量机》中第103课时《大间隔分类器的数学原理》的视频原文字幕。为本人在视频学习过程中记录下来并加以修正，使其更加简洁，方便阅读，以便日后查阅使用。现分享给大家。如有错误，欢迎大家批评指正，在此表示诚挚地感谢！同时希望对大家的学习能有所帮助.
————————————————

In this video, I'd like to tell you a bit about the math behind large margin classification. This video is optional, so please feel free to skip it. But it may also give you better intuition about how the optimization problem of the SVM, how that leads to large margin classifier.

In order to get started, let me first remind you a couple of properties of what vector inner product look like. Let's say I have two vectors and . Both 2 dimensional vectors. Let's see what looks like. And is also called the inner product between the vectors and . is a two dimensional vector, so I can plot it on this figure. So, let's say that is the vector . And what I mean by that is on the horizontal axis, that value takes whatever value is; and on the vertical axis, the height of that is whatever is. Now, one quantity that will be nice to have is the norm (范数) of the vector . So, these are double bars on the left and right, that denotes the norm or length of . So, this just means really the euclidean length of the vector . And this is by Pythagoras theorem is just equal to . And this is the length of the vector . That is a real number. Now, let's go back and look at the vector because we want to compute the inner product. So will be some other vector with some other value . And so the vector will look like that. Now, let's go back and look at how to compute the inner product between and . Let me take vector and project it down onto the vector . So, I'm going to take a orthogonal projection or a 90 degree projection, so project it down on like so. And I'm going to do is measure length of this red line I just draw here. So, I'm going to call the length of that red line . So is the length or is the magnitude of the projection of the vector onto the vector . And it's possible to show that inner product is going to equal to . So, this is one way to compute the inner product. And if you actually do the geometry and figure out what is and figure out what the norm is. This should give you the same answer as the other way of computing the inner product which is if you take . And so the theorem of linear algebra that these two formulas give you the same answer. And by the way,  is also equal to . So, if you were to do the same process in reverse, instead of projecting onto , you could project onto . Then do the same process, but with the rows of  and reversed. And you should actually get the same number whatever that number is. And just to clarify what's going on in this equation: the norm of is a real number, and is also a real number. So is the regular multiplication of two real numbers, of the length of and the norm of . Just one last detail, which is if you look at the norm of , is actually signed, and it can either positive or negative. So, let me say what I mean by that, if is a vector like this, And if is a vector that looks like this. So, if the angle between and is greater than , then if I project onto , what I get is a projection it looks like this and so you have that length . And in this case, I will still have that . Except that in this example, will be negative. So, inner product if the angle between and is less than , then is the positive length for that red line. Whereas if the angle is greater than , then will be negative of the length of the little line. So the inner product between two vectors can be also negative. So, that's how vector inner products work. We're going to use these properties of vector inner product to try to understand the SVM optimization objective over there.

Here's the optimization objective for SVM. Just for the purpose of this side, I am going to make one simplification or once just make the objective easier to analyze. What I'm going to do is ignore the intercept term. Just set . To make things easier to plot, I'm also going to set , the number of features, to be equal to 2. So, we have only 2 features and . Now, let's look at the objective function. The optimization objective of SVM. This can be written as . What I'm going to do is rewrite this a bit, as . And the reason I can do that is because for any number , . Now what you may notice is that this term inside the parenthesis, that's equal to the norm or the length of the vector . And what I mean by that is that if we write out the vector , then this term that I've just underlined in red, that's exactly the length or the norm of the vector . And in fact, this is actually equal to the length of vector , whether you write this as , if , or just the length of . But for this side, I'm going to igore . So finally, this means that my optimization objective is . So all the SVM is doing in the optimization objective is it's minimizing the squared norm or the squared length of the parameter vector . Now, what I'd like to do is look at these terms and understand better what they're doing. So, given the parameter vector , and given an example , what is this equal to? On the previous slide, we've figured out what looks like with different and . So, we're going to take those definitions, with and playing the roles of and . And let's see what that picture looks like. Let's say I look at a single training example. let's say I have a positive example that was drawing across there and let's say that's my example . So what that really means is that I plotted on the horizontal axis some value and on the vertical axis . That's how I plot my training examples. And although we haven't been really thinking of this as a vector, what this really is, this is a vector from the origin , out to the location of the training example. And now, let's say we have a parameter , and I'm goging to plot that as a vector as well. So what is the inner product ? Well, using the early method, the way we compute that is we take my example and project it onto my parameter vector . And then I'm goging to look at the length of this segment I'm coloring it in red. And I'm going to call that to denote that this is a projection of the training example onto the parameter vector . And so what we have is that . So, each of this is an equally valid way of computing the inner product between and . So, what does this leave us? This contraints that  or  , what does this means is that it can replace the use of constrains that or .

So, writing that into our optimization objective, this is what we get. Just to remind you we worked out earlier too that this optimization objective is . So, now let's consider the training example that we have at the bottom. And for now, continuing to use the simplification that . Let's see what decision boundary the SVM will choose. Here's one option, let's say the SVM were to choose this decision boundary (green line). This is not a very good choise because it has very small margin. This decision boundary comes very close to the training examples. Let's see why the support vector machine will not do this. For this choice of parameters, it's possible to show that the parameter vector (blud line) is actually at to the decision boundary. So that green decision boundary corresponds to a parameter vector that points in that direction. And by the way, the simplification that that just means that the decision boundary has to pass through the origin over there. So now, let's look at what this implies for the optimization objective. Let's say that this example here is my first example . If we look at the projection of this example onto my parameter , that little red segment. That is equal to . That is going to be pretty small, right? And similarly, if this example here happens to be . Then, if I look at the projection, this little magenta line segment is going to be . It will actually be a negative number because it is in the opposite direction. This vector has greater than angle with my parameter vector . So is going to be less than . And so what we're finding is that these terms are going to be pretty small numbers. And so if we look at the optimization objective, and see for positive examples, we need . But if or over here is pretty small, that means that we need the to be pretty large. And similarly for our negative example, we need . And we saw that in this example, is going to be pretty small negative number, and so the only way for that to happen as well is for the norm of to be large. But what we're doing in the optimization objective is we're trying to find a setting of parameters where the is small, so this doesn't seem like a good direction for the parameter vector . In contrast, just look at a different decision boundary. Here, let' say, this SVM choose that decision boundary. Now the picture is going to be very different. If that is the decision boundary, here is the corresponding direction for . So, with the decision boundary, being that vertical line, it is possible to show using linear algebra, that the way to get green decision bounary is having the vector of be at to it. And now, if you look at the projection of your data onto the vector , let's say as befor that this example is my example of . So, when I project this onto , what I find is that this is . And for the other example , I do the same projection. This length here (magenta) is which is going to be less than . And you notice that now and , these length of projections are going to be much bigger. So, if we still need to enforce these constraints, , because is so much bigger now, The can be small. And so, what this means is that by choosing the decision boundary shown on the right, the SVM can make the norm of the parameter much smaller and therefore make the smaller. Which is why the SVM would choose this hypothesis on the right instead. And this is how the SVM gives rise to this large margin certification effect. Mainly, if you look at this green line, if you look at this green hypothesis, we want the projections of my positive and negative examples onto to be large. And the only way for that to hold true is if surrounding the green line. This is a large margin that separates the positive and negative examples. And this is really the magnitude of this gap. The magnitude of this margin is exactly the values of , , and so on. And so by making the margin large, the SVM can end up with a smaller value for the norm of which is it is trying to do in the objective. And this is why the SVM ends up with large margin classifiers because it is trying to maximize the norm of these which is the distance from the training examples to the decision boundary. Finally, we did this whole derivation using the simplification that . The effect of that as I mentioned briefly is that if , we're entertaining only decision boundaries that pass through the origin. If you allow to be non zero, then what that means is that you entertain the decision boundaries that didn't pass through the origin like that one I just drew. I'm not going to do the full derivation. But it turns out that the same large margin works in pretty much exactly the same way. And there's a generalization of this argument that we just went through and not to go through. It shows that even when is non 0, what the SVM is trying to do, when you have this optimization objective, is still finding large margin separator between the positive and negative examples. Which again corresponds to the case when is very large. So, that explains how the SVM is a large margin classifier. In the next video, we'll start to talk about how to take some of these SVM ideas and start to apply them to build a complex nonlinear classifier as well.

<end>