SVM from another perspective

This article records my process of study.Link:SVM.There are not Lagrange and KKT condition.Here,we mainly use gradient descent.

Binary Classification

Because $g(x)$ only outputs $+1$ or $-1$.Thus $\delta$ can’t use gradient descent.We use another loss function.(PS: $\delta$ is a piecewise function).${\hat{y}}^n$ is 1 or -1,it just is defined for convenience. This can unify two classes express about loss function by ${\hat{y}}^n\ast f(x)$. For ${\hat{y}}^n=1$,we hope bigger $f(x)$.For ${\hat{y}}^n=-1$,we hope smaller $f(x)$.$f(x)=0$ is bound between tow classes.

For intuition(Suppose $\delta$ or $l$ is y axis):
We use square loss,it is unreasonable.Because when ${\hat{y}}^{n}f(x)$ is big,the loss is big.

PS:I think express above is not meaningful($f(x)$ close to 1 or -1).Next,we use $sigmoid$ + square loss.The curve is blue.

Refer to 简单谈谈Cross Entropy Loss About
$Softmax$ and $CrossEntropy$.Next,we use $sigmoid$ + $CrossEntropy$.It is reasonable.And if $l$ is divided by $ln2$,this method result is upper-bound of ideal loss,we minimize the $l$ to minimize ideal loss. In contrast with $sigmoid+Squareloss$, gradient descent in this method represent better. $sigmoid+Squareloss$ don’t operate better in ${\hat{y}}^{n}f(x)$ being tiny positive number.The $CrossEntropy$ is easier to train. So in some regressions we often use it.

Next,we give $hingeloss$.Sometimes the $hingeloss$ has stronger robustness than $crossentropy$.For example,there are outliers.Because it has sparsity for training data and only considers support vectors(In kernel section,we can understand it). While $CrossEntropy$ considers all training data.

Why do we use 1?Because you can think that the hinge function is upper-bound of ideal loss.

Linear SVM

We give linear SVM model.The $l$ and normalization are convex function. So we can use gradient descent to minimize hinge loss function.Thus we can use SVM as deep learning classifier layer.There is a reference.

SVM gradient descent

The $c^n(w)=+1,-1,or0$.The $x_i^n$ is real number which is a dimension of feature vector.

Linear SVM another formulation

Suppose training data set is linear separable.Express above is different with that we know.We transform hinge loss function to that we know.With using hinge loss function,we are going to compute soft margin SVM.For hard margin,you can see my another article which is supplementary for Z.H Zhou Watermelon Book SVM part.The $max$ is ${ϵ}^n$ here.The variant is following:
$$\begin{gathered} MinimizingtheLossfunction \\ L(f)=\sum _n{ϵ}^n+\lambda ||w|{|}_2 \\ s.t{ϵ}^n>=0 \\ \widehat{y^n}f(x^n)>=1-{ϵ}^n \end{gathered}$$
You can clearly think express above. In fact,it is similar with Watermelon Book by ZhiHua Zhou.Here,I think we should add a hyper-parameter in front of ${\sum }_n{ϵ}^n$.When the hyper-parameter is infinite,we hope get a hard margin SVM.
$$\begin{gathered} \underset{w,b}{\min }\frac{1}{2}||w|{|}^2 \\ s.t.y_i(w^T{x}_i+b)>=1,i=1,2,...m. \\ wherewisweights. \end{gathered}$$
In common,we find smallest length of weights that satisfies $y^{n}f(x^n)>=1$ as possible.If the training data set is linear separable,we can get ${ϵ}^n=0$.For example,you can scale the $w$ and $b$ to make minimum interval $1$.So,we need minimize $||w|{|}^2$.Between difference of two express is type of margin.
In fact,it is hard to make data linear separable.For relaxing this problem,we use soft margin which allows to wrongly classifier a few data samplers.

Why is the variant equivalent to proposed express before?We want to minimize $L(f)$,i.e.,$ϵ$.When $ϵ$ is minimized,this quadratic programming problem is equivalent to minimize hinge loss ${ϵ}^n=max(0,1-\widehat{y^n}f(x^n))$.$ϵ$ can’t be very large.${ϵ}^n$ is the smallest number but bigger than $0$ and $1-\widehat{y^n}f(x^n)$ .So with minimization,the two formulations are equal.The ${ϵ}^n$ is a slack variable.

Dual representation

Many SVM slides interpret express below by Lagrange.Now,we interpret why $\hat{w}$ is a linear combination of $x^n$ from another perspective.
$$\hat{w}=\sum _n{\hat{a}}_n\ast x^n$$
Where ${\hat{a}}_n$ may be sparse. It is a integer.
If we initialize $w$ 0:
$$\begin{gathered} w=w-\eta \sum _n{c}^n(w)x^n \\ {c}^n(x)=\frac{\partial l(f(x^n),{\hat{y}}^n)}{\partial f(x^n)}=0or-{\hat{y}}^n(+1or-1) \end{gathered}$$
What the $\hat{w}$ is linear combination of $x^n$ is obvious.The hinge loss function usually gets zero. So many $x$ is not used to determine $\hat{w}$.These have non-zero $\hat{a}$ are defined support vectors.For logistic regression,it is always non-zero.Because it uses $Entropy$.For $Entropy$ loss function,there is not zero for gradient. So if we use $Entropy$ loss,we can’t get sparse $\hat{a}$.Every data can influence the result($\hat{w}$).As mentioned above,hinge loss has stronger robustness.

Step 1

Now,we can get new formulation for our $f(x)$.Suppose $x$ is in linear space.
$$\begin{gathered} duetow=\sum _n{a}_n{x}^n=[x^1,x^2...x^n].dot(\left[\begin{array}{c}{a}_1\\ a_2\\ ...\\ a_n\end{array}\right])=Xa \\ f(x)=w^{T}x=a^T{X}^{T}x=[a_1,a_2,...a_n]\left[\begin{array}{c}(x^1{)}^T\\ (x^2{)}^T\\ ...\\ (x^n{)}^T\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ ...\\ x_m\end{array}\right] \\ Finally \\ f(x)=\sum _n{a}_n((x^n{)}^T.dot(x))=\sum _n{a}_{n}K((x^n{)}^T,x) \end{gathered}$$
Where $x^n$ is a column vector,the $K$ is defined as kernel function.Now,we get a new model,and we want to get $a_{1-n}$.Because our x isn’t usually in linear space,we use function $K$ to replace inner product in linear space.

Step 2、3

We find the best $a_{1-n}$ to minimize loss function(PS:here,YouTuber doesn’t give update method.But I think it can use gradient descent and QP packages).We use new model $f(x)$ to replace origin $f(x)$ in loss function.
$$L(f)=\sum _{n}l(\sum _{n^{\prime }}{a}_{n^{\prime }}K((x^{n^{\prime }}{)}^T,x^n),{\hat{y}}^n)$$
We need not to know specific $x^n$,and we only need know the $K$.We use kernel trick to solve it.For kernel trick,you can use wherever it is effective,e.g,logistic regression,linear regression.

Kernel trick

Why do we define kernel function,rather than using directly inner product between two vectors? Because it makes computation effective and efficient for inner product.We can define different kernel function to easily compute inner product.How to release efficient computation?If we don’t use kernel function.For using linear model,We need to compute feature transformation.For example,in neural network we add hidden layer to compute feature transformation.The feature is a high dimension vector(denoted by $\varphi$). Then we compute inner product. It is expensive.Conversely,we directly compute inner product of x and z by defining kernel concept. Tow ways are equivalent. As following:

$\varphi$ is feature vector of x or z.
For example:

Radial Basis Function Kernel

Another kernel function(we use Taylor Expansion to expand exponent computation):

What you should pay attention to is that it may be over fit due to infinite dimension. In kernel trick section,we use a square expression. Its feature dimension is finite(at most 4).

Sigmoid Kernel

$$K(x,z)=tanh(x.dot(z))$$
You can use similar Taylor Expansion to find tow high dimension vectors（$\varphi (x),\varphi (z)$） whose inner product is equal to $K(x,z)$.
If we use sigmoid kernel,we get a neural network with a hidden layer.

Thinking about inner product computation,when tow data points are very closed,the value of $K$ is very big. So we think $K$ is something like similarity between two data points.We can only consider value of $K$.You can define your kernel function,then using Mercer’s theory to check whether this kernel function is equal to inner product of tow high dimension vectors. Here,we give the reference.

The kernel function is hyper-parameter. It can influence effectiveness of model.Some kernel function can’t make data separate.For different task,we choose different kernel function.If you don’t know,you can choose RBF kernel.For text data,we usually use linear kernel($k(x_i,x_j)=x_i^T{x}_j$).

SVM related methods

Our SVM can’t conduct multi-classifier task. It need to be extended. In contrast with SVM,regression need more data.

Relationship with deep learning

Hidden layer is to transform feature.Our kernel function is to change data to a high dimension space,too. In high dimension space, data is linearly separable.Then we use hinge loss to get linear classifier.The kernel function is learnable.The below paper is reference.