Machine Learning:Classification:Probabilistic Generative Model

Classification

X inputs into Function and generates output.The output will be classified into different classes.Like credit scoring:Input is income,savings,profession and so on.Output is accept or refuse.Medical Diagnosis,handwritten,face recognition.

How to do Classification

Ideal Alternatives.Loss Function is:
$$L(f)=\sum _n\delta (f(x^n)\ne {\hat{y}}^n)$$
And this time we can not differential.Because it’s not continuous.

Two Classes

We all know:
$$P(C_1|X)=\frac{P(X|C_1)P(C_1)}{P(X)}$$
We need use training data to calculate the probabilities.

Generative Model

$$P(X)=P(X|C_1)P(C_1)+P(X|C_2)P(C_2)$$

Prior

$P(C_1)$ and $P(C_2)$ are Prior.

Feature

Under a class, analyze the feature distribution of the training data.

Gaussian Distribution
Like this:
f μ , ∑ ( ) = 1 ( 2 π ) D / 2 1 ∣ ∑ ∣ 1 / 2 e x p { − 1 2 ( x − μ ) T ∑ − 1 ( x − μ ) } f_{\mu,\sum}()=\frac{1}{(2\pi)^{D/2}}\frac{1}{|\sum|^{1/2}}exp\{-\frac{1}{2}(x-\mu)^T\sum^{-1}(x-\mu)\} fμ,∑​()=(2π)D/21​∣∑∣1/21​exp{−21​(x−μ)T∑−1​(x−μ)}
Function.This function’s shape determines by mean $\mu$ and covariance matrix $\sum$.$\mu$ likes the center of circles,and $\sum$ likes radius.The features of trainning data can be sampled by Gaussian Distribution.If the data is close to $\mu$,it can be easily sampled.Otherwise,it is hard.How to get $\mu$ and $\sum$?

Maximum Likelihood

The key is find Maximum Likelihood.Any sample image can use any $\mu$ and $\sum$.The Gaussian with any mean $\mu$ and covariance matrix $\sum$ can generate these points.Different likelihood.Likelihood of a Gaussian with mean $\mu$ and covariance matrix $\sum$=the probability of the Gaussian samples $x^1,x^2,...,x^{79}$.
$$L(\mu ,\sum )=f_{\mu ,\sum }(x^1)f_{\mu ,\sum }(x^2)...f_{\mu ,\sum }(x^n)$$
We assume $x^1,x^2,x^3,...,x^n$ generate from the Gaussian(${\mu }^{\ast },{\sum }^{\ast }$) with the maximum likelihood.
$${\mu }^{\ast },\sum ^{\ast }=arg\underset{\mu ,\sum }{\max }L(\mu ,\sum )$$

How to quickly calculate the parameters?

Average:
$${\mu }^{\ast }=\frac{1}{n}\sum _{i=1}^n{x}^i$$
$$\sum ^{\ast }=\frac{1}{n}\sum _{i=1}^n(x^i-{\mu }^{\ast })(x^i-{\mu }^{\ast }{)}^T$$

Now we can do classification!

If $P(C_1|X)>0.5$ ,it means that x belongs to class 1.Then we already know every class’s $\mu$ and $\sum$.
$$P(C_1|X)=\frac{P(X|C_1)P(C_1)}{P(X|C_1)P(C_1)+P(X|C_2)P(C_2)}$$
We know $P(C_1)$ and $P(C_2)$.$P(X|C_1)$ and $P(X|C_2)$ use Guassian.

Modifying Model

We can make different classes the same covariance matrix to reduce the number of the parameters.How to generate out Likelihood Function?:
$$L({\mu }^1,{\mu }^2,\sum )=f_{{\mu }^1,\sum }(x^1)...f_{{\mu }^1,\sum }(x^n)f_{{\mu }^2,\sum }(x^{n+1})...f_{{\mu }^2,\sum }(x^m)$$
How to get the same $\sum$?We assume that there are two classes, 60 and 80 in the training data.We need to combine two $\sum$.
$$\sum =\frac{60}{140}\sum ^1+\frac{80}{140}\sum ^2$$
And boundry will be linearbut not curve,called linear model.

Three Steps

Function Set(Model):
$$P(C_1|x)=\frac{P(x|C_1)P(C_1)}{P(x|C_1)P(C_1)+P(x|C_2)P(C_2)}$$
(if $P(C_1|x)>0.5$,output:class1 Otherwise,output:class2)
Then Goodness of a function:The mean $\mu$ and covariance $\sum$ that maximizing the likelihood.And find the best function.