Machine Learning Project 2 Part B

Machine Learning Project 2 Part B

Our strategy will attempt to classify a test shape based on naive Bayes classi cation. Based on our observation of section 1, we assume that we are able to build pairwise classi ers reliable enough. Our problem is then to decide which class a test shape should belong to given the class cationresults of all possible pairs.

Question A1

We denote by $\omega$ the set of all classes. Suppose that for any two classes $\omega_i$ $\omega_j$, we have trained a classi er$C_{ij}$ , that is, we know the conditional laws $Pr[X|w_i;C_{ij}]$and $Pr[X|w_j,C_{ij} ]$ (notice the conditional to $Cij$ , which reflects these laws are the opinion of that specif c classifier, and may perfectly by dierent for another. We remind that Bayesian classi cation assigns class to a test pattern X according to

$$\omega^* = arg (max_w Pr[w|X])$$

1. The classi er $C_{ij}$ must express its decision by returning $Pr[w_i|X,C_{ij}]$. Using Bayes rule, express this probability as a function of $Pr[X|w_i,C_{ij}]$and $Pr[X|w_j,C_{ij}]$. According to Bayeslaw, what is$Pr[w_j|X,C_{ij}]$ anyway ?

1. Answer

---

$$Pr[\omega_i|X,C_{ij}]={{Pr[\omega,X,C_{ij}]}\over {Pr[X,C_{ij}]}} ------[1]$$
$$Pr[X|\omega_i,C_{ij}] ={Pr[{X, \omega_i, C_{ij}}]\over Pr[{\omega_i,C_{ij}}]}-------[2]$$

Based on formula [1] and[2],

$$Pr[\omega_i|X,C_{ij}]={{{Pr[X|\omega_i,C_{ij}] * Pr[{\omega_i,C_{ij}}]}}\over{P[X,C_{ij}]}}$$

---

2.Still using Bayes rule, express $Pr[\omega_i|X]$ assuming equal priors for all classi fiers.

2. Answer

---

$$Pr[\omega_i|X]=\sum_{n=0}^m Pr[\omega_i|X,C_{ij,n}]=\sum_{n=0}^m{{{Pr[X|\omega_i,C_{ijn}] * Pr[{\omega_i,C_{ijn}}]}}\over {P[X,C_{ijn}]} }$$
3. Explain why, despite theoretically correct, the values of $Pr[\omega_i|X]$may lead to a very unreliable decision once plugged into (1) { hint : look at what happens when a shape in neither i in j.

3. Answer

In this situation the item of $Pr[X|\omega_i,C_{ijn}]$ which is:

$$Pr[X|\omega_i] = {number(X~in~w_i) \over number(w_i)} = 0$$

which will lead that

$$Pr[\omega_i|X]=0$$
class i nor j.

4.Propose a remedy to this issue, possibly violating the assumption of constant and equal priors for all classi ers (hint: you may consider that your classi er should not return one, but several probabilities). Experiment it, and show it does ectively correct the problem.

4. Answer

A method to solve this problem is to use a threshold value $\gamma$ to check the probabilities weather pass this threshold $\gamma$. If any of the probabilities pass, this X must belong among the set of classes , and the $\omega_i$ which maximum $Pr[\omega_i|X,C_{ij}]$ is the class that X belong to, else if no probabilities pass the threshold, the X is not belong to any of the classes.

---

Question A2

Choose 5 classes (visually quite dierent), and train all pairs of classi ers using autosvm. Store the results in a matrix.

Answer 1

cls_mat =

10x1 struct array with fields:

SupportVectors
Alpha
Bias
KernelFunction
KernelFunctionArgs
GroupNames
SupportVectorIndices
ScaleData
FigureHandles

In the sequel, you will assume that the euclidean distances of samples to the separating hyper- plane are normally distributed, and use the relevant distributions for probabilities $Pr[X|\omega_i;C_{ij }]$and $Pr[X|\omega_j;C_{ij }]$. Do the margin, or these distances, have a probabilistic meaning anyway?

Answer 2

Here the meaning of ‘the euclidean distances of samples to the separating hyper- plane’ means that the ‘good degree’ of the correspond classifier $C_{ij}$.
And considering the ‘good degree‘satisfied normal distribution, we get,

$$P[X,C_{ijn}]=N(\mu,\sigma^2)$$
This probability can be estimated by statistic method (ML etc.) in our example.
As we known before,
$$Pr[X|\omega_i,C_{ij}] ={Pr[{X, \omega_i, C_{ij}}]\over Pr[{\omega_i,C_{ij}}]}$$

$$Pr[\omega_i|X,C_{ij}]={{{Pr[X|\omega_i,C_{ij}] * Pr[{\omega_i,C_{ij}}]}}\over{P[X,C_{ij}]}}$$

Then the best classifier is the classifier make the formula below stand,

$$max~{Pr[C_{ij}|(\omega_i,X)]}-------[3]$$
with,
$${Pr[C_{ij}|(\omega_i,X)]}={Pr[{X, \omega_i, C_{ij}}]\over Pr[{\omega_i,X}]}={{Pr[{X,C_{ij}}]*Pr[\omega_i]}\over{Pr[{\omega_i,X}]}}------[4]$$

Question A3

Using the fi nal classifi cation methodology of question A2, classify 1 shape amongst the remaining 25% of data for each class, and report final posterior probabilities. Choose an “outlier” shape from another class, and compare its posterior again. Comment your results.

Answer

Based on formula [3] and [4] in Question A2, and the classifiers’ matrix ‘cls_mat ’ in answer 1 of Question 2. We can get the calculation relationship below,

$$max~{Pr[C_{ij}|(\omega_i,X)]}$$ $$=max{Pr[{X, \omega_i, C_{ij}}]\over Pr[{\omega_i,X}]}$$ $$=max{{Pr[{X,C_{ij}}]*Pr[\omega_i]}\over{Pr[{\omega_i,X}]}}$$

$[X,C_{ijn}]$~$N(\mu,\sigma^2)$ which can be estimated by ML algorithm.

$Pr[\omega_i]$ = ${number~of~samples~of~class~i}\over{number~of~samples ~in~all~classes}$

${Pr[{\omega_i,X}]}= {{number~of~sample~X~inside~class~ \omega_i}\over{number of sample~ \omega_i}}$