homework2_ZhankunLuo

Zhankun Luo

PUID: 0031195279

Email: luo333@pnw.edu

Fall-2018-ECE-59500-009

Instructor: Toma Hentea

#Homework 2

Problem 2.12

%% Problem 2.12
m = [ 1 1.5; 1 1.5]; % repeat: set m = [ 1 3; 1 3];
S(:, :, 1) = 0.2 * [1 0;0 1];
S(:, :, 2) = 0.2 * [1 0;0 1];
P = [0.5 0.5]';
Loss = [0 1; 0.5 0];
N = 200;  % produce 100 feature vectors foe each class

%%  Use the classifiers designed to classify the generated 200 vectors
[X,y] = gen_gauss(m, S, P, N); figure(1); plot_data(X, y, m);
% Bayesian classifier that minimizes the error probability
z1 = bayes_classifier(m, S, P, X);
[clas_error_bayes, percent_error_1] = compute_error(y, z1)
% Bayesian classifier that minimizes the average risk with loss matrix
z2 = bayes_loss_classifier(m, S, P, X, Loss);
[clas_error_loss_bayes, percent_error_2] = compute_error(y, z2)

(a) Bayesian classifier that minimizes the error probability

bayes_classifier.m

function [z] = bayes_classifier(m, S, P, X)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% FUNCTION
%   [z]=bayes_classifier(m,S,P,X)
% Bayesian classification rule for c classes, modeled by Gaussian
% distributions (also used in Chapter 2).
%
% INPUT ARGUMENTS:
%   m:      lxc matrix, whose j-th column is the mean of the j-th class.
%   S:      lxlxc matrix, where S(:,:,j) corresponds to
%           the covariance matrix of the normal distribution of the j-th
%           class.
%   P:      c-dimensional vector, whose j-th component is the a priori
%           probability of the j-th class.
%   X:      lxN matrix, whose columns are the data vectors to be
%           classified.
%
% OUTPUT ARGUMENTS:
%   z:      N-dimensional vector, whose i-th element is the label
%           of the class where the i-th data vector is classified.
%
% (c) 2010 S. Theodoridis, A. Pikrakis, K. Koutroumbas, D. Cavouras
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
[l, c] = size(m);
[l, N] = size(X);

for i = 1:N
    for j = 1:c
        t(j) = P(j) * comp_gauss_dens_val(m(:, j), S(:, :, j),  belogX(:, i));
    end
    [num, z(i)] = max(t); % maximizes the correct probability 
end

(b) Bayesian classifier that minimizes the average risk with loss matrix

bayes_loss_classifier.m

function [z] = bayes_loss_classifier(m, S, P, X, Loss)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% FUNCTION
%   [z] = bayes_loss_classifier(m, S, P, X, Loss)
% Bayesian classification rule that minimizes the average risk with loss matrix,
% modeled by Gaussian distributions (also used in Chapter 2).
% 
% INPUT ARGUMENTS:
% Loss:  cxc matrix, L_i,j means the loss when Class_i treated as Class_j
%        then (P(x|w_j). * P)' * Loss => get loss = [loss_1 ... loss_c]
%             t(j) = (P(x|w_j). * P)' = P(j) * comp_gauss_dens_val(m(:,j),S(:,:,j),X(:,i));
%        While P(x|w_j) = comp_gauss_dens_val(m(:,j),S(:,:,j),X(:,i)) 
%                         j = 1, ..., c;  i = 1, ..., N
%        find Loss_i min => x belong to Class_i
% OUTPUT ARGUMENTS:
%    z:  N-dimensional vector, whose i-th element is the label
%        of the class where the i-th data vector is classified.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
[l, c] = size(m);
[l, N] = size(X);

for i = 1:N
    for j = 1:c
        t(j) = P(j) * comp_gauss_dens_val(m(:, j), S(:, :, j), X(:, i));
    end
    loss = t * Loss;
    [num, z(i)] = min(loss); % minimizes the average risk with loss matrix
end

result

$${\mu }_2=[1.5,1.5{]}^T$$

% Bayesian classifier that minimizes the error probability
clas_error_bayes =    44
percent_error_1 =    0.2200
% Bayesian classifier that minimizes the average risk with loss matrix
clas_error_loss_bayes =    51
percent_error_2 =    0.2550

$${\mu }_2=[3.0,3.0{]}^T$$

% Bayesian classifier that minimizes the error probability
clas_error_bayes =     0
percent_error_1 =     0
% Bayesian classifier that minimizes the average risk with loss matrix
clas_error_loss_bayes =     0
percent_error_2 =     0

conclusion

1. When centers of classes are far away, it would be no error for Bayesian classifier

2. When centers of classes are close,

   error_bayes < error_loss_bayes

   • error_bayes: Bayesian classifier that minimizes the error probability
   • error_loss_bayes: Bayesian classifier that minimizes the average risk with loss matrix

Problem 2.35

%% Problem 2.35
m = [ 1 1.5; 1 1.5]; % repeat: set m = [ 1 3; 1 3];
% m = [ 1 3; 1 3];
S(:, :, 1) = 0.2 * [1 0;0 1];
S(:, :, 2) = 0.2 * [1 0;0 1];
P = [0.5 0.5]';
% Loss = [0 1; 0.5 0]; % not used in Problem 2.35
N_train = 100; % training data: 50 for each class
N = 300;

%%  Use the classifiers designed to classify the generated 200 vectors
[X_train, y_train]= gen_gauss(m, S, P, N_train); % training data
[X, y] = gen_gauss(m, S, P, N); figure(1); plot_data(X, y, m);

% 1-NN classifier using training data of 100 vectors
z = k_nn_classifier(X_train, y_train, 1, X);
[clas_error_nn_1, percent_error_1] = compute_error(y, z)
% 3-NN classifier using training data of 100 vectors
z = k_nn_classifier(X_train, y_train, 3, X);
[clas_error_nn_3, percent_error_2] = compute_error(y, z) 
% Bayesian classifier that minimizes the error probability
z = bayes_classifier(m,S,P,X);
[clas_error_bayes, percent_error_3] = compute_error(y, z)

K-NN classifier

function z = k_nn_classifier(Z, v, k, X)

[l, N1] = size(Z); %in Z we have the trainning data
[l, N] = size(X); %in X we have the points to be classified

c = max(v); %The number of classes
%in v we have the classes to which the vectors in Z belong

%Computation of the (squared) Euclidean distance of a point in X from each
%reference vector
index2 = 1;

for i = 1:N
    dist = sum((X(:, i) * ones(1, N1) - Z).^2);
    %sorting the above distances in ascending order
    [sorted, nearest]=sort(dist); % MODE=‘ASCEND’
    %counting the class occurrences among the k-closest reference vectors
    %Z(:,i)
    refe = zeros(1, c); %Counting the reference vectors per class
    for q = 1:k
        class = v(nearest(q));
        refe(class) = refe(class) + 1;
    end
    [val,z(i)] = max(refe); % maximizes the occurrences among the k-closest reference vectors
end

result

$${\mu }_2=[1.5,1.5{]}^T$$

% 1-NN classifier using training data of 100 vectors
clas_error_nn_1 =    91
percent_error_1 =    0.3033
% 3-NN classifier using training data of 100 vectors
clas_error_nn_3 =    77
percent_error_2 =    0.2567
% Bayesian classifier that minimizes the error probability
clas_error_bayes =    71
percent_error_3 =    0.2367

$${\mu }_2=[3.0,3.0{]}^T$$

% 1-NN classifier using training data of 100 vectors
clas_error_nn_1 =     1
percent_error_1 =    0.0033
% 3-NN classifier using training data of 100 vectors
clas_error_nn_3 =     0
percent_error_2 =     0
% Bayesian classifier that minimizes the error probability
clas_error_bayes =     0
percent_error_3 =     0

conclusion

1. When centers of classes are far away, it would be no error for K-NN classifiers and Bayesian classifier

2. When centers of classes are close,

   error_1-NN > error_3-NN > error_bayes

   • error_bayes: Bayesian classifier that minimizes the error probability
   • error_1-NN: 1-NN classifier using training data of 100 vectors
   • error_3-NN: 3-NN classifier using training data of 100 vectors

Experiment

experiment 2.7

%% Experiment 2.7 
m = [ 1 4 8; 1 4 1];
S(:,:,1) = 2 * [1 0;0 1];
S(:,:,2) = 2 * [1 0;0 1];
S(:,:,3) = 2 * [1 0;0 1];
P1 = [1.0/3 1.0/3 1.0/3]';
P2 = [0.8 0.1 0.1]';
N = 1000;
%% the Bayesian, the Euclidean, and the Mahalanobis classifiers on X
% When Vector of P = [1.0/3; 1.0/3; 1.0/3]
[X1,y1] = gen_gauss(m, S, P1, N); figure(1); plot_data(X1, y1, m);
z1 = bayes_classifier(m, S, P1, X1);
[clas_error_bayes1, percent_error1] = compute_error(y1, z1)
z1 = euclidean_classifier(m, X1);
[clas_error_euclidean1, percent_error1] = compute_error(y1, z1)
% When Vector of P = [0.8; 0.1; 0.1]
[X2,y2] = gen_gauss(m, S, P2, N); figure(2); plot_data(X2, y2, m);
z2 = bayes_classifier(m, S, P2, X2);
[clas_error_bayes2, percent_error2] = compute_error(y2, z2)
z2 = euclidean_classifier(m, X2);
[clas_error_euclidean2, percent_error2] = compute_error(y2, z2)

When Vector of P = [1.0/3; 1.0/3; 1.0/3]

When Vector of P = [0.8; 0.1; 0.1]

result

% When Vector of P = [1.0/3; 1.0/3; 1.0/3]
clas_error_bayes1 =    59
percent_error1 =    0.0591

clas_error_euclidean1 =    59
percent_error1 =    0.0591

% When Vector of P = [0.8; 0.1; 0.1]
clas_error_bayes2 =    37
percent_error2 =    0.0370

clas_error_euclidean2 =    62
percent_error2 =    0.0620

conclusion

Only When

• covariance matrices S1 = S2 = S3 = kI (k>0)
• three equiprobable classes modeled by normal distributions

==> Errors of the Bayesian, the Euclidean, and the Mahalanobis classifiers Equal

When the classes are assumed to be Not equiprobable

==>Error_Bayes < Error_Euclidean