第十五周作业

Assignment of Sklearn

Task

In the second ML assignment you have to compare the performance of three different classification algorithms, namely Naive Bayes, SVM, and Random Forest.

For this assignment you need to generate a random binary classification problem, and then train and test (using 10-fold cross validation) the three algorithms. For some algorithms inner cross validation (5-fold) for choosing the parameters is needed. Then, show the classification performace (per-fold and averaged) in the report, and briefly discussing the results.

• Create a classification dataset (n_samples $\geq 1000$ , n_features $\geq 10$ )
• Split the dataset using 10-fold cross validation
• Train the algorithms
• • GaussianNB
• • SVC (possible C values [1e-02, 1e-01, 1e00, 1e01, 1e02], RBF kernel)
• • RandomForestClassifier (possible n estimators values [10, 100, 1000])
• Evaluate the cross-validated performance
• • Accuracy
• • F1-score
• • AUC ROC
• Write a short report summarizing the methodology and the results

Step 1

Create a classification dataset (n_samples $\geq 1000$ , n_features $\geq 10$ )

from sklearn import datasets

iris = datasets.load_digits(2)
dataset = datasets.make_classification(n_samples=1000, n_features=10, n_informative=2, n_redundant=2, n_repeated=0, n_classes=2)
print(iris.DESCR)
print(iris.data)
print(iris.target)

Optical Recognition of Handwritten Digits Data Set
===================================================

Notes
-----
Data Set Characteristics:
    :Number of Instances: 5620
    :Number of Attributes: 64
    :Attribute Information: 8x8 image of integer pixels in the range 0..16.
    :Missing Attribute Values: None
    :Creator: E. Alpaydin (alpaydin '@' boun.edu.tr)
    :Date: July; 1998

This is a copy of the test set of the UCI ML hand-written digits datasets
http://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits

The data set contains images of hand-written digits: 10 classes where
each class refers to a digit.

Preprocessing programs made available by NIST were used to extract
normalized bitmaps of handwritten digits from a preprinted form. From a
total of 43 people, 30 contributed to the training set and different 13
to the test set. 32x32 bitmaps are divided into nonoverlapping blocks of
4x4 and the number of on pixels are counted in each block. This generates
an input matrix of 8x8 where each element is an integer in the range
0..16. This reduces dimensionality and gives invariance to small
distortions.

For info on NIST preprocessing routines, see M. D. Garris, J. L. Blue, G.
T. Candela, D. L. Dimmick, J. Geist, P. J. Grother, S. A. Janet, and C.
L. Wilson, NIST Form-Based Handprint Recognition System, NISTIR 5469,
1994.

References
----------
  - C. Kaynak (1995) Methods of Combining Multiple Classifiers and Their
    Applications to Handwritten Digit Recognition, MSc Thesis, Institute of
    Graduate Studies in Science and Engineering, Bogazici University.
  - E. Alpaydin, C. Kaynak (1998) Cascading Classifiers, Kybernetika.
  - Ken Tang and Ponnuthurai N. Suganthan and Xi Yao and A. Kai Qin.
    Linear dimensionalityreduction using relevance weighted LDA. School of
    Electrical and Electronic Engineering Nanyang Technological University.
    2005.
  - Claudio Gentile. A New Approximate Maximal Margin Classification
    Algorithm. NIPS. 2000.

[[ 0.  0.  5. ...  0.  0.  0.]
 [ 0.  0.  0. ... 10.  0.  0.]
 [ 0.  0.  1. ...  3.  0.  0.]
 ...
 [ 0.  0.  5. ...  8.  1.  0.]
 [ 0.  0.  6. ...  4.  0.  0.]
 [ 0.  0.  6. ...  6.  0.  0.]]
[0 1 0 1 0 1 0 0 1 1 0 0 0 1 1 0 0 0 1 1 1 1 1 0 1 0 0 1 0 1 0 1 0 0 1 1 0
 0 0 1 1 0 0 0 1 1 1 1 1 0 1 0 0 1 0 1 0 1 0 0 1 1 0 0 0 1 1 0 0 0 1 1 1 1
 1 0 1 0 0 1 0 1 0 1 0 0 1 1 0 0 0 1 1 0 0 0 1 1 1 1 1 0 1 0 0 1 0 1 0 1 0
 0 1 1 0 0 0 1 1 0 0 0 1 1 1 1 1 0 1 0 0 1 0 1 0 1 0 0 1 1 0 0 0 1 1 0 0 0
 1 1 1 1 1 0 1 0 0 1 0 1 0 1 0 0 1 1 0 0 0 1 1 0 0 0 1 1 1 1 1 0 1 0 1 0 1
 0 1 0 0 1 1 0 1 1 0 0 0 1 1 1 1 1 0 1 0 0 1 0 1 0 1 0 0 1 1 0 0 0 1 1 0 0
 0 1 1 1 1 1 0 1 0 0 1 0 1 0 1 0 0 1 1 0 0 0 1 1 0 0 0 1 1 1 1 1 0 1 0 0 1
 0 1 0 1 0 0 1 1 0 0 0 1 1 0 0 0 1 1 1 1 1 0 1 0 0 1 0 1 0 1 0 0 1 1 0 0 0
 1 1 0 0 0 1 1 1 1 1 0 1 0 0 1 0 1 0 1 0 0 1 1 0 0 0 1 1 0 0 1 1 1 1 1 0 1
 0 0 1 0 1 0 1 0 0 1 1 0 0 0 1 1 0 0 0 1 1 1 1 1 0 1 0]

Step 2

Split the dataset using 10-fold cross validation

from sklearn import cross_validation

kf = cross_validation.KFold(len(iris.data), n_folds=10, shuffle=True)
for train_index, test_index in kf:
    X_train, y_train = iris.data[train_index], iris.target[train_index]
    X_test, y_test = iris.data[test_index], iris.target[test_index]

print(X_train)
print(X_test)
print(y_train)
print(y_test)

[[ 0.  0.  5. ...  0.  0.  0.]
 [ 0.  0.  0. ... 10.  0.  0.]
 [ 0.  0.  1. ...  3.  0.  0.]
 ...
 [ 0.  0.  5. ...  8.  1.  0.]
 [ 0.  0.  6. ...  4.  0.  0.]
 [ 0.  0.  6. ...  6.  0.  0.]]
[[ 0.  0.  0. ... 12.  0.  0.]
 [ 0.  0.  5. ...  2.  0.  0.]
 [ 0.  0.  5. ...  1.  0.  0.]
 ...
 [ 0.  0.  4. ...  2.  0.  0.]
 [ 0.  0.  1. ...  9.  0.  0.]
 [ 0.  0.  9. ...  4.  0.  0.]]
[0 1 0 1 0 1 0 0 1 1 0 0 0 1 0 0 0 1 1 1 1 1 0 1 0 0 1 0 1 0 1 1 1 0 0 0 1
 0 1 1 1 1 1 0 1 0 0 1 0 1 0 1 0 0 1 0 0 1 1 0 0 1 1 1 1 1 0 0 1 0 1 0 1 0
 0 0 1 1 0 0 0 1 1 1 1 0 1 0 0 1 0 1 0 1 0 0 1 1 0 0 0 1 1 0 0 0 1 1 1 1 1
 0 1 0 1 0 0 1 0 0 1 1 0 0 1 1 0 0 0 1 1 1 1 0 1 0 0 1 0 1 0 1 0 0 1 1 0 0
 0 1 1 0 0 0 1 1 1 1 1 0 1 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 0 1 1 1 1 1 0 1 0
 0 1 0 1 0 1 0 0 1 1 0 0 0 1 1 0 0 0 1 1 1 1 1 0 1 0 0 1 1 1 1 0 0 0 1 1 0
 0 0 1 1 1 1 1 0 1 0 0 1 0 1 0 1 0 0 1 1 0 0 0 1 0 0 1 1 1 1 1 1 0 0 1 0 1
 0 1 0 0 1 1 0 0 0 1 1 0 0 0 1 1 1 1 1 1 0 0 1 1 0 1 0 0 1 1 0 0 0 1 0 1 1
 1 1 1 0 1 0 1 0 1 0 1 0 1 0 0 0 1 1 0 0 1 1 1 1 1 0 1 0]
[1 0 0 1 0 0 1 0 0 0 1 1 0 0 1 1 0 1 0 1 0 1 0 0 0 1 0 0 0 0 1 0 0 0 1 0]

Step 3 + Step 4 Combined

Train the algorithms and evaluate the cross-validated performance

GaussianNB

from sklearn import datasets
from sklearn import cross_validation
from sklearn.naive_bayes import GaussianNB
from sklearn import metrics

iris = datasets.load_digits(2)

dataset = datasets.make_classification(n_samples=1000, n_features=10, n_informative=2, n_redundant=2, n_repeated=0, n_classes=2)

kf = cross_validation.KFold(len(iris.data), n_folds=10, shuffle=True)
for train_index, test_index in kf:
    X_train, y_train = iris.data[train_index], iris.target[train_index]
    X_test, y_test = iris.data[test_index], iris.target[test_index]

clf = GaussianNB()
clf.fit(X_train, y_train)
pred = clf.predict(X_test)
acc = metrics.accuracy_score(y_test, pred)
f1 = metrics.f1_score(y_test, pred)
auc = metrics.roc_auc_score(y_test, pred)

print('--- Naive Bayes ---')
print('Predict: ', pred)
print('Exact: ', y_test)
print('-------------------')
print('Accuracy: ', acc)
print('F1-score: ', f1)
print('AUC ROC: ', auc)

--- Naive Bayes ---
Predict:  [1 1 0 1 1 0 1 1 0 0 1 0 0 1 1 0 1 0 1 1 0 0 1 1 0 1 0 0 0 1 0 1 1 0 1 0]
Exact:  [1 1 0 1 1 0 1 1 0 0 1 0 0 1 1 0 1 0 1 1 0 0 1 1 0 1 0 0 0 1 0 1 1 0 1 0]
-------------------
Accuracy:  1.0
F1-score:  1.0
AUC ROC:  1.0

As we can see, the Accuracy, F1-score and AUC-ROC are all equal to 1.0.

It means that the predict values are all correct!

SVC

• C values 1e-02

from sklearn import datasets
from sklearn import cross_validation
from sklearn.svm import SVC
from sklearn import metrics

iris = datasets.load_digits(2)

dataset = datasets.make_classification(n_samples=1000, n_features=10, n_informative=2, n_redundant=2, n_repeated=0, n_classes=2)

kf = cross_validation.KFold(len(iris.data), n_folds=10, shuffle=True)
for train_index, test_index in kf:
    X_train, y_train = iris.data[train_index], iris.target[train_index]
    X_test, y_test = iris.data[test_index], iris.target[test_index]

clf = SVC(C=1e-02, kernel='rbf', gamma=0.1)
clf.fit(X_train, y_train)
pred = clf.predict(X_test)
acc = metrics.accuracy_score(y_test, pred)
f1 = metrics.f1_score(y_test, pred)
auc = metrics.roc_auc_score(y_test, pred)

print('--- SVM ---')
print('Predict: ', pred)
print('Exact: ', y_test)
print('-------------------')
print('Accuracy: ', acc)
print('F1-score: ', f1)
print('AUC ROC: ', auc)

--- SVM ---
Predict:  [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
Exact:  [1 0 0 0 1 1 0 0 0 0 1 0 0 1 0 1 0 1 0 0 1 1 1 0 1 1 1 0 1 1 0 0 1 0 0 0]
-------------------
Accuracy:  0.4444444444444444
F1-score:  0.6153846153846153
AUC ROC:  0.5

• C values 1e-01

from sklearn import datasets
from sklearn import cross_validation
from sklearn.svm import SVC
from sklearn import metrics

iris = datasets.load_digits(2)

dataset = datasets.make_classification(n_samples=1000, n_features=10, n_informative=2, n_redundant=2, n_repeated=0, n_classes=2)

kf = cross_validation.KFold(len(iris.data), n_folds=10, shuffle=True)
for train_index, test_index in kf:
    X_train, y_train = iris.data[train_index], iris.target[train_index]
    X_test, y_test = iris.data[test_index], iris.target[test_index]

clf = SVC(C=1e-01, kernel='rbf', gamma=0.1)
clf.fit(X_train, y_train)
pred = clf.predict(X_test)
acc = metrics.accuracy_score(y_test, pred)
f1 = metrics.f1_score(y_test, pred)
auc = metrics.roc_auc_score(y_test, pred)

print('--- SVM ---')
print('Predict: ', pred)
print('Exact: ', y_test)
print('-------------------')
print('Accuracy: ', acc)
print('F1-score: ', f1)
print('AUC ROC: ', auc)

--- SVM ---
Predict:  [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
Exact:  [1 1 0 0 0 1 0 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 1 1 0 1 0 1 1 0 0 0 0 1 1 1]
-------------------
Accuracy:  0.4722222222222222
F1-score:  0.6415094339622641
AUC ROC:  0.5

• C values 1e00

from sklearn import datasets
from sklearn import cross_validation
from sklearn.svm import SVC
from sklearn import metrics

iris = datasets.load_digits(2)

dataset = datasets.make_classification(n_samples=1000, n_features=10, n_informative=2, n_redundant=2, n_repeated=0, n_classes=2)

kf = cross_validation.KFold(len(iris.data), n_folds=10, shuffle=True)
for train_index, test_index in kf:
    X_train, y_train = iris.data[train_index], iris.target[train_index]
    X_test, y_test = iris.data[test_index], iris.target[test_index]

clf = SVC(C=1e00, kernel='rbf', gamma=0.1)
clf.fit(X_train, y_train)
pred = clf.predict(X_test)
acc = metrics.accuracy_score(y_test, pred)
f1 = metrics.f1_score(y_test, pred)
auc = metrics.roc_auc_score(y_test, pred)

print('--- SVM ---')
print('Predict: ', pred)
print('Exact: ', y_test)
print('-------------------')
print('Accuracy: ', acc)
print('F1-score: ', f1)
print('AUC ROC: ', auc)

--- SVM ---
Predict:  [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
Exact:  [0 0 1 1 0 1 0 1 0 1 1 1 1 0 0 0 1 1 1 0 0 1 1 1 0 1 0 0 1 0 0 0 0 0 1 0]
-------------------
Accuracy:  0.4722222222222222
F1-score:  0.6415094339622641
AUC ROC:  0.5

• C values 1e01

from sklearn import datasets
from sklearn import cross_validation
from sklearn.svm import SVC
from sklearn import metrics

iris = datasets.load_digits(2)

dataset = datasets.make_classification(n_samples=1000, n_features=10, n_informative=2, n_redundant=2, n_repeated=0, n_classes=2)

kf = cross_validation.KFold(len(iris.data), n_folds=10, shuffle=True)
for train_index, test_index in kf:
    X_train, y_train = iris.data[train_index], iris.target[train_index]
    X_test, y_test = iris.data[test_index], iris.target[test_index]

clf = SVC(C=1e01, kernel='rbf', gamma=0.1)
clf.fit(X_train, y_train)
pred = clf.predict(X_test)
acc = metrics.accuracy_score(y_test, pred)
f1 = metrics.f1_score(y_test, pred)
auc = metrics.roc_auc_score(y_test, pred)

print('--- SVM ---')
print('Predict: ', pred)
print('Exact: ', y_test)
print('-------------------')
print('Accuracy: ', acc)
print('F1-score: ', f1)
print('AUC ROC: ', auc)

--- SVM ---
Predict:  [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
Exact:  [1 0 0 0 1 1 1 0 1 1 0 0 1 0 0 1 1 1 0 0 0 1 1 1 0 0 1 0 0 1 1 1 0 1 1 0]
-------------------
Accuracy:  0.5277777777777778
F1-score:  0.6909090909090909
AUC ROC:  0.5

• C values 1e02

from sklearn import datasets
from sklearn import cross_validation
from sklearn.svm import SVC
from sklearn import metrics

iris = datasets.load_digits(2)

dataset = datasets.make_classification(n_samples=1000, n_features=10, n_informative=2, n_redundant=2, n_repeated=0, n_classes=2)

kf = cross_validation.KFold(len(iris.data), n_folds=10, shuffle=True)
for train_index, test_index in kf:
    X_train, y_train = iris.data[train_index], iris.target[train_index]
    X_test, y_test = iris.data[test_index], iris.target[test_index]

clf = SVC(C=1e02, kernel='rbf', gamma=0.1)
clf.fit(X_train, y_train)
pred = clf.predict(X_test)
acc = metrics.accuracy_score(y_test, pred)
f1 = metrics.f1_score(y_test, pred)
auc = metrics.roc_auc_score(y_test, pred)

print('--- SVM ---')
print('Predict: ', pred)
print('Exact: ', y_test)
print('-------------------')
print('Accuracy: ', acc)
print('F1-score: ', f1)
print('AUC ROC: ', auc)

--- SVM ---
Predict:  [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
Exact:  [0 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 0 0 1 0 0 1 1 1 1 0 1 0 0 1 0 0 1 1 0 1]
-------------------
Accuracy:  0.4722222222222222
F1-score:  0.6415094339622641
AUC ROC:  0.5

As we can see, in the 5 cases above, the predict values are all ‘1’ s, which don’t work at all.

SVM algorithm runs very bad in above test cases.

RandomForest

from sklearn import datasets
from sklearn import cross_validation
from sklearn.ensemble import RandomForestClassifier
from sklearn import metrics

iris = datasets.load_digits(2)

dataset = datasets.make_classification(n_samples=1000, n_features=10, n_informative=2, n_redundant=2, n_repeated=0, n_classes=2)

kf = cross_validation.KFold(len(iris.data), n_folds=10, shuffle=True)
for train_index, test_index in kf:
    X_train, y_train = iris.data[train_index], iris.target[train_index]
    X_test, y_test = iris.data[test_index], iris.target[test_index]

clf = RandomForestClassifier(n_estimators=6)
clf.fit(X_train, y_train)
pred = clf.predict(X_test)
acc = metrics.accuracy_score(y_test, pred)
f1 = metrics.f1_score(y_test, pred)
auc = metrics.roc_auc_score(y_test, pred)

print('--- Random forest ---')
print('Predict: ', pred)
print('Exact: ', y_test)
print('-------------------')
print('Accuracy: ', acc)
print('F1-score: ', f1)
print('AUC ROC: ', auc)

--- Random forest ---
Predict:  [1 1 1 0 0 0 1 0 1 0 1 1 1 1 1 0 0 1 0 0 1 1 0 0 1 0 1 0 0 0 1 0 0 0 1 1]
Exact:  [1 1 1 0 0 0 1 0 1 0 1 1 1 1 1 0 0 1 0 0 1 1 0 0 1 0 1 0 0 0 1 0 0 0 1 1]
-------------------
Accuracy:  1.0
F1-score:  1.0
AUC ROC:  1.0

As we can see, the Accuracy, F1-score and AUC-ROC are all equal to 1.0.

It means that the predict values are all correct!

Summary

GaussianNB and RandomForest get all correct in the above test cases.

SVC does not work well.