Python-scikit-learn 习题

Scikit-Learn

Assignment

In the second ML assignment you have to compare the performance of three di↵erent classification algorithms, namely Naive Bayes, SVM, and Random Forest.

For this assignment you need to generate a random binary classification problem, and then train and test (using 10-fold cross validation) the three algorithms. For some algorithms inner cross validation (5-fold) for choosing the parameters is needed. Then, show the classification performace (per-fold and averaged) in the report, and briefly discussing the results.

Steps

1. Create a classification dataset (n_samples >= 1000, n_features >= 10)
2. Split the dataset using 10-fold cross validation
3. Train the algorithms
   
   1. GaussianNB
   2. SVC (possible C values [1e-02, 1e-01, 1e00, 1e01, 1e02], RBF kernel)
   3. RandomForestClassifier (possible n estimators_values [10, 100, 1000])
4. Evaluate the cross-validated performance
   
   1. Accuracy
   2. F1-score
   3. AUC ROC
5. Write a short report summarizing the methodology and the results

Code

Step 1: Create a classification dataset (n_samples >= 1000, n_features >= 10)

from sklearn import datasets
X,y = datasets.make_classification(n_samples=1000,n_features=10,n_classes=2)

print(X)
print(y)

Step 2: Split the dataset using 10-fold cross validation

from sklearn import cross_validation
kf = cross_validation.KFold(len(X),n_folds=10,shuffle=True)
for train_index,test_index in kf:
    X_train,y_train=X[train_index],y[train_index]
    X_test,y_test=X[test_index],y[test_index]

print(X_train)
print(y_train)
print(X_test)
print(y_test)

Step 3 and Step 4: Train the algorithms(GaussianNB, SVC and RandomForestClassifier) and Evaluate the cross-validated performance(Accuracy, F1-score and AUC ROC)

GaussianNB

# Train
from sklearn.naive_bayes import GaussianNB
clf=GaussianNB()
clf.fit(X_train,y_train)
pred=clf.predict(X_test)

print(pred)
print(y_test)

# Evaluate
from sklearn import metrics
acc = metrics.accuracy_score(y_test,pred)
print(acc)
f1 = metrics.f1_score(y_test,pred)
print(f1)
auc = metrics.roc_auc_score(y_test,pred)
print(auc)

SVC

# Train
from sklearn.svm import SVC
clf = SVC(C=1e-01, kernel='rbf', gamma=0.1)
clf.fit(X_train, y_train)
pred = clf.predict(X_test)

print(pred)
print(y_test)

# Evaluate
acc = metrics.accuracy_score(y_test,pred)
print(acc)
f1 = metrics.f1_score(y_test,pred)
print(f1)
auc = metrics.roc_auc_score(y_test,pred)
print(auc)

RandomForestClassifier

# Train
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(n_estimators=6)
clf.fit(X_train, y_train)
pred = clf.predict(X_test)

print(pred)
print(y_test)

# Evaluate
acc = metrics.accuracy_score(y_test,pred)
print(acc)
f1 = metrics.f1_score(y_test,pred)
print(f1)
auc = metrics.roc_auc_score(y_test,pred)
print(auc)