CS231n 2018作业1-KNN

GitHub 地址 https://github.com/Wangxb06/CS231n

# Run some setup code for this notebook.

import random
import numpy as np
from cs231n.data_utils import load_CIFAR10
import matplotlib.pyplot as plt

from __future__ import print_function

# This is a bit of magic to make matplotlib figures appear inline in the notebook
# rather than in a new window.
%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# Some more magic so that the notebook will reload external python modules;
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

In [2]:

# Load the raw CIFAR-10 data.
cifar10_dir = 'cs231n/datasets/cifar-10-batches-py'

# Cleaning up variables to prevent loading data multiple times (which may cause memory issue)
try:
   del X_train, y_train
   del X_test, y_test
   print('Clear previously loaded data.')
except:
   pass

X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)

# As a sanity check, we print out the size of the training and test data.
print('Training data shape: ', X_train.shape)
print('Training labels shape: ', y_train.shape)
print('Test data shape: ', X_test.shape)
print('Test labels shape: ', y_test.shape)

Training data shape:  (50000, 32, 32, 3)
Training labels shape:  (50000,)
Test data shape:  (10000, 32, 32, 3)
Test labels shape:  (10000,)

In [3]:

# Visualize some examples from the dataset.
# We show a few examples of training images from each class.
classes = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']
num_classes = len(classes)
samples_per_class = 7
for y, cls in enumerate(classes):
    idxs = np.flatnonzero(y_train == y)
    idxs = np.random.choice(idxs, samples_per_class, replace=False)
    for i, idx in enumerate(idxs):
        plt_idx = i * num_classes + y + 1
        plt.subplot(samples_per_class, num_classes, plt_idx)
        plt.imshow(X_train[idx].astype('uint8'))
        plt.axis('off')
        if i == 0:
            plt.title(cls)
plt.show()

In [4]:

# Subsample the data for more efficient code execution in this exercise
num_training = 5000
mask = list(range(num_training))
X_train = X_train[mask]
y_train = y_train[mask]

num_test = 500
mask = list(range(num_test))
X_test = X_test[mask]
y_test = y_test[mask]

In [5]:

# Reshape the image data into rows
X_train = np.reshape(X_train, (X_train.shape[0], -1))
X_test = np.reshape(X_test, (X_test.shape[0], -1))
print(X_train.shape, X_test.shape)

(5000, 3072) (500, 3072)

In [6]:

from cs231n.classifiers import KNearestNeighbor

# Create a kNN classifier instance. 
# Remember that training a kNN classifier is a noop: 
# the Classifier simply remembers the data and does no further processing 
classifier = KNearestNeighbor()
classifier.train(X_train, y_train)

We would now like to classify the test data with the kNN classifier. Recall that we can break down this process into two steps:

1. First we must compute the distances between all test examples and all train examples. 
2. Given these distances, for each test example we find the k nearest examples and have them vote for the label

Lets begin with computing the distance matrix between all training and test examples. For example, if there are Ntr training examples and Nte test examples, this stage should result in a Nte x Ntr matrix where each element (i,j) is the distance between the i-th test and j-th train example.

First, open cs231n/classifiers/k_nearest_neighbor.py and implement the function compute_distances_two_loops that uses a (very inefficient) double loop over all pairs of (test, train) examples and computes the distance matrix one element at a time.

In [7]:

# Open cs231n/classifiers/k_nearest_neighbor.py and implement
# compute_distances_two_loops.

# Test your implementation:
dists = classifier.compute_distances_two_loops(X_test)
print(dists.shape)

(500, 5000)

In [8]:

# We can visualize the distance matrix: each row is a single test example and
# its distances to training examples
plt.imshow(dists, interpolation='none')
plt.show()

Inline Question #1: Notice the structured patterns in the distance matrix, where some rows or columns are visible brighter. (Note that with the default color scheme black indicates low distances while white indicates high distances.)

• What in the data is the cause behind the distinctly bright rows?
• What causes the columns?

Your Answer: fill this in.

1:如果某一行都很亮，表明这个测试数据，和所有的其他图片都差距很大，很可能是outlier； 2:说明训练数据可能有坏点；

In [9]:

# Now implement the function predict_labels and run the code below:
# We use k = 1 (which is Nearest Neighbor).
y_test_pred = classifier.predict_labels(dists, k=1)

# Compute and print the fraction of correctly predicted examples
num_correct = np.sum(y_test_pred == y_test)
accuracy = float(num_correct) / num_test
print('Got %d / %d correct => accuracy: %f' % (num_correct, num_test, accuracy))

Got 142 / 500 correct => accuracy: 0.284000

You should expect to see approximately 27% accuracy. Now lets try out a larger k, say k = 5:

In [10]:

y_test_pred = classifier.predict_labels(dists, k=5)
num_correct = np.sum(y_test_pred == y_test)
accuracy = float(num_correct) / num_test
print('Got %d / %d correct => accuracy: %f' % (num_correct, num_test, accuracy))

Got 137 / 500 correct => accuracy: 0.274000

You should expect to see a slightly better performance than with k = 1.

Inline Question 2 We can also other distance metrics such as L1 distance. The performance of a Nearest Neighbor classifier that uses L1 distance will not change if (Select all that apply.):

1. The data is preprocessed by subtracting the mean.
2. The data is preprocessed by subtracting the mean and dividing by the standard deviation.
3. The coordinate axes for the data are rotated.
4. None of the above.

Your Answer:1，2

Your explanation:1和2都是对坐标值的线性的，线性处理之前(xk+yk+zk...)最小，则坐标值线性处理之后，也最小，不影响结果。 3是坐标轴旋转，L1距离会变换，L2距离不变。[x y]T * [cos(alp) -sin(alp),sin(alp) cos(alp)] = x(cos-sin)+y(sin+cos) ,随alp的值变化,由于角度旋转矩阵的行列式为1，因此L2距离不变。

In [11]:

# Now lets speed up distance matrix computation by using partial vectorization
# with one loop. Implement the function compute_distances_one_loop and run the
# code below:
dists_one = classifier.compute_distances_one_loop(X_test)

# To ensure that our vectorized implementation is correct, we make sure that it
# agrees with the naive implementation. There are many ways to decide whether
# two matrices are similar; one of the simplest is the Frobenius norm. In case
# you haven't seen it before, the Frobenius norm of two matrices is the square
# root of the squared sum of differences of all elements; in other words, reshape
# the matrices into vectors and compute the Euclidean distance between them.
difference = np.linalg.norm(dists - dists_one, ord='fro')
print('Difference was: %f' % (difference, ))
if difference < 0.001:
    print('Good! The distance matrices are the same')
else:
    print('Uh-oh! The distance matrices are different')

Difference was: 0.000000
Good! The distance matrices are the same

In [12]:

# Now implement the fully vectorized version inside compute_distances_no_loops
# and run the code
dists_two = classifier.compute_distances_no_loops(X_test)

# check that the distance matrix agrees with the one we computed before:
difference = np.linalg.norm(dists - dists_two, ord='fro')
print('Difference was: %f' % (difference, ))
if difference < 0.001:
    print('Good! The distance matrices are the same')
else:
    print('Uh-oh! The distance matrices are different')

Difference was: 0.000000
Good! The distance matrices are the same

In [13]:

# Let's compare how fast the implementations are
def time_function(f, *args):
    """
    Call a function f with args and return the time (in seconds) that it took to execute.
    """
    import time
    tic = time.time()
    f(*args)
    toc = time.time()
    return toc - tic

two_loop_time = time_function(classifier.compute_distances_two_loops, X_test)
print('Two loop version took %f seconds' % two_loop_time)

one_loop_time = time_function(classifier.compute_distances_one_loop, X_test)
print('One loop version took %f seconds' % one_loop_time)

no_loop_time = time_function(classifier.compute_distances_no_loops, X_test)
print('No loop version took %f seconds' % no_loop_time)

# you should see significantly faster performance with the fully vectorized implementation

Two loop version took 49.666010 seconds
One loop version took 34.676075 seconds
No loop version took 0.437404 seconds

Cross-validation

We have implemented the k-Nearest Neighbor classifier but we set the value k = 5 arbitrarily. We will now determine the best value of this hyperparameter with cross-validation.

In [17]:

num_folds = 5
k_choices = [1, 3, 5, 8, 10, 12, 15, 20, 50, 100]

X_train_folds = []
y_train_folds = []
################################################################################
# TODO:                                                                        #
# Split up the training data into folds. After splitting, X_train_folds and    #
# y_train_folds should each be lists of length num_folds, where                #
# y_train_folds[i] is the label vector for the points in X_train_folds[i].     #
# Hint: Look up the numpy array_split function.                                #
################################################################################

# Your code
################################################################################

X_train_folds=np.array_split(X_train,num_folds)
y_train_folds=np.array_split(y_train,num_folds)

#                                 END OF YOUR CODE                             #
################################################################################

# A dictionary holding the accuracies for different values of k that we find
# when running cross-validation. After running cross-validation,
# k_to_accuracies[k] should be a list of length num_folds giving the different
# accuracy values that we found when using that value of k.
k_to_accuracies = {}


################################################################################
# TODO:                                                                        #
# Perform k-fold cross validation to find the best value of k. For each        #
# possible value of k, run the k-nearest-neighbor algorithm num_folds times,   #
# where in each case you use all but one of the folds as training data and the #
# last fold as a validation set. Store the accuracies for all fold and all     #
# values of k in the k_to_accuracies dictionary.                               #
################################################################################
# Your code
################################################################################
for validation_loc in range(0,num_folds):
    selectfold = range(0,validation_loc) + range(validation_loc+1,num_folds)
    for ii in selectfold:
        train = []
        if len(train) :
            train  = np.vstack((train,X_train_folds[ii]))
            label  = np.vstack((label,y_train_folds[ii]))
        else:
            train = X_train_folds[ii]
            label = y_train_folds[ii]
            
    classifier.train(train, label)
    dists = classifier.compute_distances_no_loops(X_train_folds[validation_loc])
    for k  in k_choices:
        y_test_pred = classifier.predict_labels(dists, k=k)
        num_correct = np.sum(y_test_pred == y_train_folds[validation_loc])
        accuracy = float(num_correct) / len(y_train_folds[validation_loc])
        if(k in k_to_accuracies):
            k_to_accuracies[k].append(accuracy)
        else:
            k_to_accuracies[k] = [accuracy]
#                                 END OF YOUR CODE                             #
################################################################################

# Print out the computed accuracies
bestacc = 0
best_k=1
for k in sorted(k_to_accuracies):
    for accuracy in k_to_accuracies[k]:
        if(accuracy > bestacc):
            bestacc = accuracy
            best_k = k
        print('k = %d, accuracy = %f' % (k, accuracy))
print("best_k = %d while the acc is %.3f"%(best_k,bestacc))

k = 1, accuracy = 0.235000
k = 1, accuracy = 0.237000
k = 1, accuracy = 0.258000
k = 1, accuracy = 0.242000
k = 1, accuracy = 0.227000
k = 3, accuracy = 0.232000
k = 3, accuracy = 0.235000
k = 3, accuracy = 0.259000
k = 3, accuracy = 0.219000
k = 3, accuracy = 0.222000
k = 5, accuracy = 0.246000
k = 5, accuracy = 0.263000
k = 5, accuracy = 0.262000
k = 5, accuracy = 0.251000
k = 5, accuracy = 0.240000
k = 8, accuracy = 0.252000
k = 8, accuracy = 0.273000
k = 8, accuracy = 0.264000
k = 8, accuracy = 0.242000
k = 8, accuracy = 0.235000
k = 10, accuracy = 0.264000
k = 10, accuracy = 0.272000
k = 10, accuracy = 0.267000
k = 10, accuracy = 0.255000
k = 10, accuracy = 0.230000
k = 12, accuracy = 0.259000
k = 12, accuracy = 0.269000
k = 12, accuracy = 0.264000
k = 12, accuracy = 0.254000
k = 12, accuracy = 0.247000
k = 15, accuracy = 0.262000
k = 15, accuracy = 0.268000
k = 15, accuracy = 0.253000
k = 15, accuracy = 0.252000
k = 15, accuracy = 0.235000
k = 20, accuracy = 0.260000
k = 20, accuracy = 0.253000
k = 20, accuracy = 0.252000
k = 20, accuracy = 0.242000
k = 20, accuracy = 0.246000
k = 50, accuracy = 0.247000
k = 50, accuracy = 0.255000
k = 50, accuracy = 0.240000
k = 50, accuracy = 0.249000
k = 50, accuracy = 0.240000
k = 100, accuracy = 0.243000
k = 100, accuracy = 0.239000
k = 100, accuracy = 0.240000
k = 100, accuracy = 0.230000
k = 100, accuracy = 0.233000
best_k = 8 while the acc is 0.273

In [18]:

# plot the raw observations
for k in k_choices:
    accuracies = k_to_accuracies[k]
    plt.scatter([k] * len(accuracies), accuracies)

# plot the trend line with error bars that correspond to standard deviation
accuracies_mean = np.array([np.mean(v) for k,v in sorted(k_to_accuracies.items())])
accuracies_std = np.array([np.std(v) for k,v in sorted(k_to_accuracies.items())])
plt.errorbar(k_choices, accuracies_mean, yerr=accuracies_std)
plt.title('Cross-validation on k')
plt.xlabel('k')
plt.ylabel('Cross-validation accuracy')
plt.show()

In [28]:

# Based on the cross-validation results above, choose the best value for k,   
# retrain the classifier using all the training data, and test it on the test
# data. You should be able to get above 28% accuracy on the test data.
best_k = 5
classifier = KNearestNeighbor()
classifier.train(X_train, y_train)
y_test_pred = classifier.predict(X_test, k=best_k)

# Compute and display the accuracy
num_correct = np.sum(y_test_pred == y_test)
accuracy = float(num_correct) / num_test
print('Got %d / %d correct => accuracy: %f' % (num_correct, num_test, accuracy))

Got 142 / 500 correct => accuracy: 0.284000

Inline Question 3 Which of the following statements about k-Nearest Neighbor (k-NN) are true in a classification setting, and for all k? Select all that apply.

1. The training error of a 1-NN will always be better than that of 5-NN.
2. The test error of a 1-NN will always be better than that of a 5-NN.
3. The decision boundary of the k-NN classifier is linear.
4. The time needed to classify a test example with the k-NN classifier grows with the size of the training set.
5. None of the above.

Your Answer:1 4

Your explanation: 选项1: k=1表示只有最近的点做判断的依据，因此训练没有误差，k=5的时候，根据判断/投票的规则不同，会有不一样的训练误差。 选项2: k越小，表示从数据里面获取的知识愈多，如果数据存在噪声，过拟合，则泛化能力就差，因此k=1不一定优于k=5； 选项3: 首先，Knn不是线性分类器，因为输入和输出没有线性关系，其次，knn的分界面是由很多小的线性空间组成，分界面局部是线性的。 选项4: 搜索的量增大，