cs231n assignment(一) knn分类器

cs231n assignment(一) knn分类器

这是我自己对所学内容的总结与思考，如有错误，欢迎大家指正，若有侵权，烦请告知

一、预备知识

1. KNN算法的核心思想是

1）计算已知类别中数据集的点与当前点的距离。
2）按照距离递增次序排序。
3）选取与当前点距离最小的k个点。
4）确定前k个点所在类别的出现频率。
5）返回前k个点中出现频率最高的类别作为当前点的预测分类。

2. 评价两幅图像之间距离的指标：
   
   
   如上图所示，L1评价指标对应的是一个菱形，菱形上的任意一点在L1上与原点等距，而L2对应的是一个圆形，圆形上的任意一点在L2上与原点等距。如果转动坐标系，将会改变点之间的L1距离，而L2距离仍然不变。因此，如果输入的特征向量、向量中的一些值对任务有一些重要的意义，那么也许L1更合适，但如果它只是空间中的一个通用向量，可能L2更好一些。
   事实上，度量方式不止这两种，我们可以通过改变距离度量方式来使KNN算法适应不同的数据类型，如文本、图像等。
   在本次实验中，我们使用L2欧氏距离来度量，并且用二层循环、一层循环和无循环三种方式来实现。下面我们重点看一下这部分的原理和关键代码

二层循环：直接按照公式进行

for i in range(num_test):
    for j in range(num_train):
          dists[i, j] = np.sqrt(np.sum((X[i, :] - self.X_train[j, :]) ** 2))

一层循环：和二层循环类似，使用numpy的广播机制同时计算一个测试数据和所有训练数据之间的距离

for i in range(num_test):
    dists[i, :] = np.sqrt(np.sum(np.square(self.X_train - X[i, :]), axis=1))

numpy广播机制的原理图如下所示，想了解更多的同学可以参见[numpy广播机制]

无循环：原理图如下所示

test_sum = np.sum(np.square(X), axis=1)
        train_sum = np.sum(np.square(self.X_train), axis=1)
        inner_product = np.dot(X, self.X_train.T)
        dists = np.sqrt(-2 * inner_product + test_sum.reshape(-1, 1) + train_sum)
  

3. 超参数的调整
   1）k设置为什么数字最有效？
   2）哪种距离函数最合适？
   这些都称为超参数，我们可以通过设置不同的值，来调整它的效果。
   我们通常可将数据集分为三部分：训练集、验证集和测试集。
   
   在训练集上用不同的超参来训练算法，在验证集上进行评估，然后选择一组在验证集上表现最好的超参，再在测试集上跑一跑，最后这个数据才是可以写在报告里的。
   需要注意的是，我们必须将验证集和测试集分开。因为，如果没有验证集，直接在训练集上选择不同的超参，然后去测试集上跑，最后选择一组在测试集上表现最好的超参，这是不合理的，因为算法的最终目的是要应对没有见过的、全新的数据集，如果采用这种方法，选出来的超参可能只是适合这一部分测试集。
4. 交叉验证
   
   将训练集拆分为多个部分，例如这里分成了5个部分。其中4份用来训练，1份用来验证。之后循环取其中4份来训练，剩下1份做验证，取5次验证结果的平均值作为算法验证结果。最后，一旦对模型进行训练并确定了所有的最佳超参数，就可以在测试数据集上对模型进行一次评估。
5. 实际上，KNN算法现在并不常用，因为它的计算量大，而且像L1、L2这样的距离度量方法实在不适合用在图像上，还有维度灾难（需要训练集密集分布）

二、实验代码
接下来，我们按照作业1的内容，用KNN算法实现对cifar10图像的分类





至此，我们完成了数据集的载入，下一步，我们需要打开 cs231n/classifiers/k_nearest_neighbor.py 完成KNN算法的核心工作

from builtins import range
from builtins import object
import numpy as np
from past.builtins import xrange
class KNearestNeighbor(object):
    """ a kNN classifier with L2 distance """
    def __init__(self):
        pass
    def train(self, X, y):
        """
        Train the classifier. For k-nearest neighbors this is just
        memorizing the training data.
        Inputs:
        - X: A numpy array of shape (num_train, D) containing the training data
          consisting of num_train samples each of dimension D.
        - y: A numpy array of shape (N,) containing the training labels, where
             y[i] is the label for X[i].
        """
        self.X_train = X
        self.y_train = y
    def predict(self, X, k=1, num_loops=0):
        """
        Predict labels for test data using this classifier.
        Inputs:
        - X: A numpy array of shape (num_test, D) containing test data consisting
             of num_test samples each of dimension D.
        - k: The number of nearest neighbors that vote for the predicted labels.
        - num_loops: Determines which implementation to use to compute distances
          between training points and testing points.
        Returns:
        - y: A numpy array of shape (num_test,) containing predicted labels for the
          test data, where y[i] is the predicted label for the test point X[i].
        """
        if num_loops == 0:
            dists = self.compute_distances_no_loops(X)
        elif num_loops == 1:
            dists = self.compute_distances_one_loop(X)
        elif num_loops == 2:
            dists = self.compute_distances_two_loops(X)
        else:
            raise ValueError('Invalid value %d for num_loops' % num_loops)
        return self.predict_labels(dists, k=k)
    def compute_distances_two_loops(self, X):
        """
        Compute the distance between each test point in X and each training point
        in self.X_train using a nested loop over both the training data and the
        test data.
        Inputs:
        - X: A numpy array of shape (num_test, D) containing test data.
        Returns:
        - dists: A numpy array of shape (num_test, num_train) where dists[i, j]
          is the Euclidean distance between the ith test point and the jth training
          point.
        """
        num_test = X.shape[0]
        num_train = self.X_train.shape[0]
        dists = np.zeros((num_test, num_train))
        for i in range(num_test):
            for j in range(num_train):
                #####################################################################
                # TODO:                                                             #
                # Compute the l2 distance between the ith test point and the jth    #
                # training point, and store the result in dists[i, j]. You should   #
                # not use a loop over dimension, nor use np.linalg.norm().          #
                #####################################################################
                # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
                dists[i, j] = np.sqrt(np.sum((X[i, :] - self.X_train[j, :]) ** 2))
               # pass

                # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
        return dists

    def compute_distances_one_loop(self, X):
        """
        Compute the distance between each test point in X and each training point
        in self.X_train using a single loop over the test data.

        Input / Output: Same as compute_distances_two_loops
        """
        num_test = X.shape[0]
        num_train = self.X_train.shape[0]
        dists = np.zeros((num_test, num_train))
        for i in range(num_test):
            #######################################################################
            # TODO:                                                               #
            # Compute the l2 distance between the ith test point and all training #
            # points, and store the result in dists[i, :].                        #
            # Do not use np.linalg.norm().                                        #
            #######################################################################
            # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
            dists[i, :] = np.sqrt(np.sum(np.square(self.X_train - X[i, :]), axis=1))
          #  pass

            # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
        return dists

    def compute_distances_no_loops(self, X):
        """
        Compute the distance between each test point in X and each training point
        in self.X_train using no explicit loops.

        Input / Output: Same as compute_distances_two_loops
        """
        num_test = X.shape[0]
        num_train = self.X_train.shape[0]
        dists = np.zeros((num_test, num_train))
        #########################################################################
        # TODO:                                                                 #
        # Compute the l2 distance between all test points and all training      #
        # points without using any explicit loops, and store the result in      #
        # dists.                                                                #
        #                                                                       #
        # You should implement this function using only basic array operations; #
        # in particular you should not use functions from scipy,                #
        # nor use np.linalg.norm().                                             #
        #                                                                       #
        # HINT: Try to formulate the l2 distance using matrix multiplication    #
        #       and two broadcast sums.                                         #
        #########################################################################
        # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
        test_sum = np.sum(np.square(X), axis=1)
        train_sum = np.sum(np.square(self.X_train), axis=1)
        inner_product = np.dot(X, self.X_train.T)
        dists = np.sqrt(-2 * inner_product + test_sum.reshape(-1, 1) + train_sum)
        #pass

        # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
        return dists

    def predict_labels(self, dists, k=1):
        """
        Given a matrix of distances between test points and training points,
        predict a label for each test point.

        Inputs:
        - dists: A numpy array of shape (num_test, num_train) where dists[i, j]
          gives the distance betwen the ith test point and the jth training point.

        Returns:
        - y: A numpy array of shape (num_test,) containing predicted labels for the
          test data, where y[i] is the predicted label for the test point X[i].
        """
        num_test = dists.shape[0]
        y_pred = np.zeros(num_test)
        for i in range(num_test):
            # A list of length k storing the labels of the k nearest neighbors to
            # the ith test point.
            closest_y = []
            #########################################################################
            # TODO:                                                                 #
            # Use the distance matrix to find the k nearest neighbors of the ith    #
            # testing point, and use self.y_train to find the labels of these       #
            # neighbors. Store these labels in closest_y.                           #
            # Hint: Look up the function numpy.argsort.                             #
            #########################################################################
            # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
            closest_y = []
            y_indicies = np.argsort(dists[i, :], axis=0)
            closest_y = self.y_train[y_indicies[: k]]  
            y_pred[i] = np.argmax(np.bincount(closest_y))
           # pass

            # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
            #########################################################################
            # TODO:                                                                 #
            # Now that you have found the labels of the k nearest neighbors, you    #
            # need to find the most common label in the list closest_y of labels.   #
            # Store this label in y_pred[i]. Break ties by choosing the smaller     #
            # label.                                                                #
            #########################################################################
            # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

           # pass

            # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

        return y_pred

这里对比了二层循环、一层循环和无循环的执行时间，可以看出无循环的执行时间和其它两种相比非常短，所以我们应尽量减少循环，而一层循环比二层循环的时间还要长，猜想可能是广播机制中的维度扩展会花费一点时间。





从最后的结果中可以看出，准确率只有28.2%，因此在实际中很少使用该算法，但是作为一种基础性算法，也希望大家都可以很好地掌握