cs231n_KNN

KNN原理

KNN 就是 k-Nearest Neighbor, 过程如下：

1. train: 将数据集的数据和对应的标签读入到类中
2. test: 首先计算一个矩阵（num_test, num_train）存储测试集中每一个数据与训练集中每个数据的距离，这里我们采用L2距离进行计算。
3. 随后在测试第i张测试图像时，我们选取距离最小的前k个训练图片，将其对应的label存入到closest_y中，然后找出出现次数最多的label，将其作为该测试图像的预测结果。

KNN类

import numpy as np
from past.builtins import xrange


class KNearestNeighbor(object):
  """ a kNN classifier with L2 distance """

  def __init__(self):
    pass

  def train(self, X, y):
    """
    Train the classifier. For k-nearest neighbors this is just 
    memorizing the training data.

    Inputs:
    - X: A numpy array of shape (num_train, D) containing the training data
      consisting of num_train samples each of dimension D.
    - y: A numpy array of shape (N,) containing the training labels, where
         y[i] is the label for X[i].
    """
    self.X_train = X
    self.y_train = y
    
  def predict(self, X, k=1, num_loops=0):
    """
    Predict labels for test data using this classifier.

    Inputs:
    - X: A numpy array of shape (num_test, D) containing test data consisting
         of num_test samples each of dimension D.
    - k: The number of nearest neighbors that vote for the predicted labels.
    - num_loops: Determines which implementation to use to compute distances
      between training points and testing points.

    Returns:
    - y: A numpy array of shape (num_test,) containing predicted labels for the
      test data, where y[i] is the predicted label for the test point X[i].  
    """
    if num_loops == 0:
      dists = self.compute_distances_no_loops(X)
    elif num_loops == 1:
      dists = self.compute_distances_one_loop(X)
    elif num_loops == 2:
      dists = self.compute_distances_two_loops(X)
    else:
      raise ValueError('Invalid value %d for num_loops' % num_loops)

    return self.predict_labels(dists, k=k)

  def compute_distances_two_loops(self, X):
    """
    Compute the distance between each test point in X and each training point
    in self.X_train using a nested loop over both the training data and the 
    test data.

    Inputs:
    - X: A numpy array of shape (num_test, D) containing test data.

    Returns:
    - dists: A numpy array of shape (num_test, num_train) where dists[i, j]
      is the Euclidean distance between the ith test point and the jth training
      point.
    """
    num_test = X.shape[0]
    num_train = self.X_train.shape[0]
    dists = np.zeros((num_test, num_train))
    for i in xrange(num_test):
       for j in xrange(num_train):
        #####################################################################
        # TODO:                                                             #
        # Compute the l2 distance between the ith test point and the jth    #
        # training point, and store the result in dists[i, j]. You should   #
        # not use a loop over dimension.                                    #
        #####################################################################
        dists[i][j] = np.sqrt(np.sum(np.square(X[i]-self.X_train[j])))
        #####################################################################
        #                       END OF YOUR CODE                            #
        #####################################################################
    return dists

  def compute_distances_one_loop(self, X):
    """
    Compute the distance between each test point in X and each training point
    in self.X_train using a single loop over the test data.

    Input / Output: Same as compute_distances_two_loops
    """
    num_test = X.shape[0]
    num_train = self.X_train.shape[0]
    dists = np.zeros((num_test, num_train))
    for i in xrange(num_test):
      #######################################################################
      # TODO:                                                               #
      # Compute the l2 distance between the ith test point and all training #
      # points, and store the result in dists[i, :].                        #
      #######################################################################
      dists[i] = np.sqrt(np.sum(np.square(self.X_train-X[i]),axis=1))
      #######################################################################
      #                         END OF YOUR CODE                            #
      #######################################################################
    return dists

  def compute_distances_no_loops(self, X):
    """
    Compute the distance between each test point in X and each training point
    in self.X_train using no explicit loops.

    Input / Output: Same as compute_distances_two_loops
    """
    num_test = X.shape[0]
    num_train = self.X_train.shape[0]
    dists = np.zeros((num_test, num_train)) 
    #########################################################################
    # TODO:                                                                 #
    # Compute the l2 distance between all test points and all training      #
    # points without using any explicit loops, and store the result in      #
    # dists.                                                                #
    #                                                                       #
    # You should implement this function using only basic array operations; #
    # in particular you should not use functions from scipy.                #
    #                                                                       #
    # HINT: Try to formulate the l2 distance using matrix multiplication    #
    #       and two broadcast sums.                                         #
    #########################################################################
    # (a-b)^2 = a^2-2a*b+b^2
    dist_left = np.sum(np.square(X),axis=1,keepdims=True) #size(num_test,1)
    dist_middle = np.multiply(np.dot(X,self.X_train.T),-2) # size(num_test, num_train)
    dist_right = np.sum(np.square(self.X_train),axis=1) #size(num_train,)
    dists = np.sqrt(dist_middle + dist_left + dist_right)

    #########################################################################
    #                         END OF YOUR CODE                              #
    #########################################################################
    return dists

  def predict_labels(self, dists, k=1):
    """
    Given a matrix of distances between test points and training points,
    predict a label for each test point.

    Inputs:
    - dists: A numpy array of shape (num_test, num_train) where dists[i, j]
      gives the distance betwen the ith test point and the jth training point.

    Returns:
    - y: A numpy array of shape (num_test,) containing predicted labels for the
      test data, where y[i] is the predicted label for the test point X[i].  
    """
    num_test = dists.shape[0]
    y_pred = np.zeros(num_test)
    for i in xrange(num_test):
      # A list of length k storing the labels of the k nearest neighbors to
      # the ith test point.
      closest_y = []
      #########################################################################
      # TODO:                                                                 #
      # Use the distance matrix to find the k nearest neighbors of the ith    #
      # testing point, and use self.y_train to find the labels of these       #
      # neighbors. Store these labels in closest_y.                           #
      # Hint: Look up the function numpy.argsort.                             #
      #########################################################################
      closest_y = self.y_train[np.argsort(dists[i])[:k]]
      #########################################################################
      # TODO:                                                                 #
      # Now that you have found the labels of the k nearest neighbors, you    #
      # need to find the most common label in the list closest_y of labels.   #
      # Store this label in y_pred[i]. Break ties by choosing the smaller     #
      # label.                                                                #
      #########################################################################
      y_pred[i] = np.argmax(np.bincount(closest_y))
#       dists_k_min = np.argsort(dists[i])[:k]
#       close_y = self.y_train[dists_k_min]
#       y_pred[i] = np.argmax(np.bincount(close_y))
      #########################################################################
      #                           END OF YOUR CODE                            # 
      #########################################################################

    return y_pred

小tips

1. np.flatnonzero（array）将array平铺并得到不为零的元素的索引
2. np.linalg.norm 求向量的范数
   x_norm=np.linalg.norm(x, ord=None, axis=None, keepdims=False)
   ord=1: 列和的最大值
   ord=2: 求特征值，然后求最大特征值得算术平方根
   ord=np.inf, 行和的最大值
   axis: 处理类型
   axis=1表示按行向量处理，求多个行向量的范数
   axis=0表示按列向量处理，求多个列向量的范数
   axis=None表示矩阵范数
3. np.concatenate 数组拼接
   np.concatenate((arr1,arr2),axis)
   axis=0时表示，对数组的第0个中括号后的数组维度进行拼接
   axis=1时表示，对数组的第1个中括号后的数组维度进行拼接
4. 使用k折运算时，绘制误差图使用plt.errorbar

plt.errorbar(x, 
	y, 
	yerr=None, 
	xerr=None, 
	fmt='', 
	ecolor=None, 
	elinewidth=None, 
	capsize=None, 
	capthick=None
)

x,y: 数据点的位置坐标
xerr,yerr: 数据的误差范围，经常使用标准差
fmt: 数据点的标记样式以及相互之间连接线样式
ecolor: 误差棒的线条颜色
elinewidth: 误差棒的线条粗细
capsize: 误差棒边界横杠的大小
capthick: 误差棒边界横杠的厚度
ms: 数据点的大小
mfc: 数据点的颜色
mec: 数据点边缘的颜色

5. numpy的广播机制
   两个数组的size,从后往前一致时可以进行广播，或者前面的维度一致，有一个数组的维度为1时也可以进行广播
   例： （4,2，3）与（2,3）可以进行广播，在第一个维度上进行扩展
   （4,3）和（4,1）在1的维度上进行扩展
   （4,3）和（3，）在4的维度上进行扩展

6. numpy的随机数组

# 创建2行2列取值范围为[0,1）的数组
np.random.rand(2,2)
# 创建2行3列，取值范围为标准正态分布的数组
np.random.randn(2,3)
# 创建指定大小的数组，数组数值随机取于[low, high）之间。
np.random.randint(1,20,size=(2,2,3))
# 随机选取一个数
np.random.choice(a, size=None, replace=True, p=None)
a: 指定的一维数组或者整数。如果是整数，则该方法等同于np.arange(a)
size:数组大小
replace:生成的数组中元素是否可以重复。默认为True,即可以重复