Supervised Learning 003: k-Nearest Neighbor - True useful case

Use Case:

     The case is about the number recognition.  关于数字识别的例子

     There are some training data for classifier, they are all files and every file present a number structure. all files is 32*32 matrix to mean the shape of the number using 1 and 0. 分类器有一些训练数据， 每一个训练数据都代表一个数字的结构，使用32*32的0，1矩阵结构来表示数字的形状。

     When receiving test data, the classifier will calculate the 'distance' between the test data and all the training data and select the most similar training data. the classifier is the k-NN classifier. 当收到测试数据后， 分类器将会计算样本数据和测试数据的"距离"并选择出最相近的数据。这是一个k-NN分类器。

     At first we should transfer the file data into the matrix data or vector data format because the classifier want the matrix and vector as the samples and input data. 首先需要把所有的数据都转换为矩阵或者向量， 因为分类器需要这类数据结构。

There should be a function to transfer the file into the vector format:

# trans 32* 32 matrix image to a 1*1024 vector
#  return the vector
def img2vector(filename):
    # transfer the img 32*32 to 1*1024 vector
    # read all image data and fill into the vector
    returnVec = zeros((1,1024))
    fr = open(filename)
    for i in range(32):
        lineStr = fr.readline()
        for j in range(32):
            returnVec[0, i*32+j] = int(lineStr[j])
    return returnVec

And then we can do the testing:

# recognize the handwriting class using the kNN classifier algorithm
# training data is a set of file, every file is 32*32 matrix which identify the shape of the number using 1 and 0
# test data is another set of file, every file is same format to the training data
def handwritingClassTest():
    ## get the training data and store to training Matrix

    # the lables of the handwriting
    #  usually using the number as the label
    hwLabels = []
    # travel the dir of the training data
    trainingFileList = listdir('trainingDigits')
    m = len(trainingFileList)
    # define a m*1024 matrix for all file data
    # each vector present a file data
    trainingMat = zeros( (m,1024))
    for i in range(m):
        # each file name format : #_#.txt
        # first number means the present number
        # second number means the sample index of the number
        fileNameStr = trainingFileList[i]
        # get the filename of this file ( not include the extend name )
        fileStr = fileNameStr.split('.')[0]
        # get the present number/class number of the file
        classNumberStr = int(fileStr.split('_')[0])
        hwLabels.append(classNumberStr)
        trainingMat[i,:] = img2vector( "trainingDigits/" + fileNameStr )

    # get the test file data
    testFileList = listdir('testDigits')
    errorCount = 0.0
    mTest = len(testFileList)
    for i in range(mTest):
        fileNameStr = testFileList[i]
        fileStr = fileNameStr.split('.')[0]
        classNumberStr = int(fileStr.split('_')[0])
        vectorUnderTest = img2vector('testDigits/' + fileNameStr)
        classifierResult = kNN.classify0(vectorUnderTest, trainingMat, hwLabels, 3)

        if ( classifierResult !=  classNumberStr ):
            print "came back with %d, real answer is %d,  file is %s" % (classifierResult, classNumberStr, fileNameStr)
            errorCount += 1.0
    print "the total error number is %d, error rate is %f" % ( errorCount , errorCount/float(mTest) )

At last ,we can run it :

if __name__ == '__main__':
    handwritingClassTest()

The result is :

came back with 7, real answer is 1,  file is 1_86.txt
came back with 9, real answer is 3,  file is 3_11.txt
came back with 9, real answer is 3,  file is 3_55.txt
came back with 3, real answer is 5,  file is 5_42.txt
came back with 6, real answer is 5,  file is 5_43.txt
came back with 6, real answer is 8,  file is 8_11.txt
came back with 3, real answer is 8,  file is 8_23.txt
came back with 1, real answer is 8,  file is 8_36.txt
came back with 1, real answer is 8,  file is 8_45.txt
came back with 1, real answer is 9,  file is 9_14.txt
came back with 7, real answer is 9,  file is 9_60.txt
the total error number is 11, error rate is 0.011628

Process finished with exit code 0

Summary:

Depending on your computer’s speed, you may think this algorithm is slow, and you’d be right. For each of our 900 test cases, you had to do 2,000 distance calcula-tions on a 1024-entry floating point vector. Additionally, our test dataset size was 2 MB.Is there a way to make this smaller and take fewer computations? One modification tokNN, called kD-trees, allows you to reduce the number of calculations.  实际上使用这个算法时效率并不高，每个测试向量要做2000次距离计算，每次距离计算要包1024维度浮点运算，这种向量有900次左右。 此外我们还需要为测试数据准备2M内存。 是否存在一种算法可以减少存储空间和计算时间的开销呢? kD-trees就是k-NN的优化版。