二、Python实现
对于机器学习而已,Python需要额外安装三件宝,分别是Numpy,scipy和Matplotlib。前两者用于数值计算,后者用于画图。安装很简单,直接到各自的官网下载回来安装即可。安装程序会自动搜索我们的python版本和目录,然后安装到python支持的搜索路径下。反正就python和这三个插件都默认安装就没问题了。
另外,如果我们需要添加我们的脚本目录进Python的目录(这样Python的命令行就可以直接import),可以在系统环境变量中添加:PYTHONPATH环境变量,值为我们的路径,例如:E:\Python\Machine Learning in Action
2.1、kNN基础实践
一般实现一个算法后,我们需要先用一个很小的数据库来测试它的正确性,否则一下子给个大数据给它,它也很难消化,而且还不利于我们分析代码的有效性。
首先,我们新建一个kNN.py脚本文件,文件里面包含两个函数,一个用来生成小数据库,一个实现kNN分类算法。代码如下:
- #########################################
- # kNN: k Nearest Neighbors
- # Input: newInput: vector to compare to existing dataset (1xN)
- # dataSet: size m data set of known vectors (NxM)
- # labels: data set labels (1xM vector)
- # k: number of neighbors to use for comparison
- # Output: the most popular class label
- #########################################
- from numpy import *
- import operator
- # create a dataset which contains 4 samples with 2 classes
- def createDataSet():
- # create a matrix: each row as a sample
- group = array([[1.0, 0.9], [1.0, 1.0], [0.1, 0.2], [0.0, 0.1]])
- labels = ['A', 'A', 'B', 'B'] # four samples and two classes
- return group, labels
- # classify using kNN
- def kNNClassify(newInput, dataSet, labels, k):
- numSamples = dataSet.shape[0] # shape[0] stands for the num of row
- ## step 1: calculate Euclidean distance
- # tile(A, reps): Construct an array by repeating A reps times
- # the following copy numSamples rows for dataSet
- diff = tile(newInput, (numSamples, 1)) - dataSet # Subtract element-wise
- squaredDiff = diff ** 2 # squared for the subtract
- squaredDist = sum(squaredDiff, axis = 1) # sum is performed by row
- distance = squaredDist ** 0.5
- ## step 2: sort the distance
- # argsort() returns the indices that would sort an array in a ascending order
- sortedDistIndices = argsort(distance)
- classCount = {} # define a dictionary (can be append element)
- for i in xrange(k):
- ## step 3: choose the min k distance
- voteLabel = labels[sortedDistIndices[i]]
- ## step 4: count the times labels occur
- # when the key voteLabel is not in dictionary classCount, get()
- # will return 0
- classCount[voteLabel] = classCount.get(voteLabel, 0) + 1
- ## step 5: the max voted class will return
- maxCount = 0
- for key, value in classCount.items():
- if value > maxCount:
- maxCount = value
- maxIndex = key
- return maxIndex
然后我们在命令行中这样测试即可:
- import kNN
- from numpy import *
- dataSet, labels = kNN.createDataSet()
- testX = array([1.2, 1.0])
- k = 3
- outputLabel = kNN.kNNClassify(testX, dataSet, labels, 3)
- print "Your input is:", testX, "and classified to class: ", outputLabel
- testX = array([0.1, 0.3])
- outputLabel = kNN.kNNClassify(testX, dataSet, labels, 3)
- print "Your input is:", testX, "and classified to class: ", outputLabel
这时候会输出:
- Your input is: [ 1.2 1.0] and classified to class: A
- Your input is: [ 0.1 0.3] and classified to class: B
2.2、kNN进阶
这里我们用kNN来分类一个大点的数据库,包括数据维度比较大和样本数比较多的数据库。这里我们用到一个手写数字的数据库,可以到这里下载。这个数据库包括数字0-9的手写体。每个数字大约有200个样本。每个样本保持在一个txt文件中。手写体图像本身的大小是32x32的二值图,转换到txt文件保存后,内容也是32x32个数字,0或者1,如下:
数据库解压后有两个目录:目录trainingDigits存放的是大约2000个训练数据,testDigits存放大约900个测试数据。
这里我们还是新建一个kNN.py脚本文件,文件里面包含四个函数,一个用来生成将每个样本的txt文件转换为对应的一个向量,一个用来加载整个数据库,一个实现kNN分类算法。最后就是实现这个加载,测试的函数。
- #########################################
- # kNN: k Nearest Neighbors
- # Input: inX: vector to compare to existing dataset (1xN)
- # dataSet: size m data set of known vectors (NxM)
- # labels: data set labels (1xM vector)
- # k: number of neighbors to use for comparison
- # Output: the most popular class label
- #########################################
- from numpy import *
- import operator
- import os
- # classify using kNN
- def kNNClassify(newInput, dataSet, labels, k):
- numSamples = dataSet.shape[0] # shape[0] stands for the num of row
- ## step 1: calculate Euclidean distance
- # tile(A, reps): Construct an array by repeating A reps times
- # the following copy numSamples rows for dataSet
- diff = tile(newInput, (numSamples, 1)) - dataSet # Subtract element-wise
- squaredDiff = diff ** 2 # squared for the subtract
- squaredDist = sum(squaredDiff, axis = 1) # sum is performed by row
- distance = squaredDist ** 0.5
- ## step 2: sort the distance
- # argsort() returns the indices that would sort an array in a ascending order
- sortedDistIndices = argsort(distance)
- classCount = {} # define a dictionary (can be append element)
- for i in xrange(k):
- ## step 3: choose the min k distance
- voteLabel = labels[sortedDistIndices[i]]
- ## step 4: count the times labels occur
- # when the key voteLabel is not in dictionary classCount, get()
- # will return 0
- classCount[voteLabel] = classCount.get(voteLabel, 0) + 1
- ## step 5: the max voted class will return
- maxCount = 0
- for key, value in classCount.items():
- if value > maxCount:
- maxCount = value
- maxIndex = key
- return maxIndex
- # convert image to vector
- def img2vector(filename):
- rows = 32
- cols = 32
- imgVector = zeros((1, rows * cols))
- fileIn = open(filename)
- for row in xrange(rows):
- lineStr = fileIn.readline()
- for col in xrange(cols):
- imgVector[0, row * 32 + col] = int(lineStr[col])
- return imgVector
- # load dataSet
- def loadDataSet():
- ## step 1: Getting training set
- print "---Getting training set..."
- dataSetDir = 'E:/Python/Machine Learning in Action/'
- trainingFileList = os.listdir(dataSetDir + 'trainingDigits') # load the training set
- numSamples = len(trainingFileList)
- train_x = zeros((numSamples, 1024))
- train_y = []
- for i in xrange(numSamples):
- filename = trainingFileList[i]
- # get train_x
- train_x[i, :] = img2vector(dataSetDir + 'trainingDigits/%s' % filename)
- # get label from file name such as "1_18.txt"
- label = int(filename.split('_')[0]) # return 1
- train_y.append(label)
- ## step 2: Getting testing set
- print "---Getting testing set..."
- testingFileList = os.listdir(dataSetDir + 'testDigits') # load the testing set
- numSamples = len(testingFileList)
- test_x = zeros((numSamples, 1024))
- test_y = []
- for i in xrange(numSamples):
- filename = testingFileList[i]
- # get train_x
- test_x[i, :] = img2vector(dataSetDir + 'testDigits/%s' % filename)
- # get label from file name such as "1_18.txt"
- label = int(filename.split('_')[0]) # return 1
- test_y.append(label)
- return train_x, train_y, test_x, test_y
- # test hand writing class
- def testHandWritingClass():
- ## step 1: load data
- print "step 1: load data..."
- train_x, train_y, test_x, test_y = loadDataSet()
- ## step 2: training...
- print "step 2: training..."
- pass
- ## step 3: testing
- print "step 3: testing..."
- numTestSamples = test_x.shape[0]
- matchCount = 0
- for i in xrange(numTestSamples):
- predict = kNNClassify(test_x[i], train_x, train_y, 3)
- if predict == test_y[i]:
- matchCount += 1
- accuracy = float(matchCount) / numTestSamples
- ## step 4: show the result
- print "step 4: show the result..."
- print 'The classify accuracy is: %.2f%%' % (accuracy * 100)
测试非常简单,只需要在命令行中输入:
- import kNN
- kNN.testHandWritingClass()
输出结果如下:
-
- step 1: load data...
- ---Getting training set...
- ---Getting testing set...
- step 2: training...
- step 3: testing...
- step 4: show the result...
- The classify accuracy is: 98.84%
个人修改一些注释:
# -*- coding: utf-8 -*-
""" KNN: K Nearest Neighbors Input: newInput:vector to compare to existing dataset(1xN) dataSet:size m data set of known vectors(NxM) labels:data set labels(1xM vector) k:number of neighbors to use for comparison Output: the most popular class labels N为数据的维度 M为数据个数 """ from numpy import * import operator #create a dataset which contains 4 samples with 2 classes def createDataSet(): #create a matrix:each row as a sample group = array([[1.0,0.9],[1.0,1.0],[0.1,0.2],[0.0,0.1]]) #four samples and two classes labels = ['A','A','B','B'] return group,labels #classify using KNN def KNNClassify(newInput, dataSet, labels, k): numSamples = dataSet.shape[0] #shape[0] stands for the num of row 即是m ##step 1:calculate Euclidean distance #tile(A,reps):Construct an array by repeating A reps times #the following copy numSamples rows for dataSet diff = tile(newInput,(numSamples,1)) - dataSet #Subtract element-wise squaredDiff = diff ** 2 #squared for the subtract squaredDist = sum(squaredDiff, axis = 1) #sum is performed by row distance = squaredDist ** 0.5 ##step 2:sort the distance #argsort() return the indices that would sort an array in a ascending order sortedDistIndices = argsort(distance) classCount = {} #define a dictionary (can be append element) for i in xrange(k): ##step 3:choose the min k diatance voteLabel = labels[sortedDistIndices[i]] ##step 4:count the times labels occur #when the key voteLabel is not in dictionary classCount,get() #will return 0 #按classCount字典的第2个元素(即类别出现的次数)从大到小排序 #即classCount是一个字典,key是类型,value是该类型出现的次数,通过for循环遍历来计算 classCount[voteLabel] = classCount.get(voteLabel,0) + 1 ##step 5:the max voted class will return #eg:假设classCount={'A':3,'B':2} maxCount = 0 for key,value in classCount.items(): if value > maxCount: maxCount = value maxIndex = key return maxIndex