K-近邻算法学习及实现
K-近邻原理
K-近邻算法采用测量不同特征值之见的距离方法进行分类。
将新数据与已知数据集(带标签)的每个样本数据进行对比(采用距离),然后算法提取出最相近的K个样本的分类标签,最相似的的k个样本对应标签出现次数最多的分类,作为新数据的分类。
k-近邻算法优缺点
优点:精度高,对异常值不敏感,无数据输入假定
缺点:计算复杂度高,空间复杂度高,预测结果不具有可解释性,维数灾难(高纬度数据距离可能比想象的大)
适用数据范围:数值型和标称型。
注意事项:k-近邻算法采用求距离的方法,各指标量纲和量纲单位不同,因此需要对数据进行数据标准化(归一化)处理,原始数据经过数据标准化(归一化)处理后,各指标处于同一数量纲,适合作为K-近邻算法的输入参数。
K-近邻算法分析
1)计算已知类别数据集中的点与当前点之间的距离;
2)按照距离递增次序排序;
3)选取与当前点距离最小的k个点;
4)确定前k个点所在类别的出现频率;
5)返回前k个点出现频率最高的类别作为当前点的预测分类。
K-近邻算法实现-欧式距离
# 准备数据
import numpy as np
def createDataset():
group = np.array([[1.0,1.1],[1.0,1.0],[0,0],[0,0.1]])
labels = np.array(['A','A','B','B'])
return group, labels
# 新的样本点
x = np.array([0.9,1.0])
X_train, y_train = createDataset()
X_train
array([[ 1. , 1.1],
[ 1. , 1. ],
[ 0. , 0. ],
[ 0. , 0.1]])
y_train
array(['A', 'A', 'B', 'B'],
dtype='<U1')
x
array([ 0.9, 1. ])
import matplotlib.pyplot as plt
plt.scatter(X_train[y_train=='A',0], X_train[y_train=='A',1],color ='g')
plt.scatter(X_train[y_train=='B',0], X_train[y_train=='B',1],color ='r')
plt.scatter(x[0],x[1],color ='b')
plt.show()
欧氏距离公式
a和b两个点的欧式距离
∑(Xai−Xbi)2‾‾‾‾‾‾‾‾‾‾‾‾‾√ ∑ ( X i a − X i b ) 2
# for循环实现距离计算
distance = []
for x_train in X_train:
d = np.sqrt(np.sum((x_train - x)**2))
distance.append(d)
distance
[0.14142135623730953,
0.099999999999999978,
1.3453624047073711,
1.2727922061357855]
# 简化距离计算
distances = [np.sqrt(np.sum((x_train - x)**2)) for x_train in X_train]
distances
[0.14142135623730953,
0.099999999999999978,
1.3453624047073711,
1.2727922061357855]
# 选取前k个值,此处为K=3
# 按照距离排序(由小到大)
nearsort = np.argsort(distances)
k = 3
near3 = [y_train[i] for i in nearsort[:k]]
# 求出占多数的标签
from collections import Counter
votelabel = Counter(near3)
votelabel.most_common(1)[0][0]
'A'
K-近邻算法类封装
import numpy as np
from collections import Counter
class KNNClassifier():
def __init__(self, k):
# 初始化分类器
assert k>=1, 'k must be valid !'
self.k = k
self._X_train = None
self._y_train = None
def fit(self, X_train, y_train):
'''根据训练数据集X_train和y_train来训练KNN分类器'''
self._X_train = X_train
self._y_train = y_train
return self
def predict(self, X_predict):
'''给定待预测数据X_predict,返回X_predict的预测结果向量'''
assert self._X_train is not None and self._y_train is not None ,'Must be fited before predict !'
assert X_predict.shape[1] == self._X_train.shape[1],"The feather number of X_predict must be equal to self._X_train's"
y_predict = [self._predict(x) for x in X_predict]
return np.array(y_predict)
def _predict(self, x):
'''给定单个数据样本,返回该数据样本的预测结果'''
assert x.shape[0] == self._X_train.shape[1],"x's feather is not equal to x_train"
# 简化距离计算
distances = [np.sqrt(np.sum((x_train - x)**2)) for x_train in self._X_train]
# 选取前k个值,此处为K=3,按照距离排序(由小到大)
distance_argsort = np.argsort(distances)
near_k = [y_train[i] for i in distance_argsort[:self.k]]
# 求出占多数的标签
vote_label = Counter(near_k)
return vote_label.most_common(1)[0][0]
def __repr__(self):
return 'KNN(k = %d)' % self.k
验证封装的K-近邻算法
x = x.reshape(1,2)
X_train
array([[ 1. , 1.1],
[ 1. , 1. ],
[ 0. , 0. ],
[ 0. , 0.1]])
y_train
array(['A', 'A', 'B', 'B'],
dtype='<U1')
knn = KNNClassifier(3)
knn.fit(X_train,y_train)
knn.predict(x)
array(['A'],
dtype='<U1')
本学习笔记参考:
《机器学习实战》和《Python3入门机器学习 经典算法与应用》