K邻近算法(KNN)过程
- 计算出待分类点与所有标好类点的距离
- 对所有距离进行排序
- 找出最近的K个点
- 待分类点所属的类为K个点中频数最高的类
这个过程看起来特别简单,只有两个问题需要考虑:距离怎么定义,K值
定义距离
常用的距离:欧式距离,曼哈顿距离(Manhattan distance),余弦值(cos), 相关度 (correlation),
欧氏距离:X=(
x
1
x_1
x1,
x
2
x_2
x2,
x
3
x_3
x3…,
x
n
x_n
xn),Y=(
y
1
y_1
y1,
y
2
y_2
y2,
y
3
y_3
y3…,
y
n
y_n
yn)
d
=
∑
i
=
1
n
(
x
i
−
y
i
)
2
d = \sqrt{\sum_{i=1}^n(x_i -y_i)^2}
d=i=1∑n(xi−yi)2
曼哈顿距离:X=(
x
1
x_1
x1,
x
2
x_2
x2,
x
3
x_3
x3…,
x
n
x_n
xn),Y=(
y
1
y_1
y1,
y
2
y_2
y2,
y
3
y_3
y3…,
y
n
y_n
yn)
d
=
∑
i
=
1
n
∣
(
x
i
−
y
i
)
∣
d = \sum_{i=1}^n|(x_i -y_i)|
d=i=1∑n∣(xi−yi)∣
闵可夫斯基距离:X=(
x
1
x_1
x1,
x
2
x_2
x2,
x
3
x_3
x3…,
x
n
x_n
xn),Y=(
y
1
y_1
y1,
y
2
y_2
y2,
y
3
y_3
y3…,
y
n
y_n
yn)
d
=
∑
i
=
1
n
(
x
i
−
y
i
)
p
p
d = \sqrt[p]{\sum_{i=1}^n(x_i -y_i)^p}
d=pi=1∑n(xi−yi)p
切比雪夫距离:X=(
x
1
x_1
x1,
x
2
x_2
x2,
x
3
x_3
x3…,
x
n
x_n
xn),Y=(
y
1
y_1
y1,
y
2
y_2
y2,
y
3
y_3
y3…,
y
n
y_n
yn)
d
=
max
1
⩽
i
⩽
n
∣
x
i
−
y
i
∣
d = \max_{1\leqslant{i}\leqslant{n}}|x_i -y_i|
d=1⩽i⩽nmax∣xi−yi∣
夹角余弦:X=(
x
1
x_1
x1,
x
2
x_2
x2,
x
3
x_3
x3…,
x
n
x_n
xn),Y=(
y
1
y_1
y1,
y
2
y_2
y2,
y
3
y_3
y3…,
y
n
y_n
yn)
c
o
s
(
θ
)
=
X
⋅
Y
∣
X
∣
⋅
∣
Y
∣
=
∑
i
=
1
n
x
i
y
i
∑
i
=
1
n
x
i
2
⋅
∑
i
=
1
n
y
i
2
cos(θ) = \frac{X·Y}{|X|·|Y|} = \frac{\sum_{i=1}^n{x_iy_i}}{\sqrt{\sum_{i=1}^nx_i ^2} ·\sqrt{\sum_{i=1}^ny_i^2}}
cos(θ)=∣X∣⋅∣Y∣X⋅Y=∑i=1nxi2⋅∑i=1nyi2∑i=1nxiyi
夹角余弦越大,表示两个向量的夹角越小,用来衡量样本向量之间的差异。但是只能分辨个体在维之间的差异,没法衡量每个维数值的差异
相关度:
K的选择:
1.直接用原数据集,得到K取1-51的精确度进行画图,选出最优
2.数据量小的时候用交叉验证,得到K取1-51的精确度进行画图,选出最优
KNN例子
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import validation_curve
# 加载数据集
dataset = load_digits()
score =[]
# 划分数据
X_train, X_test, y_train, y_test = train_test_split(dataset['data'],dataset['target'],random_state= 0)
# scoring: 分类用 accuracy, 回归用 mean_squared_error
train_scores, test_scores = validation_curve(
KNeighborsClassifier(), X_train, y_train, cv=10,
param_name='n_neighbors', param_range=range(1, 51),
scoring='accuracy')
train_scores_mean = np.mean(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
plt.plot(range(1, 51),train_scores_mean, color='red', label='train')
plt.plot(range(1, 51), test_scores_mean, color='green', label='test')
plt.legend('best')
plt.xlabel('param range of k')
plt.ylabel('scores mean')
plt.show()
所以带入K=2:
from sklearn.datasets import load_digits
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
import numpy as np
# 加载数据集
dataset = load_digits()
score =[]
# 划分数据
X_train, X_test, y_train, y_test = train_test_split(dataset['data'],dataset['target'],random_state= 0)
knn = KNeighborsClassifier(n_neighbors = 2)
# 构建模型
knn.fit(X_train,y_train)
# 得出分数
print("score:{:f}".format(knn.score(X_test,y_test)))
score:0.984444
KNeighborsClassifier参数
语法:
KNeighborsClassifier(n_neighbors=5, weights=’uniform’, algorithm=’auto’,
leaf_size=30, p=2, metric=’minkowski’, metric_params=None,
n_jobs=None, **kwargs)
- n_neighbors:K的值(默认5)
- weights:点的权重可以是,‘uniform’,‘distance’,或者callable function(还没看懂怎么用)
- ‘uniform’:所有类中的每个点权重一样
- ‘distance’:把距离的倒数作为权重,越近的点权重越大,因此距离近的点对类的影响大
- algorithm:找最近点的算法{‘auto’, ‘ball_tree’, ‘kd_tree’, ‘brute’}
- ‘ball_tree’ will use BallTree
- ‘kd_tree’ will use KDTree
- ‘brute’ will use a brute-force search.
- ‘auto’ will attempt to decide the most appropriate algorithm based on the values passed to fit method.
- leaf_size : 整数(默认为30)
使用BallTree or KDTree算法时可以用叶节点个数来控制构造树、进行预测的速度、以及存储空间大小 - p : 整数(默认2)
使用闵可夫斯基距离时的p值 - metric:距离(默认闵可夫斯基距离‘minkowski’)
- metric_params:其他距离参数
- n_jobs ::整数或者None (默认None)
找临近点的时候可以并行的运算数,None表示1,-1表示用上所有可用的