神经网络
计算机视觉所面临的挑战
照射角度;光照强度;形状改变;部分遮蔽;背景混入…
常规套路:1.收集数据并给定标签2.训练一个分类器3.测试,评估
K近邻尝试图像分类
对于未知类别属性数据集中的点:
1.计算已知类别数据集中的点与当前点的距离
2.按照距离依次排序
3.选取与当前点距离最小的K个点
4.确定前K个点所在类别的出现概率
5.返回前K个点出现频率最高的类别作为当前点预测分类。
概述:
KNN 算法本身简单有效,它是一种 lazy-learning 算法。
分类器不需要使用训练集进行训练,训练时间复杂度为0。
KNN 分类的计算复杂度和训练集中的文档数目成正比,也就是说,如果训练集中文档总数为 n,那么 KNN 的分类时间复杂度为O(n)。
K 值的选择,距离度量和分类决策规则是该算法的三个基本要素
问题:该算法在分类时有个主要的不足是,当样本不平衡时,如一个类的样本容量很大,而其他类样本容量很小时,有可能导致当输入一个新样本时,该样本的 K 个邻居中大容量类的样本占多数
解决:不同的样本给予不同权重项
数据库样例:CIFAR-10
10类标签;50000个训练数据;10000个测试数据;大小均为32*32
如何计算呢?
代码:
超参数:
L1 (Manhattan) distance d 1 ( I 1 , I 2 ) = ∑ p ∣ I 1 p − I 2 p ∣ d_{1}\left(I_{1}, I_{2}\right)=\sum_{p}\left|I_{1}^{p}-I_{2}^{p}\right| d1(I1,I2)=∑p∣I1p−I2p∣
L2 (Euclidean) distance d 2 ( I 1 , I 2 ) = ∑ p ( I 1 p − I 2 p ) 2 d_{2}\left(I_{1}, I_{2}\right)=\sqrt{\sum_{p}\left(I_{1}^{p}-I_{2}^{p}\right)^{2}} d2(I1,I2)=∑p(I1p−I2p)2
问题:1.对于距离如何设定?2.对于K近邻的K该如何选择?3.如果有的话,其它的超参数该怎么设定呢?
如何找到最好的参数:多次用测试数据试验,找到做好的一组参数组合?错误的的想法,测试数据只能最终用
测试结果:
1.选取超参数的正确方法是:将原始训练集分为训练集和验证集 验证集,我们在验证集上尝试不同的超参数,最后保留表现最好那个
2.如果训练数据量不够,使用交叉验证 交叉验证方法,它能帮助我们在选取最优超参数的时候减少噪音。
3.一旦找到最优的超参数,就让算法以该参数在测试集跑且只跑一次,并根据测试结果评价算法。
4.最近邻分类器能够在CIFAR-10上得到将近40%的准确率。该算法简单易实现,但需要存储所有训练数据,并且在测试的时候过于耗费计算能力
5.最后,我们知道了仅仅使用L1和L2范数来进行像素比较是不够的,图像更多的是按照背景和颜色被分类,而不是语义主体分身。
1.预处理你的数据:对你数据中的特征进行归一化(normalize),让其具有零平均值(zero mean)和单位方差(unit variance)。
2.如果数据是高维数据,考虑使用降维方法,比如PCA
3.将数据随机分入训练集和验证集。按照一般规律,70%-90% 数据作为训练集
4.在验证集上调优,尝试足够多的k值,尝试L1和L2两种范数计算方式。
线性分类
实例:
损失函数
f ( x , W ) = W x L = 1 N ∑ i = 1 N ∑ j ≠ y i max ( 0 , f ( x i ; W ) j − f ( x i ; W ) y i + 1 ) x = [ 1 , 1 , 1 , 1 ] w 1 = [ 1 , 0 , 0 , 0 ] w 2 = [ 0.25 , 0.25 , 0.25 , 0.25 ] \begin{aligned} & f(x, W)=W x \\ L &=\frac{1}{N} \sum_{i=1}^{N} \sum_{j \neq y_{i}} \max \left(0, f\left(x_{i} ; W\right)_{j}-f\left(x_{i} ; W\right)_{y_{i}}+1\right) \\ x &=[1,1,1,1] \\ w_{1} &=[1,0,0,0] \\ w_{2} &=[0.25,0.25,0.25,0.25] \end{aligned} Lxw1w2f(x,W)=Wx=N1i=1∑Nj=yi∑max(0,f(xi;W)j−f(xi;W)yi+1)=[1,1,1,1]=[1,0,0,0]=[0.25,0.25,0.25,0.25]
加入正则化惩罚项:
R ( W ) = ∑ k ∑ l W k , l 2 x = [ 1 , 1 , 1 , 1 ] w 1 = [ 1 , 0 , 0 , 0 ] w 2 = [ 0.25 , 0.25 , 0.25 , 0.25 ] w 1 T x = w 2 T x = 1 \begin{array}{l} R(W)=\sum_{k} \sum_{l} W_{k, l}^{2} \\ x=[1,1,1,1] \\ w_{1}=[1,0,0,0] \\ w_{2}=[0.25,0.25,0.25,0.25] \\ w_{1}^{T} x=w_{2}^{T} x=1 \end{array} R(W)=∑k∑lWk,l2x=[1,1,1,1]w1=[1,0,0,0]w2=[0.25,0.25,0.25,0.25]w1Tx=w2Tx=1
损失函数终极版: L = 1 N ∑ i ∑ j ≠ y i [ max ( 0 , f ( x i ; W ) j − f ( x i ; W ) y i + Δ ) ] + λ ∑ k ∑ l W k , l 2 L=\frac{1}{N} \sum_{i} \sum_{j \neq y_{i}}\left[\max \left(0, f\left(x_{i} ; W\right)_{j}-f\left(x_{i} ; W\right)_{y_{i}}+\Delta\right)\right]+\lambda \sum_{k} \sum_{l} W_{k, l}^{2} L=N1∑i∑j=yi[max(0,f(xi;W)j−f(xi;W)yi+Δ)]+λ∑k∑lWk,l2
Softmax分类器
Softmax的输出(归一化的分类概率)
损失函数:交叉熵损失 交叉熵损失(cross-entropy loss)
L i = − log ( e f y i ∑ j e f j ) L i=-\log \left(\frac{e^{f_{y_{i}}}}{\sum_{j} e^{f_{j}}}\right) Li=−log(∑jefjefyi)
f
j
(
z
)
=
e
z
j
∑
k
e
z
k
f_{j}(z)=\frac{e^{z_{j}}}{\sum_{k} e^{z_{k}}}
fj(z)=∑kezkezj 被称作softmax 函数
其输入值是一个向量,向量中元素为任意实数的评分值
输出一个向量,其中每个元素值在0到1之间,且所有元素之和为1
实例:
最优化
跟随梯度: d f ( x ) d x = lim h → 0 f ( x + h ) − f ( x ) h \frac{d f(x)}{d x}=\lim _{h \rightarrow 0} \frac{f(x+h)-f(x)}{h} dxdf(x)=limh→0hf(x+h)−f(x)
反向传播
神经网络架构
这些线表示权重系数w1,w2,w3
单层的神经网络: f = W 2 max ( 0 , W 1 x ) f=W_{2} \max \left(0, W_{1} x\right) f=W2max(0,W1x)
双层的神经网络: f = W 3 max ( 0 , W 2 max ( 0 , W 1 x ) ) f=W_{3} \max \left(0, W_{2} \max \left(0, W_{1} x\right)\right) f=W3max(0,W2max(0,W1x))
激活函数
缺点:会出现梯度消失的现象
实例演示
演示网站:https://cs.stanford.edu/people/karpathy/convnetjs/demo/classify2d.html
越多的神经元,就越能够表达能复杂的模型。神经元就是权重参数,也要防止过拟合现象
drawData.py
import numpy as np
import matplotlib.pyplot as plt
# ubuntu 16.04 sudo pip instal matplotlib
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'
np.random.seed(0)
N = 100 # number of points per class
D = 2 # dimensionality
K = 3 # number of classes
X = np.zeros((N * K, D))
y = np.zeros(N * K, dtype='uint8')
for j in range(K):
ix = range(N * j, N * (j + 1))
r = np.linspace(0.0, 1, N) # radius
t = np.linspace(j * 4, (j + 1) * 4, N) + np.random.randn(N) * 0.2 # theta
X[ix] = np.c_[r * np.sin(t), r * np.cos(t)]
y[ix] = j
fig = plt.figure()
plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap=plt.cm.Spectral)
plt.xlim([-1, 1])
plt.ylim([-1, 1])
plt.show()
linercla.py
# Train a Linear Classifier
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(0)
N = 100 # number of points per class
D = 2 # dimensionality
K = 3 # number of classes
X = np.zeros((N * K, D))
y = np.zeros(N * K, dtype='uint8')
for j in range(K):
ix = range(N * j, N * (j + 1))
r = np.linspace(0.0, 1, N) # radius
t = np.linspace(j * 4, (j + 1) * 4, N) + np.random.randn(N) * 0.2 # theta
X[ix] = np.c_[r * np.sin(t), r * np.cos(t)]
y[ix] = j
# 初始化
W = 0.01 * np.random.randn(D, K)
b = np.zeros((1, K))
# some hyperparameters
# 正则化惩罚项
step_size = 1e-0
reg = 1e-3 # regularization strength
# gradient descent loop
num_examples = X.shape[0]
for i in range(1000):
# print X.shape
# evaluate class scores, [N x K]
scores = np.dot(X, W) + b # x:300*2 scores:300*3
# print scores.shape
# compute the class probabilities
exp_scores = np.exp(scores)
probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True) # [N x K] probs:300*3
print(probs.shape)
# compute the loss: average cross-entropy loss and regularization
corect_logprobs = -np.log(probs[range(num_examples), y]) # corect_logprobs:300*1
print(corect_logprobs.shape)
data_loss = np.sum(corect_logprobs) / num_examples
reg_loss = 0.5 * reg * np.sum(W * W)
loss = data_loss + reg_loss
if i % 100 == 0:
print("iteration %d: loss %f" % (i, loss))
# compute the gradient on scores
dscores = probs
dscores[range(num_examples), y] -= 1
dscores /= num_examples
# backpropate the gradient to the parameters (W,b)
dW = np.dot(X.T, dscores)
db = np.sum(dscores, axis=0, keepdims=True)
dW += reg * W # regularization gradient
# perform a parameter update
W += -step_size * dW
b += -step_size * db
scores = np.dot(X, W) + b
predicted_class = np.argmax(scores, axis=1)
print('training accuracy: %.2f' % (np.mean(predicted_class == y)))
h = 0.02
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
np.arange(y_min, y_max, h))
Z = np.dot(np.c_[xx.ravel(), yy.ravel()], W) + b
Z = np.argmax(Z, axis=1)
Z = Z.reshape(xx.shape)
fig = plt.figure()
plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral, alpha=0.8)
plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap=plt.cm.Spectral)
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.show()
NNcla.py
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(0)
N = 100 # number of points per class
D = 2 # dimensionality
K = 3 # number of classes
X = np.zeros((N * K, D))
y = np.zeros(N * K, dtype='uint8')
for j in range(K):
ix = range(N * j, N * (j + 1))
r = np.linspace(0.0, 1, N) # radius
t = np.linspace(j * 4, (j + 1) * 4, N) + np.random.randn(N) * 0.2 # theta
X[ix] = np.c_[r * np.sin(t), r * np.cos(t)]
y[ix] = j
h = 100 # size of hidden layer
W = 0.01 * np.random.randn(D, h) # x:300*2 2*100
b = np.zeros((1, h))
W2 = 0.01 * np.random.randn(h, K)
b2 = np.zeros((1, K))
# some hyperparameters
step_size = 1e-0
reg = 1e-3 # regularization strength
# gradient descent loop
num_examples = X.shape[0]
for i in range(2000):
# evaluate class scores, [N x K]
hidden_layer = np.maximum(0, np.dot(X, W) + b) # note, ReLU activation hidden_layer:300*100
# print hidden_layer.shape
scores = np.dot(hidden_layer, W2) + b2 # scores:300*3
# print scores.shape
# compute the class probabilities
exp_scores = np.exp(scores)
probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True) # [N x K]
# print probs.shape
# compute the loss: average cross-entropy loss and regularization
corect_logprobs = -np.log(probs[range(num_examples), y])
data_loss = np.sum(corect_logprobs) / num_examples
reg_loss = 0.5 * reg * np.sum(W * W) + 0.5 * reg * np.sum(W2 * W2)
loss = data_loss + reg_loss
if i % 100 == 0:
print
"iteration %d: loss %f" % (i, loss)
# compute the gradient on scores
dscores = probs
dscores[range(num_examples), y] -= 1
dscores /= num_examples
# backpropate the gradient to the parameters
# first backprop into parameters W2 and b2
dW2 = np.dot(hidden_layer.T, dscores)
db2 = np.sum(dscores, axis=0, keepdims=True)
# next backprop into hidden layer
dhidden = np.dot(dscores, W2.T)
# backprop the ReLU non-linearity
dhidden[hidden_layer <= 0] = 0
# finally into W,b
dW = np.dot(X.T, dhidden)
db = np.sum(dhidden, axis=0, keepdims=True)
# add regularization gradient contribution
dW2 += reg * W2
dW += reg * W
# perform a parameter update
W += -step_size * dW
b += -step_size * db
W2 += -step_size * dW2
b2 += -step_size * db2
hidden_layer = np.maximum(0, np.dot(X, W) + b)
scores = np.dot(hidden_layer, W2) + b2
predicted_class = np.argmax(scores, axis=1)
print
'training accuracy: %.2f' % (np.mean(predicted_class == y))
h = 0.02
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
np.arange(y_min, y_max, h))
Z = np.dot(np.maximum(0, np.dot(np.c_[xx.ravel(), yy.ravel()], W) + b), W2) + b2
Z = np.argmax(Z, axis=1)
Z = Z.reshape(xx.shape)
fig = plt.figure()
plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral, alpha=0.8)
plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap=plt.cm.Spectral)
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.show()
问题解决
PyCharm错误解决办法:ModuleNotFoundError: No module named 'matplotlib’
https://blog.csdn.net/quantum7/article/details/82918206
NameError: name ‘xrange’ is not defined
https://www.cnblogs.com/hdk1993/p/8893991.html