DeepLearning tutorial(3)MLP多层感知机原理简介+代码详解
@author:wepon
@blog:http://blog.csdn.net/u012162613/article/details/43221829
本文介绍多层感知机算法,特别是详细解读其代码实现,基于Python theano,代码来自:Multilayer Perceptron,如果你想详细了解多层感知机算法,可以参考:UFLDL教程,或者参考本文第一部分的算法简介。
经详细注释的代码:放在我的github地址上,可下载。
一、多层感知机(MLP)原理简介
多层感知机(MLP,Multilayer Perceptron)也叫人工神经网络(ANN,Artificial Neural Network),除了输入输出层,它中间可以有多个隐层,最简单的MLP只含一个隐层,即三层的结构,如下图:
![](https://img-blog.csdn.net/20150128033221168?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxMjE2MjYxMw==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center)
从上图可以看到,多层感知机层与层之间是全连接的(全连接的意思就是:上一层的任何一个神经元与下一层的所有神经元都有连接)。多层感知机最底层是输入层,中间是隐藏层,最后是输出层。
输入层没什么好说,你输入什么就是什么,比如输入是一个n维向量,就有n个神经元。
隐藏层的神经元怎么得来?首先它与输入层是全连接的,假设输入层用向量X表示,则隐藏层的输出就是
f(W1X+b1),W1是权重(也叫连接系数),b1是偏置,函数f 可以是常用的sigmoid函数或者tanh函数:
最后就是输出层,输出层与隐藏层是什么关系?其实隐藏层到输出层可以看成是一个多类别的逻辑回归,也即softmax回归,所以输出层的输出就是softmax(W2X1+b2),X1表示隐藏层的输出f(W1X+b1)。
MLP整个模型就是这样子的,上面说的这个三层的MLP用公式总结起来就是,函数G是softmax
![](https://img-blog.csdn.net/20150128034130632?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxMjE2MjYxMw==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center)
因此,MLP所有的参数就是各个层之间的连接权重以及偏置,包括W1、b1、W2、b2。对于一个具体的问题,怎么确定这些参数?求解最佳的参数是一个最优化问题,解决最优化问题,最简单的就是梯度下降法了(SGD):首先随机初始化所有参数,然后迭代地训练,不断地计算梯度和更新参数,直到满足某个条件为止(比如误差足够小、迭代次数足够多时)。这个过程涉及到代价函数、规则化(Regularization)、学习速率(learning rate)、梯度计算等,本文不详细讨论,读者可以参考本文顶部给出的两个链接。
了解了MLP的基本模型,下面进入代码实现部分。
二、多层感知机(MLP)代码详细解读(基于python+theano)
这个代码实现的是一个三层的感知机,但是理解了代码之后,实现n层感知机都不是问题,所以只需理解好这个三层的MLP模型即可。概括地说,MLP的输入层X其实就是我们的训练数据,所以输入层不用实现,剩下的就是“输入层到隐含层”,“隐含层到输出层”这两部分。上面介绍原理时已经说到了,“输入层到隐含层”就是一个全连接的层,在下面的代码中我们把这一部分定义为HiddenLayer。“隐含层到输出层”就是一个分类器softmax回归(也有人叫逻辑回归),在下面的代码中我们把这一部分定义为LogisticRegression。
代码详解开始:
(1)导入必要的python模块
主要是numpy、theano,以及python自带的os、sys、time模块,这些模块的使用在下面的程序中会看到。
- import os
- import sys
- import time
-
- import numpy
-
- import theano
- import theano.tensor as T
(2)定义MLP模型(HiddenLayer+LogisticRegression)
这一部分定义MLP的基本“构件”,即上文一直在提的HiddenLayer和LogisticRegression
隐含层我们需要定义连接系数W、偏置b,输入、输出,具体的代码以及解读如下:
- class HiddenLayer(object):
- def __init__(self, rng, input, n_in, n_out, W=None, b=None,
- activation=T.tanh):
-
-
-
-
-
-
-
-
-
-
-
- self.input = input
-
-
-
-
-
-
-
-
-
- if W is None:
- W_values = numpy.asarray(
- rng.uniform(
- low=-numpy.sqrt(6. / (n_in + n_out)),
- high=numpy.sqrt(6. / (n_in + n_out)),
- size=(n_in, n_out)
- ),
- dtype=theano.config.floatX
- )
- if activation == theano.tensor.nnet.sigmoid:
- W_values *= 4
- W = theano.shared(value=W_values, name='W', borrow=True)
-
- if b is None:
- b_values = numpy.zeros((n_out,), dtype=theano.config.floatX)
- b = theano.shared(value=b_values, name='b', borrow=True)
-
-
- self.W = W
- self.b = b
-
-
- lin_output = T.dot(input, self.W) + self.b
- self.output = (
- lin_output if activation is None
- else activation(lin_output)
- )
-
-
- self.params = [self.W, self.b]
-
-
-
-
-
-
-
-
-
-
- class LogisticRegression(object):
- def __init__(self, input, n_in, n_out):
-
-
- self.W = theano.shared(
- value=numpy.zeros(
- (n_in, n_out),
- dtype=theano.config.floatX
- ),
- name='W',
- borrow=True
- )
-
- self.b = theano.shared(
- value=numpy.zeros(
- (n_out,),
- dtype=theano.config.floatX
- ),
- name='b',
- borrow=True
- )
-
-
-
-
-
-
- self.p_y_given_x = T.nnet.softmax(T.dot(input, self.W) + self.b)
-
-
- self.y_pred = T.argmax(self.p_y_given_x, axis=1)
-
-
- self.params = [self.W, self.b]
ok!这两个基本“构件”做好了,现在我们可以将它们“组装”在一起。
我们要三层的MLP,则只需要HiddenLayer+LogisticRegression,
如果要四层的MLP,则为HiddenLayer+HiddenLayer+LogisticRegression........以此类推。
下面是三层的MLP:
-
- class MLP(object):
- def __init__(self, rng, input, n_in, n_hidden, n_out):
-
- self.hiddenLayer = HiddenLayer(
- rng=rng,
- input=input,
- n_in=n_in,
- n_out=n_hidden,
- activation=T.tanh
- )
-
-
- self.logRegressionLayer = LogisticRegression(
- input=self.hiddenLayer.output,
- n_in=n_hidden,
- n_out=n_out
- )
-
-
-
-
-
- self.L1 = (
- abs(self.hiddenLayer.W).sum()
- + abs(self.logRegressionLayer.W).sum()
- )
-
- self.L2_sqr = (
- (self.hiddenLayer.W ** 2).sum()
- + (self.logRegressionLayer.W ** 2).sum()
- )
-
-
-
- self.negative_log_likelihood = (
- self.logRegressionLayer.negative_log_likelihood
- )
-
-
- self.errors = self.logRegressionLayer.errors
-
-
- self.params = self.hiddenLayer.params + self.logRegressionLayer.params
-
MLP类里面除了隐含层和分类层,还定义了损失函数、规则化项,这是在求解优化算法时用到的。
(3)将MLP应用于MNIST(手写数字识别)
上面定义好了一个三层的MLP,接下来使用它在MNIST数据集上分类,MNIST是一个手写数字0~9的数据集。
-
-
-
- def load_data(dataset):
-
-
- data_dir, data_file = os.path.split(dataset)
- if data_dir == "" and not os.path.isfile(dataset):
-
- new_path = os.path.join(
- os.path.split(__file__)[0],
- "..",
- "data",
- dataset
- )
- if os.path.isfile(new_path) or data_file == 'mnist.pkl.gz':
- dataset = new_path
-
- if (not os.path.isfile(dataset)) and data_file == 'mnist.pkl.gz':
- import urllib
- origin = (
- 'http://www.iro.umontreal.ca/~lisa/deep/data/mnist/mnist.pkl.gz'
- )
- print 'Downloading data from %s' % origin
- urllib.urlretrieve(origin, dataset)
-
- print '... loading data'
-
-
-
-
-
- f = gzip.open(dataset, 'rb')
- train_set, valid_set, test_set = cPickle.load(f)
- f.close()
-
-
-
-
- def shared_dataset(data_xy, borrow=True):
- data_x, data_y = data_xy
- shared_x = theano.shared(numpy.asarray(data_x,
- dtype=theano.config.floatX),
- borrow=borrow)
- shared_y = theano.shared(numpy.asarray(data_y,
- dtype=theano.config.floatX),
- borrow=borrow)
- return shared_x, T.cast(shared_y, 'int32')
-
-
- test_set_x, test_set_y = shared_dataset(test_set)
- valid_set_x, valid_set_y = shared_dataset(valid_set)
- train_set_x, train_set_y = shared_dataset(train_set)
-
- rval = [(train_set_x, train_set_y), (valid_set_x, valid_set_y),
- (test_set_x, test_set_y)]
- return rval
加载了数据,可以开始训练这个模型了,以下就是主体函数test_mlp(),将MLP用在MNIST上:
-
- def test_mlp(learning_rate=0.01, L1_reg=0.00, L2_reg=0.0001, n_epochs=10,
- dataset='mnist.pkl.gz', batch_size=20, n_hidden=500):
-
-
-
-
-
-
-
-
-
-
-
-
- datasets = load_data(dataset)
- train_set_x, train_set_y = datasets[0]
- valid_set_x, valid_set_y = datasets[1]
- test_set_x, test_set_y = datasets[2]
-
-
-
- n_train_batches = train_set_x.get_value(borrow=True).shape[0] / batch_size
- n_valid_batches = valid_set_x.get_value(borrow=True).shape[0] / batch_size
- n_test_batches = test_set_x.get_value(borrow=True).shape[0] / batch_size
-
-
-
-
- print '... building the model'
-
-
-
-
- index = T.lscalar()
- x = T.matrix('x')
- y = T.ivector('y')
-
-
- rng = numpy.random.RandomState(1234)
-
- classifier = MLP(
- rng=rng,
- input=x,
- n_in=28 * 28,
- n_hidden=n_hidden,
- n_out=10
- )
-
-
-
- cost = (
- classifier.negative_log_likelihood(y)
- + L1_reg * classifier.L1
- + L2_reg * classifier.L2_sqr
- )
-
-
-
-
-
-
-
-
- test_model = theano.function(
- inputs=[index],
- outputs=classifier.errors(y),
- givens={
- x: test_set_x[index * batch_size:(index + 1) * batch_size],
- y: test_set_y[index * batch_size:(index + 1) * batch_size]
- }
- )
-
- validate_model = theano.function(
- inputs=[index],
- outputs=classifier.errors(y),
- givens={
- x: valid_set_x[index * batch_size:(index + 1) * batch_size],
- y: valid_set_y[index * batch_size:(index + 1) * batch_size]
- }
- )
-
-
- gparams = [T.grad(cost, param) for param in classifier.params]
-
-
-
- updates = [
- (param, param - learning_rate * gparam)
- for param, gparam in zip(classifier.params, gparams)
- ]
-
- train_model = theano.function(
- inputs=[index],
- outputs=cost,
- updates=updates,
- givens={
- x: train_set_x[index * batch_size: (index + 1) * batch_size],
- y: train_set_y[index * batch_size: (index + 1) * batch_size]
- }
- )
-
-
-
-
-
- print '... training'
-
-
-
- patience = 10000
- patience_increase = 2
-
- improvement_threshold = 0.995
-
- validation_frequency = min(n_train_batches, patience / 2)
-
-
- best_validation_loss = numpy.inf
- best_iter = 0
- test_score = 0.
- start_time = time.clock()
-
-
- epoch = 0
- done_looping = False
-
-
-
-
-
-
-
-
-
-
- while (epoch < n_epochs) and (not done_looping):
- epoch = epoch + 1
- for minibatch_index in xrange(n_train_batches):
-
- minibatch_avg_cost = train_model(minibatch_index)
-
- iter = (epoch - 1) * n_train_batches + minibatch_index
-
- if (iter + 1) % validation_frequency == 0:
-
- validation_losses = [validate_model(i) for i
- in xrange(n_valid_batches)]
- this_validation_loss = numpy.mean(validation_losses)
-
- print(
- 'epoch %i, minibatch %i/%i, validation error %f %%' %
- (
- epoch,
- minibatch_index + 1,
- n_train_batches,
- this_validation_loss * 100.
- )
- )
-
- if this_validation_loss < best_validation_loss:
- if (
- this_validation_loss < best_validation_loss *
- improvement_threshold
- ):
- patience = max(patience, iter * patience_increase)
-
- best_validation_loss = this_validation_loss
- best_iter = iter
-
- test_losses = [test_model(i) for i
- in xrange(n_test_batches)]
- test_score = numpy.mean(test_losses)
-
- print((' epoch %i, minibatch %i/%i, test error of '
- 'best model %f %%') %
- (epoch, minibatch_index + 1, n_train_batches,
- test_score * 100.))
-
- if patience <= iter:
- done_looping = True
- break
-
- end_time = time.clock()
- print(('Optimization complete. Best validation score of %f %% '
- 'obtained at iteration %i, with test performance %f %%') %
- (best_validation_loss * 100., best_iter + 1, test_score * 100.))
- print >> sys.stderr, ('The code for file ' +
- os.path.split(__file__)[1] +
- ' ran for %.2fm' % ((end_time - start_time) / 60.))
如果有任何错误,或者有说不清楚的地方,欢迎评论留言。