Denoising Autoencoders (dA)

原文地址：http://deeplearning.net/tutorial/dA.html

Denoising Autoencoders (dA)

Note

This section assumes the reader has already read through Classifying MNIST digits using Logistic Regressionand Multilayer Perceptron. Additionally it uses the following Theano functionsand concepts : T.tanh, shared variables, basic arithmetic ops, T.grad, Random numbers, floatX. If you intend to run the code on GPU also read GPU.

Note

The code for this section is available for download here.

The Denoising Autoencoder (dA) is an extension of a classicalautoencoder and it was introduced as a building block for deep networksin [Vincent08]. We will start the tutorial with a short discussion onAutoencoders.

Autoencoders

See section 4.6 of [Bengio09] for an overview of auto-encoders.An autoencoder takes an input and firstmaps it (with an encoder) to a hidden representation through a deterministic mapping, e.g.:

Where is a non-linearity such as the sigmoid.The latent representation , or code is then mapped back (with a decoder) into areconstruction of same shape as through a similar transformation, e.g.:

where ‘ does not indicate transpose, and should be seen as a prediction of , given the code .The weight matrix of the reverse mapping may beoptionally constrained by , which isan instance of tied weights. The parameters of this model (namely, , and, if one doesn’t use tied weights, also) are optimized such that the average reconstructionerror is minimized. The reconstruction error can be measured in many ways, dependingon the appropriate distributional assumptions on the input given the code, e.g., using thetraditional squared error ,or if the input is interpreted as either bit vectors or vectors ofbit probabilities by the reconstruction cross-entropy defined as :

The hope is that the code is a distributed representationthat captures the coordinates along the main factors of variation in the data(similarly to how the projection on principal components captures the main factorsof variation in the data).Because is viewed as a lossy compression of , it cannotbe a good compression (with small loss) for all , so learningdrives it to be one that is a good compression in particular for trainingexamples, and hopefully for others as well, but not for arbitrary inputs.That is the sense in which an auto-encoder generalizes: it gives low reconstructionerror to test examples from the same distribution as the training examples,but generally high reconstruction error to uniformly chosen configurations of theinput vector.

If there is one linear hidden layer (the code) andthe mean squared error criterion is used to train the network, then the hidden units learn to project the input in the span of the first principal components of the data. If the hiddenlayer is non-linear, the auto-encoder behaves differently from PCA,with the ability to capture multi-modal aspects of the inputdistribution. The departure from PCA becomes even more important whenwe consider stacking multiple encoders (and their corresponding decoders)when building a deep auto-encoder [Hinton06].

We want to implement an auto-encoder using Theano, in the form of a class,that could be afterwards used in constructing a stacked autoencoder. Thefirst step is to create shared variables for the parameters of theautoencoder ( , and, since we are using tied weights in this tutorial ):

class AutoEncoder(object):

  def __init__(self, numpy_rng, input=None, n_visible=784, n_hidden=500,
           W=None, bhid=None, bvis=None):
    """

    :type numpy_rng: numpy.random.RandomState
    :param numpy_rng: number random generator used to generate weights


    :type input: theano.tensor.TensorType
    :paran input: a symbolic description of the input or None for standalone
                  dA

    :type n_visible: int
    :param n_visible: number of visible units

    :type n_hidden: int
    :param n_hidden:  number of hidden units

    :type W: theano.tensor.TensorType
    :param W: Theano variable pointing to a set of weights that should be
              shared belong the dA and another architecture; if dA should
              be standalone set this to None

    :type bhid: theano.tensor.TensorType
    :param bhid: Theano variable pointing to a set of biases values (for
                 hidden units) that should be shared belong dA and another
                 architecture; if dA should be standalone set this to None

    :type bvis: theano.tensor.TensorType
    :param bvis: Theano variable pointing to a set of biases values (for
                 visible units) that should be shared belong dA and another
                 architecture; if dA should be standalone set this to None


    """
    self.n_visible = n_visible
    self.n_hidden = n_hidden


    # note : W' was written as `W_prime` and b' as `b_prime`
    if not W:
        # W is initialized with `initial_W` which is uniformely sampled
        # from -4*sqrt(6./(n_visible+n_hidden)) and 4*sqrt(6./(n_hidden+n_visible))
        # the output of uniform if converted using asarray to dtype
        # theano.config.floatX so that the code is runable on GPU
        initial_W = numpy.asarray(numpy_rng.uniform(
                  low=-4 * numpy.sqrt(6. / (n_hidden + n_visible)),
                  high=4 * numpy.sqrt(6. / (n_hidden + n_visible)),
                  size=(n_visible, n_hidden)), dtype=theano.config.floatX)
        W = theano.shared(value=initial_W, name='W')

    if not bvis:
        bvis = theano.shared(value=numpy.zeros(n_visible,
                                    dtype=theano.config.floatX), name='bvis')

    if not bhid:
        bhid = theano.shared(value=numpy.zeros(n_hidden,
                                          dtype=theano.config.floatX), name='bhid')


    self.W = W
    # b corresponds to the bias of the hidden
    self.b = bhid
    # b_prime corresponds to the bias of the visible
    self.b_prime = bvis
    # tied weights, therefore W_prime is W transpose
    self.W_prime = self.W.T
    # if no input is given, generate a variable representing the input
    if input == None:
        # we use a matrix because we expect a minibatch of several examples,
        # each example being a row
        self.x = T.dmatrix(name='input')
    else:
        self.x = input

    self.params = [self.W, self.b, self.b_prime]

Note that we pass the symbolic input to the autoencoder as aparameter. This is such that later we can concatenate layers ofautoencoders to form a deep network: the symbolic output (the above) ofthe k-th layer will be the symbolic input of the (k+1)-th.

Now we can express the computation of the latent representation and of the reconstructedsignal:

def get_hidden_values(self, input):
    """ Computes the values of the hidden layer """
    return T.nnet.sigmoid(T.dot(input, self.W) + self.b)

def get_reconstructed_input(self, hidden):
    """ Computes the reconstructed input given the values of the hidden layer """
    return  T.nnet.sigmoid(T.dot(hidden, self.W_prime) + self.b_prime)

And using these function we can compute the cost and the updates ofone stochastic gradient descent step :

def get_cost_updates(self, learning_rate):
    """ This function computes the cost and the updates for one trainng
    step """

    y = self.get_hidden_values(self.x)
    z = self.get_reconstructed_input(y)
    # note : we sum over the size of a datapoint; if we are using minibatches,
    #        L will  be a vector, with one entry per example in minibatch
    L = -T.sum(self.x * T.log(z) + (1 - self.x) * T.log(1 - z), axis=1)
    # note : L is now a vector, where each element is the cross-entropy cost
    #        of the reconstruction of the corresponding example of the
    #        minibatch. We need to compute the average of all these to get
    #        the cost of the minibatch
    cost = T.mean(L)

    # compute the gradients of the cost of the `dA` with respect
    # to its parameters
    gparams = T.grad(cost, self.params)
    # generate the list of updates
    updates = []
    for param, gparam in zip(self.params, gparams):
        updates.append((param, param - learning_rate * gparam))

    return (cost, updates)

We can now define a function that applied iteratively will update theparameters W, b and b_prime such that thereconstruction cost is approximately minimized.

autoencoder = AutoEncoder(numpy_rng=numpy.random.RandomState(1234), n_visible=784, n_hidden=500)
cost, updates = autoencoder.get_cost_updates(learning_rate=0.1)
train = theano.function([x], cost, updates=updates)

One serious potential issue with auto-encoders is that if there is no otherconstraint besides minimizing the reconstruction error,then an auto-encoder with inputs and anencoding of dimension at least could potentially just learnthe identity function, for which many encodings would be useless (e.g.,just copying the input), i.e., the autoencoder would not differentiatetest examples (from the training distribution) from other input configurations.Surprisingly, experiments reported in [Bengio07] nonethelesssuggest that in practice, when trained withstochastic gradient descent, non-linear auto-encoders with more hidden unitsthan inputs (called overcomplete) yield useful representations(in the sense of classification error measured on a network taking thisrepresentation in input). A simple explanation is based on theobservation that stochastic gradientdescent with early stopping is similar to an L2 regularization of theparameters. To achieve perfect reconstruction of continuousinputs, a one-hidden layer auto-encoder with non-linear hidden units(exactly like in the above code)needs very small weights in the first (encoding) layer (to bring the non-linearity ofthe hidden units in their linear regime) and very large weights in thesecond (decoding) layer.With binary inputs, very large weights arealso needed to completely minimize the reconstruction error. Since theimplicit or explicit regularization makes it difficult to reachlarge-weight solutions, the optimization algorithm finds encodings whichonly work well for examples similar to those in the training set, which iswhat we want. It means that the representation is exploiting statisticalregularities present in the training set, rather than learning toreplicate the identity function.

There are different ways that an auto-encoder with more hidden unitsthan inputs could be prevented from learning the identity, and stillcapture something useful about the input in its hidden representation.One is the addition of sparsity (forcing many of the hidden units tobe zero or near-zero), and it has been exploited very successfullyby many [Ranzato07] [Lee08]. Another is to add randomness in the transformation frominput to reconstruction. This is exploited in Restricted BoltzmannMachines (discussed later in Restricted Boltzmann Machines (RBM)), as well as inDenoising Auto-Encoders, discussed below.

Denoising Autoencoders

The idea behind denoising autoencoders is simple. In order to forcethe hidden layer to discover more robust features and prevent itfrom simply learning the identity, we train theautoencoder to reconstruct the input from a corrupted version of it.

The denoising auto-encoder is a stochastic version of the auto-encoder.Intuitively, a denoising auto-encoder does two things: try to encode theinput (preserve the information about the input), and try to undo theeffect of a corruption process stochastically applied to the input of theauto-encoder. The latter can only be done by capturing the statisticaldependencies between the inputs. The denoisingauto-encoder can be understood from different perspectives( the manifold learning perspective,stochastic operator perspective,bottom-up – information theoretic perspective,top-down – generative model perspective ), all of which are explained in[Vincent08].See also section 7.2 of [Bengio09] for an overview of auto-encoders.

In [Vincent08], the stochastic corruption processconsists in randomly setting some of the inputs (as many as half of them)to zero. Hence the denoising auto-encoder is trying to predict the corrupted (i.e. missing)values from the uncorrupted (i.e., non-missing) values, for randomly selected subsets ofmissing patterns. Note how being able to predict any subset of variablesfrom the rest is a sufficient condition for completely capturing thejoint distribution between a set of variables (this is how Gibbssampling works).

To convert the autoencoder class into a denoising autoencoder class, all weneed to do is to add a stochastic corruption step operating on the input. The input can becorrupted in many ways, but in this tutorial we will stick to the originalcorruption mechanism of randomly masking entries of the input by makingthem zero. The code belowdoes just that :

from theano.tensor.shared_randomstreams import RandomStreams

def get_corrupted_input(self, input, corruption_level):
      """ This function keeps ``1-corruption_level`` entries of the inputs the same
      and zero-out randomly selected subset of size ``coruption_level``
      Note : first argument of theano.rng.binomial is the shape(size) of
             random numbers that it should produce
             second argument is the number of trials
             third argument is the probability of success of any trial

              this will produce an array of 0s and 1s where 1 has a probability of
              1 - ``corruption_level`` and 0 with ``corruption_level``
      """
      return  self.theano_rng.binomial(size=input.shape, n=1, p=1 - corruption_level) * input

In the stacked autoencoder class (Stacked Autoencoders) theweights of the dA class have to be shared with those of ancorresponding sigmoid layer. For this reason, the constructor of the dA also gets Theanovariables pointing to the shared parameters. If those parameters are leftto None, new ones will be constructed.

The final denoising autoencoder class becomes :

class dA(object):
   """Denoising Auto-Encoder class (dA)

   A denoising autoencoders tries to reconstruct the input from a corrupted
   version of it by projecting it first in a latent space and reprojecting
   it afterwards back in the input space. Please refer to Vincent et al.,2008
   for more details. If x is the input then equation (1) computes a partially
   destroyed version of x by means of a stochastic mapping q_D. Equation (2)
   computes the projection of the input into the latent space. Equation (3)
   computes the reconstruction of the input, while equation (4) computes the
   reconstruction error.

   .. math::

       \tilde{x} ~ q_D(\tilde{x}|x)                                     (1)

       y = s(W \tilde{x} + b)                                           (2)

       x = s(W' y  + b')                                                (3)

       L(x,z) = -sum_{k=1}^d [x_k \log z_k + (1-x_k) \log( 1-z_k)]      (4)

   """

   def __init__(self, numpy_rng, theano_rng=None, input=None, n_visible=784, n_hidden=500,
              W=None, bhid=None, bvis=None):
       """
       Initialize the dA class by specifying the number of visible units (the
       dimension d of the input ), the number of hidden units ( the dimension
       d' of the latent or hidden space ) and the corruption level. The
       constructor also receives symbolic variables for the input, weights and
       bias. Such a symbolic variables are useful when, for example the input is
       the result of some computations, or when weights are shared between the
       dA and an MLP layer. When dealing with SdAs this always happens,
       the dA on layer 2 gets as input the output of the dA on layer 1,
       and the weights of the dA are used in the second stage of training
       to construct an MLP.

       :type numpy_rng: numpy.random.RandomState
       :param numpy_rng: number random generator used to generate weights

       :type theano_rng: theano.tensor.shared_randomstreams.RandomStreams
       :param theano_rng: Theano random generator; if None is given one is generated
                    based on a seed drawn from `rng`

       :type input: theano.tensor.TensorType
       :paran input: a symbolic description of the input or None for standalone
                     dA

       :type n_visible: int
       :param n_visible: number of visible units

       :type n_hidden: int
       :param n_hidden:  number of hidden units

       :type W: theano.tensor.TensorType
       :param W: Theano variable pointing to a set of weights that should be
                 shared belong the dA and another architecture; if dA should
                 be standalone set this to None

       :type bhid: theano.tensor.TensorType
       :param bhid: Theano variable pointing to a set of biases values (for
                    hidden units) that should be shared belong dA and another
                    architecture; if dA should be standalone set this to None

       :type bvis: theano.tensor.TensorType
       :param bvis: Theano variable pointing to a set of biases values (for
                    visible units) that should be shared belong dA and another
                    architecture; if dA should be standalone set this to None


       """
       self.n_visible = n_visible
       self.n_hidden = n_hidden

       # create a Theano random generator that gives symbolic random values
       if not theano_rng :
           theano_rng = RandomStreams(rng.randint(2 ** 30))

       # note : W' was written as `W_prime` and b' as `b_prime`
       if not W:
           # W is initialized with `initial_W` which is uniformely sampled
           # from -4.*sqrt(6./(n_visible+n_hidden)) and 4.*sqrt(6./(n_hidden+n_visible))
           # the output of uniform if converted using asarray to dtype
           # theano.config.floatX so that the code is runable on GPU
           initial_W = numpy.asarray(numpy_rng.uniform(
                     low=-4 * numpy.sqrt(6. / (n_hidden + n_visible)),
                     high=4 * numpy.sqrt(6. / (n_hidden + n_visible)),
                     size=(n_visible, n_hidden)), dtype=theano.config.floatX)
           W = theano.shared(value=initial_W, name='W')

       if not bvis:
           bvis = theano.shared(value = numpy.zeros(n_visible,
                                        dtype=theano.config.floatX), name='bvis')

       if not bhid:
           bhid = theano.shared(value=numpy.zeros(n_hidden,
                                             dtype=theano.config.floatX), name='bhid')

       self.W = W
       # b corresponds to the bias of the hidden
       self.b = bhid
       # b_prime corresponds to the bias of the visible
       self.b_prime = bvis
       # tied weights, therefore W_prime is W transpose
       self.W_prime = self.W.T
       self.theano_rng = theano_rng
       # if no input is given, generate a variable representing the input
       if input == None:
           # we use a matrix because we expect a minibatch of several examples,
           # each example being a row
           self.x = T.dmatrix(name='input')
       else:
           self.x = input

       self.params = [self.W, self.b, self.b_prime]

   def get_corrupted_input(self, input, corruption_level):
       """ This function keeps ``1-corruption_level`` entries of the inputs the same
       and zero-out randomly selected subset of size ``coruption_level``
       Note : first argument of theano.rng.binomial is the shape(size) of
              random numbers that it should produce
              second argument is the number of trials
              third argument is the probability of success of any trial

               this will produce an array of 0s and 1s where 1 has a probability of
               1 - ``corruption_level`` and 0 with ``corruption_level``
       """
       return  self.theano_rng.binomial(size=input.shape, n=1, p=1 - corruption_level) * input


   def get_hidden_values(self, input):
       """ Computes the values of the hidden layer """
       return T.nnet.sigmoid(T.dot(input, self.W) + self.b)

   def get_reconstructed_input(self, hidden ):
       """ Computes the reconstructed input given the values of the hidden layer """
       return  T.nnet.sigmoid(T.dot(hidden, self.W_prime) + self.b_prime)

   def get_cost_updates(self, corruption_level, learning_rate):
       """ This function computes the cost and the updates for one trainng
       step of the dA """

       tilde_x = self.get_corrupted_input(self.x, corruption_level)
       y = self.get_hidden_values( tilde_x)
       z = self.get_reconstructed_input(y)
       # note : we sum over the size of a datapoint; if we are using minibatches,
       #        L will  be a vector, with one entry per example in minibatch
       L = -T.sum(self.x * T.log(z) + (1 - self.x) * T.log(1 - z), axis=1 )
       # note : L is now a vector, where each element is the cross-entropy cost
       #        of the reconstruction of the corresponding example of the
       #        minibatch. We need to compute the average of all these to get
       #        the cost of the minibatch
       cost = T.mean(L)

       # compute the gradients of the cost of the `dA` with respect
       # to its parameters
       gparams = T.grad(cost, self.params)
       # generate the list of updates
       updates = []
       for param, gparam in zip(self.params, gparams):
           updates.append((param, param - learning_rate * gparam))

       return (cost, updates)

Putting it All Together

It is easy now to construct an instance of our dA class and trainit.

# allocate symbolic variables for the data
index = T.lscalar()  # index to a [mini]batch
x = T.matrix('x')  # the data is presented as rasterized images

######################
# BUILDING THE MODEL #
######################

rng = numpy.random.RandomState(123)
theano_rng = RandomStreams(rng.randint(2 ** 30))

da = dA(numpy_rng=rng, theano_rng=theano_rng, input=x,
        n_visible=28 * 28, n_hidden=500)

cost, updates = da.get_cost_updates(corruption_level=0.2,
                            learning_rate=learning_rate)


train_da = theano.function([index], cost, updates=updates,
     givens = {x: train_set_x[index * batch_size: (index + 1) * batch_size]})

start_time = time.clock()

############
# TRAINING #
############

# go through training epochs
for epoch in xrange(training_epochs):
    # go through trainng set
    c = []
    for batch_index in xrange(n_train_batches):
        c.append(train_da(batch_index))

    print 'Training epoch %d, cost ' % epoch, numpy.mean(c)

end_time = time.clock

training_time = (end_time - start_time)

print ('Training took %f minutes' % (pretraining_time / 60.))

In order to get a feeling of what the network learned we are going toplot the filters (defined by the weight matrix). Bare in mind however,that this does not provide the entire story,since we neglect the biases and plot the weights up to a multiplicativeconstant (weights are converted to values between 0 and 1).

To plot our filters we will need the help of tile_raster_images (seePlotting Samples and Filters) so we urge the reader to familiarize himself withit. Also using the help of PIL library, the following lines of code willsave the filters as an image :

image = PIL.Image.fromarray(tile_raster_images(X=da.W.get_value(borrow=True).T,
             img_shape=(28, 28), tile_shape=(10, 10),
             tile_spacing=(1, 1)))
image.save('filters_corruption_30.png')

Running the Code

To run the code :

python dA.py

The resulted filters when we do not use any noise are :

The filters for 30 percent noise :