Deep Learning：深度前馈神经网络（二）

Output Units

The choice of cost function is tightly coupled with the choice of output unit. Most of the time, we simply use the cross-entropy between the data distribution and the model distribution. The choice of how to represent the output then determines the form of the cross-entropy function.
We suppose that the feedforward network provides a set of hidden features defined by $h = f(x; θ)$. The role of the output layer is then to provide some additional transformation from the features to complete the task that the network must perform.

Linear Units for Gaussian Output Distributions

One simple kind of output unit is an output unit based on an affine transformation with no nonlinearity. These are often just called linear units.
Given features h, a layer of linear output units produces a vector $\hat{y}=W^Th+b$. Linear output layers are often used to produce the mean of a conditional Gaussian distribution:

$$p(y|x)=\mathcal{N}(y;\hat{y},I)$$
Maximizing the log-likelihood is then equivalent to minimizing the mean squared error.
The maximum likelihood framework makes it straightforward to learn the covariance of the Gaussian too, or to make the covariance of the Gaussian be a function of the input. However, the covariance must be constrained to be a positive definite matrix for all inputs. It is difficult to satisfy such constraints with a linear output layer, so typically other output units are used to parametrize the covariance.
Because linear units do not saturate, they pose little difficulty for gradient-based optimization algorithms and may be used with a wide variety of optimization algorithms.

Sigmoid Units for Bernoulli Output Distributions

Many tasks require predicting the value of a binary variable y .
The maximum-likelihood approach is to define a Bernoulli distribution over y conditioned on x. A Bernoulli distribution is defined by just a single number. The neural net needs to predict only$P(y = 1 | x)$. For this number to be a valid probability, it must lie in the interval [0, 1].
It is better to use a different approach that ensures there is always a strong gradient whenever the model has the wrong answer. This approach is based on using sigmoid output units combined with maximum likelihood.
A sigmoid output unit is defined by

y