Deep Learning：深度前馈神经网络（五）

Back-Propagation and Other Differentiation Algorithms

• Forward propagation: The inputs x provide the initial information that then propagates up to the hidden units at each layer and finally produces $\hat{y}$.
• Back-propagation: allows the information from the cost to then flow backwards through the network, in order to compute the gradient.
• The term back-propagation is often misunderstood as meaning the whole learning algorithm for multi-layer neural networks. Actually, back-propagation refers only to the method for computing the gradient, while another algorithm, such as stochastic gradient descent, is used to perform learning using this gradient.
• In learning algorithms, the gradient we most often require is the gradient of the cost function with respect to the parameters, $\nabla_\theta J(\theta)$.

Computational Graphs

To describe the back-propagation algorithm more precisely, it is helpful to have a more precise computational graph language.

• Here, we use each node in the graph to indicate a variable. The variable may be a scalar, vector, matrix, tensor, or even a variable of another type.
• An operation is a simple function of one or more variables. Our graph language is accompanied by a set of allowable operations. Functions more complicated than the operations in this set may be described by composing many operations together.
  

Chain Rule of Calculus

The chain rule of calculus (not to be confused with the chain rule of probability) is used to compute the derivatives of functions formed by composing other functions whose derivatives are known. Back-propagation is an algorithm that computes the chain rule, with a specific order of operations that is highly efficient.
Let x be a real number, and let f and g both be functions mapping from a real number to a real number. Suppose that $y = g(x)$ and $z = f(g(x)) = f(y)$. Then the chain rule states that

$$\frac{dz}{dx}=\frac{dz}{dy}\frac{dy}{dx}$$
We can generalize this beyond the scalar case:
$$\frac{\partial z}{\partial x_i}=\sum_j\frac{\partial z}{\partial y_j}\frac{\partial y_j}{\partial x}$$
In vector notation, this may be equivalently written as
$$\nabla_xz=\left(\frac{\partial y}{\partial x}\right)^T\nabla_yz$$
where $\frac{\partial y}{\partial x}$ is the n × m Jacobian matrix of g.
From this we see that the gradient of a variable x can be obtained by multiplying a Jacobian matrix $\frac{\partial y}{\partial x}$ by a gradient $\nabla_yz$