Intro to Deep Learning & Backpropagation 深度学习模型介绍及反向传播算法推导详解

往期文章链接目录

Deep Learning v.s. Machine Learning

The major difference between Deep Learning and Machine Learning technique is the problem solving approach. Deep Learning techniques tend to solve the problem end to end, where as Machine learning techniques need the problem statements to break down to different parts to be solved first and then their results to be combine at final stage.

Forward Propagation

The general procedure is the following:

$$\begin{array}{rl}{a}^{(1)}(x)& =w^{(1{)}^T}\cdot x+b^{(1)}\\ h^{(1)}(x)& =g_1(a^{(1)}(x))\\ a^{(2)}(x)& =w^{(2{)}^T}\cdot h^{(1)}(x)+b^{(2)}\\ h^{(2)}(x)& =g_2(a^{(2)}(x))\\ & ......\\ a^{(L+1)}(x)& =w^{(L+1{)}^T}\cdot h^{(L)}(x)+b^{(L+1)}\\ h^{(L+1)}(x)& =g_{L+1}(a^{(L+1)}(x))\end{array}$$

Note:

• $w^{(i)}$ has dimension: (# of (hidden) units in layer $i$) $\times$ (# of (hidden) units in layer $i-1$).

• $b^{(i)},a^{(i)},h^{(i)}$ have the same dimension: (# of (hidden) units in layer $i$, $1$).

• $g_i$ is an activation function. Sigmoid, tanh, relu are common activation functions. In the last layer, the choose of activation function $g_{(L+1)}$ depends on problems, usually sigmoid, and softmax.

Loss functions of neural network

Common loss functions are cross-entropy loss, hinge loss, triple loss, etc. In fact, depending on specifice problems, we can define arbitrarily loss functions. We can also use AUC as a loss.

In this post, we focus on the cross-entropy loss.

For example, let the output of the neural network be $(0.6,0.2,0.1,0.1)$, and the true label $(0,1,0,0)$, then we can write the cross-entropy loss as

$$\ell \left(\left[\begin{array}{l}0.6\\ 0.2\\ 0.1\\ 0.1\end{array}\right],\left[\begin{array}{l}0\\ 1\\ 0\\ 0\end{array}\right]\right)=-(0\cdot \log 0.6+1\cdot \log 0.2+0\cdot \log 0.1+0\cdot \log 0.1)=-\log 0.2$$

From now on, we call the output of the neural network $f(x)$, and the true label w.r.t $x$ is y, then the corss-entropy loss is written by

$$\ell \left(f(x),y\right)=-\log f(x{)}_y$$

where $f(x{)}_y$ is the $y$-th entry of $f(x)$.

Back-propagation

In fitting a neural network, backpropagation computes the gradient of the loss function with respect to the weights of the network. The backpropagation algorithm works by computing the gradient of the loss function with respect to each weight by the chain rule, computing the gradient one layer at a time, iterating backward from the last layer to avoid redundant calculations of intermediate terms in the chain rule.

In the following, we would try to compute $\frac{\partial \ell }{\partial f(x)},\frac{\partial \ell }{\partial a^{(L+1)}(x)},\frac{\partial \ell }{\partial h^{(k)}(x)},\frac{\partial \ell }{\partial a^{(k)}(x)},\frac{\partial \ell }{\partial w^{(k)}},\frac{\partial \ell }{\partial b^{(k)}}$, and then make a summary of the back-propagation process.

compute $\frac{\partial \ell }{\partial f(x)}$

First consider single element:
$$\begin{array}{rl}\frac{\partial \ell }{\partial f(x{)}_j}& =\frac{\partial -\log f(x{)}_y}{\partial f(x{)}_j}\\ & =\frac{-1}{f(x{)}_y}\cdot \frac{\partial f(x{)}_y}{\partial f(x{)}_j}\\ & =\frac{-1\cdot I}{f(x{)}_y}\end{array}$$