神经网络反向传播数学推导

---

title: Mathematic Process of Back Propagation Derivatives in Neural Network
date: 2020-04-06 12:25:09
tags: Machine Learning
mathjax: true

---

I took several days to try to figure out the process of Back Propagation in Neural Network, and after I pull through I review and record the passage here.

General Speaking

As we all know, in order to do some mathematics on Neural Network, we need to define some formulas first.

We define the Training Set as
{ ( x ( 1 ) , y ( 1 ) ) , ( x ( 2 ) , y ( 2 ) ) , ( x ( 3 ) , y ( 3 ) ) , . . . ( x ( m ) , y ( m ) ) } {\{(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),(x^{(3)},y^{(3)}),...(x^{(m)},y^{(m)})}\} {(x(1),y(1)),(x(2),y(2)),(x(3),y(3)),...(x(m),y(m))}
which means we totally have $m$ samples in training set.

In the neural network we consider now has $n_l$ layers totally, and the $1^{st}$ layer is the Input Layer and the last layer i.e. the $n_l$ layer is the Output Layer. For $i$ layer the number of Neurons is $S_i$. So the dimension of the $y^{(i)}$ is $S_{n_l}$ i.e. $y^{(i)}=[y_1^{(i)},y_2^{(i)},y_3^{(i)},...y_{S_{n_l}}^{(i)}{]}^T$ and the dimension of the $x^{(i)}$ is $S_1$ i.e. $x^{(i)}=[x_1^{(i)},x_2^{(i)},x_3^{(i)},...x_{S_1}^{(i)}{]}^T$. The connection weight from $i$ nodes in $l$ layer to $j$ nodes in $l-1$ layer is defined as $w_{ji}^{(l)}$ and the bias from $l$ layer to $l$ layer to $l-1$ layer is defined as $b_l$, in which l ∈ { n l , n l − 1 , . . . , 2 } l\in\{n_l, n_l-1, ... ,2\} l∈{nl​,nl​−1,...,2}. That means in this neural network we have

The Cost Function for a particular $(x,y)$ is defined in formula$(1)$:

$$J(w,b;x,y)=\frac{1}{2}||h_{w,b}(x)-y|{|}^2\text{\hspace{1em}(1)}$$

We use the Mean Square Error as the error criteria. For a set of { ( x ( 1 ) , y ( 1 ) ) , ( x ( 2 ) , y ( 2 ) ) , ( x ( 3 ) , y ( 3 ) ) , . . . ( x ( m ) , y ( m ) ) } {\{(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),(x^{(3)},y^{(3)}),...(x^{(m)},y^{(m)})}\} {(x(1),y(1)),(x(2),y(2)),(x(3),y(3)),...(x(m),y(m))}, the Cost Function is defined as:
$$J(w,b)=[\sum _{i=1}^{m}J(w,b;x^{(i)},y^{(i)})]+\frac{\lambda }{2}\sum _{l=2}^{n_l}\sum _{i=1}^{S_{l-1}}\sum _{j=1}^{S_l}(w_{ji}^{(l)}{)}^2\text{\hspace{1em}(2)}$$
The second term to the right of the equation$\frac{\lambda }{2}{\sum }_{l=2}^{n_l}{\sum }_{i=1}^{S_{l-1}}{\sum }_{j=1}^{S_l}(w_{ji}^{(l)})$ actually is a additional term in order to avoiding “overfitting problem”, and its derivatives of any parameters is simple. So in the following section let’s just ignore it.
The target we want to solve is the $\frac{\partial J(w,b)}{\partial w_{ji}^{(l)}}$ and $\frac{\partial J(w,b)}{\partial b_i^{(l)}}$ for optimizing those parameters by some method (like Gradient Descent). It is actually pretty hard to calculate the $\frac{\partial J(w,b)}{\partial w_{ji}^{(l)}}$ and $\frac{\partial J(w,b)}{\partial b_i^{(l)}}$ in direct since we will do a huge work of matrix derivatives. so Let’s concentrate on a easier way.

Error Term $\delta$

According to the “Neural Network and Deep Learning”, based on the chain derivation rule, formula$(6)(7)$ is defined:
$$\frac{\partial J(w,b)}{\partial w_{ji}^{(l)}}=\frac{\partial J(w,b)}{\partial z_j^{(l+1)}}\frac{\partial z_j^{(l+1)}}{\partial w_{ji}^{(l)}}\text{\hspace{1em}(3)}$$

$$\frac{\partial J(w,b)}{\partial b_j^{(l)}}=\frac{\partial J(w,b)}{\partial z_j^{(l+1)}}\frac{\partial z_j^{(l+1)}}{\partial b_j^{(l)}}\text{\hspace{1em}(4)}$$

We can use an Error Term $\delta$ to calculate the above partial derivation more easily:
$${\delta }_j^{(l)}=\frac{\partial J(w,b)}{\partial z_j^{(l+1)}}\text{\hspace{1em}(5)}$$
in which,
$$z^{(l)}=w^{(l)}{a}^{(l-1)}+b^{(l)}$$

$$a^{(l)}=f(z^{(l)})$$

Here we define the function $f$ as the Activate Function (such as sigmoid, tanh, etc.)

To calculate $\delta$, we could start from the output layer to the input layer step by step.

Calculation of Output Layer ${\delta }_i^{(n_l)}$

Calculating the output layer ${\delta }_i^{(n_l)}$ is relative easily:
$${\delta }_i^{(n_l)}=-(y_i-a_i^{(n_l)})f^{\prime }(z_i^{(n_l)})\text{\hspace{1em}(6)}$$

Proof

In the proof, we denote the Training Set is $(x,y)$.
$$\begin{array}{rl}{\delta }_i^{(n_l)}& =\frac{\partial }{\partial z_i^{(n_l)}}J(w,b)\\ & =\frac{\partial }{\partial z_i^{(n_l)}}J(w,b;x,y)\\ & =\frac{\partial }{\partial z_i^{(n_l)}}\frac{1}{2}||y-h_{w,b}(x)|{|}^2\\ & =\frac{\partial }{\partial z_i^{(n_l)}}\frac{1}{2}\sum _{j=1}^{S_{n_l}}(y_j-f(z_j^{(n_l)}){)}^2\\ & =-(y_i-f(z_j^{(n_l)}))f^{\prime }(z_j^{(n_l)})\\ & =-(y_i-a_i^{(n_l)})f^{\prime }(z_i^{(n_l)})\end{array}$$

Calculation of $n_l-1$ Layers ${\delta }_i^{(n_l)}$

For l ∈ { n l − 1 , n l − 2 , . . . , 2 } l\in\{n_l-1, n_l-2, ..., 2\} l∈{nl​−1,nl​−2,...,2} (we cannot let $l=1$ since the parameters $w$ between the input layer and the second layer is define as the $w^{(2)}$ and there is no $w^{(1)}$ parameter) the $\delta$ can be set as:
$${\delta }_i^{(l)}=\left(\sum _{j=1}^{S_{l+1}}{w}_{ji}^{(l+1)}{\delta }_j^{(l+1)}\right)f^{\prime }(z_i^{(l)})\text{\hspace{1em}(7)}$$
To proof that, we could proof the ${\delta }_i^{(n_l-1)}$ in the $n_l-1$ layer first:
$${\delta }_i^{(n_l-1)}=\left(\sum _{j=1}^{S_{n_l}}{w}_{ji}^{(n_l)}{\delta }_j^{(n_l)}\right)f^{\prime }(z_i^{(l)})\text{\hspace{1em}(8)}$$

Proof

KaTeX parse error: No such environment: align* at position 8: \begin{̲a̲l̲i̲g̲n̲*̲}̲ \delta_i^{(n_l…

Based on above mathematical process, we could conclude the formula$(9)$:
$${\delta }_i^{(n_l-1)}=\left(\sum _{j=1}^{S_{n_l}}{\delta }_j^{(n_l)}{w}_{ji}^{(n_l)}\right)f^{\prime }(z_i^{(n_l-1)})\text{\hspace{1em}(9)}$$

Calculation of Other Layers ${\delta }_i^{(l)}$

Using formula$(9)$, we change the $n_l-1$ to any layer $l$:
$${\delta }_i^{(l)}=\left(\sum _{j=1}^{S_l}{\delta }_j^{(l+1)}{w}_{ji}^{(l+1)}\right)f^{\prime }(z_i^{(l)})\text{\hspace{1em}(10)}$$

Calculation of derivatives $w_{ji}^{(l)},b^{(l)}$

According to formula$(3)(4)$, we need to calculate the $\frac{\partial z_j^{(l+1)}}{\partial w_{ji}^{(l)}}$ and $\frac{\partial z_j^{(l+1)}}{\partial b_j^{(l)}}$:
$$\frac{\partial z_j^{(l+1)}}{\partial w_{ji}^{(l)}}=\frac{\partial ({\sum }_{k=1}^{S_{n_l-1}}{a}_k^{(l)}{w}_{jk}^{(l+1)}+b^{(l)})}{\partial w_{ji}^{(l)}}=a_i\text{\hspace{1em}(11)}$$

$$\frac{\partial z_j^{(l+1)}}{\partial b^{(l)}}=\frac{\partial ({\sum }_{k=1}^{S_{n_l-1}}{a}_k^{(l)}{w}_{jk}^{(l+1)}+b^{(l)})}{\partial b^{(l)}}=1\text{\hspace{1em}(12)}$$

Using formula$(10)(11)$, we could solve $\frac{\partial J(w,b)}{\partial w_{ji}^{(l)}}$ and $\frac{\partial J(w,b)}{\partial b_i^{(l)}}$ :
$$\frac{\partial J(w,b)}{\partial w_{ji}^{(l)}}=\frac{\partial J(w,b)}{\partial z_j^{(l+1)}}\frac{\partial z_j^{(l+1)}}{\partial w_{ji}^{(l)}}={\delta }_j^{(l+1)}{a}_i\text{\hspace{1em}(13)}$$

$$\frac{\partial J(w,b)}{\partial b_j^{(l)}}=\frac{\partial J(w,b)}{\partial z_j^{(l+1)}}\frac{\partial z_j^{(l+1)}}{\partial b_j^{(l)}}={\delta }_j^{(l+1)}\cdot 1\text{\hspace{1em}(14)}$$